Release Notes

pygcam 2.0

The main new feature of pygcam version 2.0 – which resulted in a wide set of changes – is the ability to run the GCAM data system per trial in a Monte Carlo simulation (MCS). This enables the application of distributions to values in data system input (CSV) files, with all dependent XML files receiving consistent values.

This change required separation of distinct portions of the setup process so that configuration files could be created and modified before updating the data system to produce new XML files, and finally modifying the potentially new XML files if needed in the final setup phase.

In addition, some files that were written when the simulation was generated (via the gensim command) in pygcam 1.x needed to be written per trial, requiring them to be stored in per-trial directories rather than in shared ones.

The pygcam sandbox structure had been stable for many years and the code sometimes built these assumptions into pathname creation. All of these had to be revisted in version 2.0.

Removed features

  • Dropped support for GCAM versions prior to version 5.3

  • Dropped support for SALib

  • Eliminated infrequently used command-line arguments, opting instead to control these via config variables

    • Config vars can be set on the command-line: gt --set variable=value

Configuration Variable Name Changes

Rationalized configuration variable names, prefixing with Sandbox or Project in most cases, with these changes:

Old name

New Name

GCAM.XmlSrcDir

GCAM.ProjectXmlSrc

GCAM.ProjectXmlFile

GCAMProjectFile

MCS.ParametersFile

MCS.ProjectParametersFile

MCS.Root

MCS.SandboxRoot

MCS.ResultsFile

MCS.ProjectResultsFile

MCS.RunSimsDir

MCS.SandboxSimsDir

MCS.RunDbDir

MCS.SandboxDbDir

MCS.RunDbPath

MCS.SandboxDbPath

MCS.DbURL

MCS.SandboxDbURL

MCS.UserFilesDir

MCS.ProjectFilesDir

MCS.RunWorkspace

MCS.SandboxWorkspace

MCS.RunInputDir

MCS.SandboxInputDir

MCS.PlotDir

MCS.SandboxPlotDir

GCAM.ScenarioSetupFile

GCAM.ScenariosFile

GCAM.ScenarioGroup

GCAM.ScenarioSubdir

Note

In pygcam terminology, a workspace is an original GCAM distribution workspace or a copy thereof. A sandbox is a subcategory of workspace created by pygcam in which GCAM is run for a specific scenario. It generally has links back to a shared (non-sandbox) workspace.

Changes to Sandbox Structure

Note

VERIFY THIS

  • (Hopefully) eliminated redundant directory level in {sandbox}/{scenario}/local-xml => Workspace/local-xml/{scenario}, resulting in {sandbox}/{scenario}/local-xml/{scenario} by

    • Moving local-xml out of Workspace, to top level shared/{scenario}/local-xml, and

    • Changing symlink to {sandbox}/{scenario}/local-xml => shared/{scenario}/local-xml

  • Revised runtime directory structure for both non-MCS and MCS cases

Code Revisions

  • Refactored to reduce the size of very long (> 2000 line) files. New files include file_utils.py, sandbox.py, xml_edit.py

  • Debugged existing tests and added many new ones

  • Added new setup options to run only config-modifying commands or the non-config-modifying commands. This allows the GCAM data system to be run between these two setup steps.

Note

REVISIT THIS NEXT SECTION

Using sub-directories with scenario groups

  • Dropped support for the <scenario> element’s subdir attribute in scenarios.xml.

  • Distinguish and create better names for:

    • <project subdir=”xxx”>

    • <scenario subdir=”xxx”> # dropped support for this, but might still be trickling through

    • scenario group subdir if <scenarioGroup useGroupDir=”true”>

      • runsim takes (scenario) group name

    • XML group subdir

      • whether to use scenario group in the xmlsrc directory as well, otherwise all files at top level under xmlsrc

    • sandbox group subdir

      • Allows a subdir between {SandboxRoot}/{ProjectName}/{SandboxSubdir}/{scenario}

      • This can differ from scenario group name (is this really necessary?)

  • sim_subdir in McsSandbox. New (?)

    • allows multiple simulations (with different reference workspace?) under the same project name

    • Note that this is different than sim_id, which allows multiple simulations using the same GCAM files