GCAM tool (`gt`)¶

The gt script unifies GCAM workflow managment functionality into a single script with sub-commands. Generic sub-commands are implemented directly by the pygcam library. Project-specific features can be added via plugins.

Note

Quick links to sub-commands: chart, config, diff, gcam, gui, init, mcs, mi, new, protect, query, run, setup, sandbox

The sub-commands support all the major workflow setups, including

Modify XML files and configuration.xml to set up a modeling experiment (See the setup sub-command and GCAM XML-Setup for more information.)

Run GCAM in an automatically-created workspace, allowing multiple instances of GCAM to run simultaneously, e.g., on parallel computing systems (See the gcam sub-command.)

Execute batch queries against the XML database to extract GCAM results, with on-the-fly regionalization based on a simple region-mapping file. (See the query sub-command.)

Compute differences between policy and baseline scenarios, including linear annualization of values between time-steps, and (See the diff sub-command.)

Plot results, with flexible control of figure features including title, axis labels, scale, and so on. (See the chart sub-command.)

Manage (create, delete, rename, run commands in) automatically-created workspaces. (See the sandbox sub-command.)

In addition, the run sub-command allows workflow steps to be defined in an XML file so that individual or groups of steps can be executed for one or more scenarios. The run sub-command supports direct invocation of other workflow steps as well as running arbitrary programs of the user’s choosing.

Finally, gt allows all project steps to be run on a compute node in a High-Performance Computing environment by specifying +b or --batch on the command-line. (Note that this is not available on Mac OS X or Windows.)

For example, the command:

gt +b +P MyProject run -S MyScenario

runs all steps for scenario MyScenario in the project MyProject by queuing a batch job on the default queue. Arguments to gt allow the user to set various resource requirements and to select the queue to use.

The command to run to queue the batch job is taken from the configuration file parameter GCAM.BatchCommand. Example batch commands for the SLURM and PBS job management systems are provided in variables GCAM.QueueSLURM and GCAM.QueuePBS, respectively.

Command-line usage is described below. Note that some command-line (e.g., batch-related) options must precede the sub-command, whereas sub-command specific options must follow it.

Note

Note that arguments that pertain regardless of the sub-command (e.g., +P to identify the project name) are specified prior to the sub-command, and use + rather than -. This is to avoid conflicts between these “main” arguments and sub-command arguments. (An exception is gt -h, which retains the -.) Long-form argument names use two hyphens, as in --projectName.)

Usage¶

usage: gt [-h] [+b] [+B] [+D DIRMAP] [+e ENVIROVARS] [+j JOBNAME]
          [+l LOGLEVEL] [+L LOGFILE] [+m MINUTES] [+M {trial,gensim}]
          [+P name] [+q QUEUENAME] [+r RESOURCES] [+s name=value] [+v]
          [--version] [--VERSION]
          {building,chart,compare,config,diff,gcam,gui,init,mcs,mi,new,protect,query,res,run,sandbox,setup}
          ...

Named Arguments¶

`+b, --batch`	Run the commands by submitting a batch job using the command given by config variable GCAM.BatchCommand. (Linux only) Default: False
`+B, --showBatch`
	Show the batch command to be run, but don’t run it. (Linux only) Default: False
`+D, --dirmap`	A comma-delimited sequence of colon-delimited directory names of the form “/some/host/path:/a/container/path, /host:cont, …”, mapping host dirs to their mount point in a docker container.
`+e, --enviroVars`
	Comma-delimited list of environment variable assignments to pass to queued batch job, e.g., -E “FOO=1,BAR=2”. (Linux only)
`+j, --jobName`	Specify a name for the queued batch job. Default is “gt”. (Linux only) Default: “gt”
`+l, --logLevel`	Sets the log level for modules of the program. A default log level can be set for the entire program, or individual modules can have levels set using the syntax “module:level, module:level,…”, where the level names must be one of {debug,info,warning,error,fatal} (case insensitive). Default: “INFO”
`+L, --logFile`	Sets the name of a log file for batch runs. Default is “gt-%j.out” where “%j” (in SLURM) is the jobid. If the argument is not an absolute pathname, it is treated as relative to the value of GCAM.LogDir.
`+m, --minutes`	Set the number of minutes to allocate for the queued batch job. Overrides config parameter GCAM.Minutes. (Linux only) Default: 20.0
`+M, --mcs`	Possible choices: trial, gensim Used only when running gcamtool from pygcam-mcs.
`+P, --projectName`
	The project name (the config file section to read from), which defaults to the value of config variable GCAM.DefaultProject Default: “”
`+q, --queueName`
	Specify the name of the queue to which to submit the batch job. Default is given by config variable GCAM.DefaultQueue. (Linux only) Default: “slurm”
`+r, --resources`
	Specify resources for the queued batch command. Can be a comma-delimited list of assignments of the form NAME=value, e.g., -r ‘pvmem=6GB’. (Linux only) Default: “”
`+s, --set`	Assign a value to override a configuration file parameter. For example, to set batch commands to start after a prior job of the same name completes, use –set “GCAM.OtherBatchArgs=-d singleton”. Enclose the argument in quotes if it contains spaces or other characters that would confuse the shell. Use multiple –set flags and arguments to set multiple variables. Default: []
`+v, --verbose`	Show diagnostic output Default: False
`--version`	show program’s version number and exit
`--VERSION`	Default: False

Subcommands¶

For help on subcommands, use the “-h” flag after the subcommand name

subcommand

Possible choices: building, chart, compare, config, diff, gcam, gui, init, mcs, mi, new, protect, query, res, run, sandbox, setup

Sub-commands:¶

building¶

Dump combinations of building energy use sectors, techs, and fuels.

gt building [-h] [-p PROJECT] [-o OUTPUTFILE] [-s SECTORS] [-r REGIONS] [-u]
            [-y YEARS]

Named Arguments¶

`-p, --project`	The name of the project to use to locate GCAM reference files. Default is “gcam_res” Default: “gcam_res”
`-o, --outputFile`
	The CSV file to create with lists of unique building sectors, subsectors, and technologies. Default is “building_tech_template.csv” Default: “building_tech_template.csv”
`-s, --sectors`	A comma-delimited list of sectors to include in the generated template. Use quotes around the argument if there are embedded blanks. By default, all known building technology sectors are included.
`-r, --regions`	A comma-delimited list of regions to include in the generated template. By default all regions are included.
`-u, --GCAM-USA`	If set, produce output compatible with GCAM-USA regions. Default: False
`-y, --years`	A hyphen-separated range of timestep years to include in the generated template. Default is “2015-2100” Default: “2015-2100”

chart¶

The chart sub-command generates plots from GCAM-style “.csv” files. Two types of plots are currently supported: (i) stacked bar plots based on summing values over all years (with optional interpolation of annual values), by the given ‘indexCol’ (default is ‘region’), and (ii) stacked bar plots by year for some data column, where the data are grouped by and summed across elements with the indicated ‘indexCol’. The first option is indicated by using the -S (--sumYears) option. Numerous options allow the appearance to be customized.

You can perform on-the-fly unit conversions using the -m / --multiplier or -V / --divisor arguments, which cause all values in “year columns” to be multiplied or divided, respectively, by the values provided. Values can be specified as numeric constants or using symbolic constants defined in the pygcam.units module.

gt chart [-h] [-b] [-B] [-c COLUMNS] [-C CONSTRAINT] [-d OUTPUTDIR]
         [-D WORKINGDIR] [-e] [-f FROMFILE] [-F DIVISORFILE] [--format FORMAT]
         [-g] [-i] [-I INDEXCOL] [-k] [-l] [-L LABELCOLOR] [-m MULTIPLIER]
         [-M MULTIPLIERFILE] [-n NCOL] [-N SCENARIO] [--negate] [-o OUTFILE]
         [-O] [-p PALETTE] [-r ROTATION] [-R REFERENCE] [--region REGION]
         [-s SKIPROWS] [-S] [-t YEARSTEP] [-T TITLE] [--timeseries]
         [--transparent] [-u UNSTACKEDCOL] [-U UNSTACKEDREGION] [-v VALUECOL]
         [-V DIVISOR] [-x SUFFIX] [-X XLABEL] [-Y YLABEL] [-y YEARS]
         [--ymax YMAX] [--ymin YMIN] [-z] [--legendY LEGENDY]
         [--barWidth BARWIDTH]
         [csvFile]

Positional Arguments¶

csvFile

The file containing the data to plot.

Named Arguments¶

`-b, --box`	Draw a box around the plot. Default is no box. Default: False
`-B, --byRegion`	Generate one plot per region. Region names are read from the CSV file, so they reflect any regional aggregation produced by the query. Default: False
`-c, --columns`	Specify the column whose values identify the segments in the stacked bar chart. (These appear in the legend.) Default: “output”
`-C, --constraint`
	Apply a constraint to limit the rows of data to plot. The constraint can be any constraint string that is valid for the DataFrame.query() method, e.g., -C ‘input == “biomass”’
`-d, --outputDir`
	The directory into which to write image files. Default is “.” Default: “.”
`-D, --workingDir`
	The directory to change to before performing any operations Default: “.”
`-e, --enumerate`
	Prefix image filenames with sequential number for easy reference. Used only with –fromFile Default: False
`-f, --fromFile`	A file from which to read argument strings, one per line. These are read as if chartGCAM.py were called on each line individually, but avoiding the ~2 sec startup time for the bigger python packages.
`-F, --divisorFile`
	A file containing a floating point value to divide data by before plotting. See also -V.
`--format`	Specify a format for the Y-axis. Possible values are ‘.’ for float, ‘,’ for int with commas, or any format recognized by print, e.g., “%.2f” to Y values as floats with 2 decimal places.
`-g, --ygrid`	Show light grey horizontal lines at the major Y-axis ticks. Default is no grid. Default: False
`-i, --interpolate`
	Interpolate (linearly) annual values between timesteps. Default: False
`-I, --indexCol`	A column to use as the index column, or blank for None. This column is displayed on the X-axis of stacked barcharts. Default value is “region”. Default: “region”
`-k, --yticks`	Show tick marks on Y-axis. Default is no tick marks. Default: False
`-l, --label`	Add text along the right side of the figure showing the filename. Default: False
`-L, --labelColor`
	Color for the text label, which defaults to lightgrey. Some users may prefer “black”, for example. (Implies -l)
`-m, --multiplier`
	A value to multiply data by before generating the plot. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. For example, “-m 3.667” and “-m C_to_CO2” are equivalent, and effectively convert values from Tg C to Tg CO2. Be sure to adjust the Y axis label. See also -M.
`-M, --multiplierFile`
	A file containing a floating point value to multiply data by before plotting. See also -m.
`-n, --ncol`	The number of columns with which to display the legend. Default is 5. Default: 5
`-N, --scenario`	When using the ‘–fromFile’ option, this argument is used to specify one or more scenario names (delimited by commas if more than one). These are substituted into each line read from the file as the value for “{scenario}” wherever it appears on each line read from the ‘fromFile’. Default: “”
`--negate`	Multiply data by -1 before plotting, which can make interpretation of some figures more intuitive. The string “-negated” is added to the file label, displayed if the “-l” or “-L” flag is specified. Default: False
`-o, --outFile`	The name of the image file to create. Format is determined from filename extension. All common formats are allowed, e.g., png, pdf, tif, and gif. Try it; it probably works. Default is the name of the data file substituting “.png” for “.csv” Default: “”
`-O, --open`	Open the plot file after generating it. Default: False
`-p, --palette`	The name of a color palette to use. Some good options include hls, husl, and Paired. See http://stanford.edu/~mwaskom/software/seaborn/tutorial/color_palettes.html
`-r, --rotation`	Set the rotation angle for X-axis labels. Defaults to 90 degrees (vertical). Use 0 for horizontal labels. Default: 90
`-R, --reference`
	When using the ‘–fromFile’ option, this argument is used to specify the name of the reference scenario. The “other” scenario is given using the “-N” option. These are substituted into each line read from the file as the value for “{scenario}” and “{reference}” (without the quotes) wherever they appear on each line read from the ‘fromFile’. Defaults to “reference” Default: “reference”
`--region`	Plot values only for the given region.
`-s, --skiprows`	The number of rows of the CSV file to skip before reading the data (starting with a header row with column names.) Default is 1, which works for GCAM batch query output. Default: 1
`-S, --sumYears`	Sum across the time horizon, typically by region. This results in a stacked bar plot. When not summed over years (the default) a stacked area plot is generated showing values grouped and summed by indexCol (-I) and presented by year. Default: False
`-t, --yearStep`	The spacing of year labels on X-axis for time-series plots. Defaults to 5. Default: 5
`-T, --title`	Adds a title to the plot. Default is no title. The string can have LaTeX math language in it, e.g., ‘CO$_2$’ causes the 2 to be subscripted, and ‘MJ$^{-1}$’ results in “MJ” with a superscripted exponent of -1. The string ‘$Delta$’ results in a capital Greek delta. See LaTeX documentation for more options. Be sure to enclose the title in single quotes so the “$” is not (mis)interpreted by the shell. Default: “”
`--timeseries`	Plot the data as a time series. Default: False
`--transparent`	Save the plot with a transparent background. (Default is white.) Default: False
`-u, --unstackedCol`
	Draw an unstacked bar plot for the column given as an argument to this option, showing three groups of bars: the region, all other regions, and the total.
`-U, --unstackedRegion`
	The region to plot separately from Rest of World in an unstacked plot. Ignored if –byRegion is specified, in which case a plot is created for all regions.
`-v, --valueCol`	Identify a single column (e.g., a year) to plot for bar plots.
`-V, --divisor`	A value to divide year column values by before plotting. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. See also -F.
`-x, --suffix`	A suffix to append to the basename of the input csv file to create the name for the output file. For example, if processing my_data.csv, indicating -x ‘-by-region.pdf’ results in an output file named my_data-by-region.pdf.
`-X, --xlabel`	Defines a label for the X-axis; defaults to blank. LaTeX math language is supported. (See the -T flag for more info.) Default: “”
`-Y, --ylabel`	Label for the Y-axis; defaults to “EJ”. LaTeX math language is supported. (See the -T flag for more info.) Default: “EJ”
`-y, --years`	Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Data for all other years are dropped. Default: “”
`--ymax`	Set the scale of a figure by indicating the value to show as the maximum Y value. (By default, scale is set according to the data.)
`--ymin`	Set the scale of a figure by indicating the value (given as abs(value), but used as -value) to show as the minimum Y value
`-z, --zeroLine`	Whether to show a line at Y=0 Default: False
`--legendY`	The Y position of the legend. Useful for fixing poorly formatted figures. Note that to pass a negative value, use the syntax –legendY=”-xxx.xxx”, otherwise the hyphen is interpreted as indicating a command-line argument.
`--barWidth`	The relative width of bars. Helpful when plotting only 1 or 2 bar, so they aren’t obnoxiously wide. Default is 0.5 Default: 0.5

compare¶

Compare two GCAM configuration files and the files they load to: find differences. Files are compared using “diff” based on matching “name” tags.

gt compare [-h] [-o OUTPUTDIR] config1 exedir1 config2 exedir2

Positional Arguments¶

`config1`	The first of two config files to compare.
`exedir1`	The “exe” from which config1 pathnames should be computed.
`config2`	The second of two config files to compare.
`exedir2`	The “exe” from which config2 pathnames should be computed.

Named Arguments¶

-o, --outputDir

The directory in which to create the normalized versions of XML: input files for comparison. Default is /tmp/xmlCompare

Default: “/tmp/xmlCompare”

config¶

The config command list the values of configuration variables from ~/.pygcam.cfg. With no arguments, it displays the values of all variables for the default project. Use the -d flag to show only values from the [DEFAULT] section.

If an argument name is provided, it is treated as a substring pattern, unless the -x flag is given (see below). All configuration variables containing the give name are displayed with their values. The match is case-insensitive.

If the -x or --exact flag is specified, the argument is treated as an exact variable name (case-sensitive) and only the value is printed. This is useful mainly for scripting. For general use the substring matching is more convenient.

Examples:

$ gt config project
[MyProject]
GCAM.DefaultProject = MyProject
GCAM.ProjectRoot = /Users/rjp/bitbucket/myProject
GCAM.ProjectXmlFile = /Users/rjp/bitbucket/myProject/etc/project.xml

$ gt config -x GCAM.DefaultProject
MyProject

$ gt config sand
MyProject]
GCAM.SandboxRoot = /Users/rjp/ws/myProject

$ gt config sand -d
[DEFAULT]
GCAM.SandboxRoot = /Users/rjp/ws

gt config [-h] [-d] [-e] [-x] [-t] [name]

Positional Arguments¶

name

Show the names and values of all parameters whose name contains the given value. The match is case-insensitive. If not specified, all variable values are shown.

Default: “”

Named Arguments¶

`-d, --useDefault`
	Indicates to operate on the DEFAULT section rather than the project section. Default: False
`-e, --edit`	Edit the configuration file. The command given by the value of config variable GCAM.TextEditor is run with the .pygcam.cfg file as an argument. Default: False
`-x, --exact`	Treat the text not as a substring to match, but as the name of a specific variable. Match is case-sensitive. Prints only the value. Default: False
`-t, --test`	Test the settings in the configuration file to ensure that the basic setup is ok, i.e., required parameters have values that make sense. If specified, no variables are displayed. Default: False

diff¶

The diff sub-command script computes the differences between results from two or more CSV files generated from batch queries run on a GCAM database, saving the results in either a CSV or XLSX file, according to the extension given to the output file. If not provided, the output filename defaults to differences.csv.

If multiple otherFiles are given (i.e., the referenceFile plus 2 or more other files named on the command-line), the resulting CSV file will contain one difference matrix for each otherFile, with a label indicating which pair of files were used to produce each result.

When the output file is in XLSX format, each result is written to a separate worksheet. If the -c flag is specified, no differences are computed; rather, the .csv file contents are combined into a single .xlsx file.

gt diff [-h] [-D WORKINGDIR] [-g GROUPSUM] [-i] [-o OUTFILE] [-c] [-p]
        [-q QUERYFILE] [-r REWRITESETSFILE] [-S] [-s SKIPROWS] [-y YEARS]
        [-Y STARTYEAR]
        csvFiles [csvFiles ...]

Positional Arguments¶

csvFiles

The files to process. For difference operations, the first file is treated: as the reference file whose time-series data is subtracted from that of each other file. If missing, “.csv” suffixes are added to all arguments (the “.csv” is optional).

Named Arguments¶

`-D, --workingDir`
	The directory to change to before performing any operations Default: “.”
`-g, --groupSum`	Group data for each timestep (or interpolated annual values) by the given column, and sum all members of each group to produce a timeseries for each group. Takes precedence over the simpler “-S” (“–sum”) option. Default: “”
`-i, --interpolate`
	Interpolate (linearly) annual values between timesteps. Default: False
`-o, --outFile`	The name of the “.csv” or “.xlsx” file containing the differences between each scenario and the reference. Default is “differences.csv”. Default: “differences.csv”
`-c, --convertOnly`
	Convert the given CSV files into an Excel workbook, one sheet per CSV file. Default: False
`-p, --asPercentChange`
	Compute percent change rather than simple difference. Default: False
`-q, --queryFile`
	If the extension is “.xml” (case insensitive), the argument must be an XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml. If the file doesn’t end in “.xml”, it must be a text file listing the names of queries to process, one per line. NOTE: When –queryFile is specified, the two positional arguments are required: the names of the baseline and policy scenarios, in that order. Default: “”
`-r, --rewriteSetsFile`
	An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”)
`-S, --sum`	Sum all timestep (or interpolated annual values) to produce a single time-series. Default: False
`-s, --skiprows`	The number of rows to skip. Default is 1, which works for GCAM batch query output. Use -s0 for outFile.csv Default: 1
`-y, --years`	Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Other years are dropped (except for annual outputs.) Default: “”
`-Y, --startYear`
	The year at which to begin interpolation Default: 0

gcam¶

The gcam sub-command runs the GCAM executable on the designated configuration file, scenario, or workspace. Typical use (e.g., from a project.xml file) would be to run GCAM by referencing a directory named the same as a scenario, holding a file called config.xml, as is generated by the setup sub-command. (See GCAM XML-Setup.)

If a workspace is specified on the command-line, it is used. Otherwise, if a scenario is specified, the workspace defined by {GCAM.SandboxDir}/{scenario} is used. If neither workspace nor scenario are defined, the value of config variable GCAM.RefWorkspace is used, i.e., GCAM is run in the reference workspace.

If the workspace doesn’t exist, it is created based on the reference GCAM workspace, defined by the configuration variable GCAM.RefWorkspace. By default, read-only directories (e.g., input and libs) are symbolically linked from the new workspace to the reference one. (See the new sub-command for more information on the creation of workspaces.)

Directories into which GCAM writes results (e.g., output and exe) are created in the new workspace, but read-only files within exe (e.g., the GCAM executable) are symbolically linked (with the same caveat for Windows users.)

Usage example:

gt gcam -S ~/MyProject/scenarios -s MyScenario -w ~/sandboxes/MyProject/MyScenario

would run the scenario MyScenario in the newly created sandbox (workspace) ~/sandboxes/MyProject/MyScenario using the configuration file ~/MyProject/scenarios/MyScenario/config.xml.

gt gcam [-h] [-C CONFIGFILE] [-f] [-g GROUPDIR] [-n] [-r REFWORKSPACE]
        [-s SCENARIO] [-S SCENARIOSDIR] [-w WORKSPACE] [-W]

Named Arguments¶

`-C, --configFile`
	Specify the one or more GCAM configuration filenames, separated by commas. If multiple configuration files are given, the are run in succession in the same “job” on the cluster.
`-f, --forceCreate`
	Re-create the workspace, even if it already exists. Default: False
`-g, --groupDir`	The scenario group directory name, if any. Default: “”
`-n, --noRun`	Don’t run GCAM; just print the command that would be run. Default: False
`-r, --refWorkspace`
	The reference workspace to use to create the new sandbox. This is used only if the indicated or implied workspace doesn’t exist. Defaults to the value of GCAM.RefWorkspace.
`-s, --scenario`	The scenario to run. Default: “”
`-S, --scenariosDir`
	Specify the directory holding scenario files. Default is the value of config variable GCAM.ScenariosDir, if set, otherwise it’s the current directory. Default: “”
`-w, --workspace`
	Specify the path to the GCAM workspace to use. If it doesn’t exist, the named workspace will be created. If not specified on the command-line, the path constructed as {GCAM.SandboxDir}/{scenario} is used.
`-W, --noWrapper`
	Do not run gcam within a wrapper that detects errors as early as possible and terminates the model run. By default, the wrapper is used. Default: False

gui¶

Run the Graphical User Interface (GUI) generated from the command-line interface in a local web server available at http://127.0.0.1:8050.

gt gui [-h] [-d]

Named Arguments¶

-d, --debug

Set the dash (flask) debug flag.

Default: False

init¶

Create the configuration file ~/.pygcam.cfg and initialize key variables, based on command-line arguments, or interactive prompts. See Initializing the configuration file for details.

gt init [-h] [-c | -C] [-g GCAMDIR] [--overwrite] [-P DEFAULTPROJECT]
        [-p PROJECTDIR] [-s SANDBOXDIR]

Named Arguments¶

`-c, --create-project`
	Create the project structure for the given default project. If neither -c/–create-project nor -C/–no-create-project is specified, the user is queried interactively.
`-C, --no-create-project`
	Do not create the project structure for the given default project. Mutually exclusive with -c / –create-project option.
`-g, --gcamDir`	The directory that is a GCAM v4.x or v5.x workspace. Sets config var GCAM.RefWorkspace. By default, looks for gcam-v5.1.2 (then v4.4.1) in ~, ~/GCAM, and ~/gcam, ~/Documents/GCAM, and ~/Documents/gcam, where “~” indicates your home directory.
`--overwrite`	Overwrite an existing config file. (Makes a backup first in ~/.pygcam.cfg~.) Default: False
`-P, --defaultProject`
	Set the value of config var GCAM.DefaultProject to the given value.
`-p, --projectDir`
	The directory in which to create pygcam project directories. Sets config var GCAM.ProjectRoot. Default is “~/GCAM/projects”.
`-s, --sandboxDir`
	The directory in which to create pygcam project directories. Sets config var GCAM.SandboxRoot. Default is “~/GCAM/sandboxes”.

mcs¶

Enable or disable Monte Carlo Simulation (MCS) mode, or check whether MCS mode is currently enabled or disabled.

gt mcs [-h] {on,off,status}

Positional Arguments¶

mode

Possible choices: on, off, status

Turn MCS mode on or off, or report current setting

mi¶

Invoke ModelInterface from the command-line after changing directory to the value of config variable GCAM.QueryDir. If the file model_interface.properties is found, it is used as is, unless the -u/--updateProperties flag is specified, in which case the file is modified so that the queryFile entry refers to the value of GCAM.MI.QueryFile, if this refer to an existing file, otherwise, by variable the GCAM.MI.RefQueryFile.

If the file model_interface.properties is not found, it is created automatically before invoking ModelInterface.

If the -d/--useDefault flag is given, the model_interface.properties file is modified to refer to the GCAM reference Main_Queries.xml file.

If you have a customized queries XML file, set the config variable GCAM.MI.QueryFile to the path to this file and it will be loaded into ModelInterface via this command.

gt mi [-h] [-d] [-u]

Named Arguments¶

`-d, --useDefault`
	Use the Main_Queries.xml file from the GCAM reference workspace. Default: False
`-u, --updateProperties`
	Update the “model_interface.properties” file in the directory indicated by config var file GCAM.QueryDir so it refers to the query file indicated by config var GCAM.MI.QueryFile, or if this does not refer to an existing file, by var GCAM.MI.RefQueryFile. Default: False

new¶

Create the directory structure and basic files required for a new pygcam project. If a directory is specified with the -r flag, the project is created with the given name in that directory; otherwise the project is created in the directory identified by the config variable GCAM.ProjectRoot.

This sub-command creates examples of xmlsrc/scenarios.py, etc/protection.xml, etc/project.xml, etc/rewriteSets.xml, and etc/scenarios.xml that can be edited to fit the needs of your project. The file etc/Instructions.txt is also created to provide further information.

If the -c flag is given, a basic entry for the new project is added to the users configuration file, $HOME/.pygcam.cfg. Before modifying the config file, a backup is created in $HOME/.pygcam.cfg~. For example, the command

gt new -c foo

generates and entry like this:

[foo]
# Added by "new" sub-command Thu Sep 22 14:30:29 2016
GCAM.ProjectDir        = %(GCAM.ProjectRoot)s/foo
GCAM.ScenarioSetupFile = %(GCAM.ProjectDir)s/etc/scenarios.xml
GCAM.RewriteSetsFile   = %(GCAM.ProjectDir)s/etc/rewriteSets.xml

The example project defines two scenario groups, consisting of a baseline and 4 carbon tax scenarios. In one group, 90% of unmanaged land is protected (i.e., removed from consideration), as in the reference GCAM scenario. In the other scenario group, this protection is not performed, so all land is considered available for use.

gt new [-h] [-c] [--overwrite] [-r PATH] name

Positional Arguments¶

name

Create the structure for the named project, and copy example: XML files into the “etc” directory.

Named Arguments¶

`-c, --addToConfig`
	Add a section for the new project to $HOME/.pygcam.cfg after making a backup of the file in $HOME/.pygcam.cfg~ Default: False
`--overwrite`	If files that are to be copied to the project directory exist, overwrite them. By default, existing files are not overwritten. Default: False
`-r, --projectRoot`
	The directory in which to create a subdirectory for the named project. Default is the value of config variable GCAM.ProjectRoot

protect¶

Generate versions of GCAM’s land_input XML files that protect a given fraction of land of the given land types in the given regions by subtracting the required land area from the “managed” land classes, thereby removing them from consideration in land allocations.

Simple protection scenarios can be specified on the command-line. More complex scenarios can be specified in an XML file, landProtection.xml.

Examples:

# Create and modify copies of the reference land files, renaming them with
# "prot\_" prefix. Protect 80% of the "UnmanagedForest" and "UnmanagedPasture"
# land classes in the specified regions only.

CLASSES=UnmanagedForest,UnmanagedPasture
REGIONS='Australia_NZ,Canada,EU-12,EU-15,Japan,Middle East,Taiwan,USA'
OUTDIR="$HOME/tmp/xml"

gt protect -f 0.8 "$INFILES" -l "$CLASSES" -r "$REGIONS" -o "$OUTDIR" -t 'prot_{filename}'

# Run the land protection scenario "s1", described in the file ``$HOME/protect.xml``,
# placing the results in the directory ``$HOME/ws/workspace1``

gt protect -s s1 -S "$HOME/protect.xml" -w "$HOME/ws/workspace1"

gt protect [-h] [-b] [-f FRACTION] [--inPlace] [-l LANDCLASSES] [-m]
           [-o OUTDIR] [-O] [-t TEMPLATE] [-r REGIONS] [-s SCENARIO]
           [-S SCENARIOFILE] [-w WORKSPACE]

Named Arguments¶

`-b, --backup`	Make a copy of the output file, if it exists (with an added ~ after filename) before writing new output. This option is ignored if a scenario file is specified. Default: False
`-f, --fraction`	The fraction of land in the given land classes to protect. (Required, unless a scenario file is specified, in which case this option is ignored.)
`--inPlace`	Edit the file in place. This must be given explicitly, to avoid overwriting files by mistake. Default: False
`-l, --landClasses`
	The land class or classes to protect in the given regions. Multiple, comma-delimited land types can be given in a single argument, or the -l flag can be repeated to indicate additional land classes. By default, all unmanaged land classes are protected. Allowed land classes are [‘UnmanagedPasture’, ‘UnmanagedForest’, ‘Shrubland’, ‘Grassland’]. This option is ignored if a scenario file is specified.
`-m, --mkdir`	Make the output dir if necessary. Default: False
`-o, --outDir`	The directory into which to write the modified files. Default is current directory. Default: “.”
`-O, --otherArable`
	Include OtherArableLand in the list of default land classes to protect. This flag is ignored if the -l (–landClasses) argument is used. Default: False
`-t, --template`	Specify a template to use for output filenames. The keywords {fraction}, {filename}, {regions}, and {classes} (with surrounding curly braces) are replaced by the following values and used to form the name of the output files, written to the given output directory. fraction: 100 times the given fraction (i.e., int(fraction * 100)); filename: the name of the input file being processed (e.g., land_input_2.xml or land_input_3.xml); basename: the portion of the input filename prior to the extension (i.e., before ‘.xml’); regions: the given regions, separated by ‘-‘, or the word ‘global’ if no regions are specified; classes: the given land classes, separated by ‘-‘, or the word ‘unmanaged’ if no land classes are specified. The default pattern is “prot_{fraction}_{filename}”. This option is ignored if a scenario file is specified. Default: “prot_{fraction}_{filename}”
`-r, --regions`	The region or regions for which to protect land. Multiple, comma-delimited regions can be given in a single argument, or the -r flag can be repeated to indicate additional regions. By default, all regions are protected. This option is ignored if a scenario file is specified.
`-s, --scenario`	The name of a land-protection scenario defined in the file given by the –scenarioFile argument or it’s default value.
`-S, --scenarioFile`
	An XML file defining land-protection scenarios. Default is the value of configuration file parameter GCAM.LandProtectionXmlFile.
`-w, --workspace`
	Specify the path to the GCAM workspace to use. The files in {workspace}/input/gcam-data-system/xml/aglu-xml/land_input_{2,3}.xml (before GCAM v5.1), or {workspace}/input/gcamdata/xml/land_input_{2,3,4,5}*.xml (starting in GCAM v5.1) are used as inputs. Default is value of configuration parameter GCAM.RefWorkspace.

query¶

Run one or more GCAM database queries by generating and running the named XML queries. The results are placed in a file in the specified output directory with a name composed of the basename of the XML query file plus the scenario name. For example,

gt query -o. -s MyReference,MyPolicyCase liquids-by-region

would run the liquids-by-region query on two scenarios, MyReference and MyPolicyCase. Query results will be stored in the files ./liquids-by-region-MyReference.csv and ./liquids-by-region-MyPolicyCase.csv.

The named queries are located using the value of config variable GCAM.QueryPath, which can be overridden with the -Q argument. The QueryPath consists of one or more colon-delimited (on Unix) or semicolon-delimited (on Windows) elements that can identify directories or XML files. The elements of QueryPath are searched in order until the named query is found. If a path element is a directory, the filename composed of the query + ‘.xml’ is sought in that directory. If the path element is an XML file, a query with a title matching the query name (first literally, then by replacing '_' and '-' characters with spaces) is sought. Note that query names are case-sensitive.

gt query [-h] [-b BATCHFILE] [-B BATCHOUTPUT] [-d XMLDB] [-D] [-g GROUPDIR]
         [-n] [-o OUTPUTDIR] [-p] [-q QUERYXMLFILE] [-Q QUERYPATH]
         [-r REGIONS] [-R REGIONMAP] [-s SCENARIO] [-S REWRITESETSFILE]
         [-w WORKSPACE]
         [queryName [queryName ...]]

Positional Arguments¶

queryName

A file or files, each holding an XML query to run. (The “.xml” suffix will be: added if needed.) If an argument is preceded by the “@” sign, it is read and its contents substituted as the values for this argument. That means you can store queries to run in a file (one per line) and just reference the file by preceding the filename argument with “@”.

Named Arguments¶

`-b, --batchFile`
	An XML batch file to run. The file will typically contain multiple queries. By default, output is written to {outputDir}/{batchFile basename}.csv. Use ‘-B’ to change this.
`-B, --batchOutput`
	Where to write the output of the XML batch file given by the ‘-b’ flag. Non-absolute paths are treated as relative to the given outputDir. Default: “”
`-d, --xmldb`	The XML database to query (default is computed as {GCAM.SandboxDir}/output/{GCAM.DbFile}. Overrides the -w flag.
`-D, --noDelete`	Don’t delete any temporary file created by extracting a query from a query file. Used mainly for debugging. Default: False
`-g, --groupDir`	The scenario group directory name, if any. Used with to compute default for –workspace argument. Default: “”
`-n, --noRun`	Show the command to be run, but don’t run it Default: False
`-o, --outputDir`
	Where to output the result (default taken from config parameter “GCAM.OutputDir”)
`-p, --prequery`	Generate the XMLDBDriver.properties file and associated batch file to be run by GCAM when GCAM.BatchMultipleQueries or GCAM.InMemoryDatabase are True. Default: False
`-q, --queryXmlFile`
	An XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml.
`-Q, --queryPath`
	A semicolon-delimited list of directories or filenames to look in to find query files. Defaults to value of config parameter GCAM.QueryPath
`-r, --regions`	A comma-separated list of regions on which to run queries found in query files structured like Main_Queries.xml. If not specified, defaults to querying all 32 regions.
`-R, --regionMap`
	A file containing tab-separated pairs of names, the first being a GCAM region and the second being the name to map this region to. Lines starting with “#” are treated as comments. Lines without a tab character are also ignored. This arg overrides the value of config variable GCAM.RegionMapFile.
`-s, --scenario`	The scenario to run the query/queries for (default is “Reference”) Note that this must refers to a scenarios in the XML database. Default: “Reference”
`-S, --rewriteSetsFile`
	An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”)
`-w, --workspace`
	The workspace directory in which to find the XML database. Defaults computed as {GCAM.SandboxDir}/{groupDir}/{scenario}. Overridden by the -d flag. Default: “”

res¶

Generate an XML file that implements a RES policy on the electricity: sector as described in the given XML input file.

gt res [-h] [-i INPUTFILE] [-o OUTPUTXML] [-S SCENARIO] [-d] [-u]

Named Arguments¶

`-i, --inputFile`
	A CSV or XML file defining the RES policy. Default is the value of configuration file parameter GCAM.RESDescriptionFile. If set to a relative pathname (i.e., not starting with “/”, “”, or drive specifier “[a-zA-Z]:”), it is assumed to be relative to %(GCAM.ProjectDir)s/etc. If a CSV file is given, it is converted to an intermediate RES policy XML file before translation to GCAM-readable input.
`-o, --outputXML`
	The directory into which to write the modified files. Default is the value of configuration file parameter GCAM.RESImplementationXmlFile. If set to a relative pathname, it is assumed to be relative to %(GCAM.SandboxRefWorkspace)s/local-xml/{scenario}, in which case, the “-s/–scenario” argument is required.
`-S, --scenario`	The name of the scenario for which to generate the policy implementation XML file. Required if no argument is given to the “-o/–outputXML” flag, or if the argument is a relative pathname.
`-d, --display`	If set, the result of the RES policy is displayed in tabular format, and the program exits. Default: False
`-u, --GCAM-USA`	If set, produce output compatible with GCAM-USA regions. Default: False

run¶

This sub-command reads instructions from the file project.xml, the location of which is taken from the user’s ~/.pygcam.cfg file. The workflow steps indicated in the XML file and command-line determine which commands to run.

Examples:

Run all steps for the default scenario group for project ‘Foo’:

gt +P Foo run

Run all steps for scenario group ‘test’ for project ‘Foo’, but only for scenarios ‘baseline’ and ‘policy-1’:

gt +P Foo run -g test -S baseline,policy1

or, equivalently:

gt +P Foo run --group test --scenario baseline --step policy1

Run only the ‘setup’ and ‘gcam’ steps for scenario ‘baseline’ in the default scenario group:

gt +P Foo run -s setup,gcam -S baseline,policy-1

Same as above, but queue a batch job to run these commands on the queue ‘short’:

gt +b +q short +P Foo run -s setup,gcam -S baseline,policy-1

Note that the command above will run the two scenarios (‘baseline’ and ‘policy-1’) in a single batch job. To run scenarios in separate batch jobs, use the -D or --distribute option to the run sub-commmand:

gt +q short +P Foo run -D -S baseline,policy-1

The “distribute” option knows that various project steps for non-baseline scenarios may depend on baseline scenarios, so the baseline is always run first, with the non-baseline scenarios queued as dependent on the successful completion of the baseline. If no scenarios are explicitly named, all scenarios in the group are run, as usual.

The -n flag displays the commands that would be executed for a command, but doesn’t run them:

gt +P Foo run -s setup,gcam -S baseline,policy-1 -n

gt run [-h] [-a] [-D] [-f PROJECTFILE] [-g GROUP] [-G] [-k SKIPSTEPS]
       [-K SKIPSCENARIOS] [-l] [-L] [-n] [-q] [-s STEPS] [-S SCENARIOS]
       [--vars] [-x SANDBOXDIR]

Named Arguments¶

`-a, --allGroups`
	Run all scenarios for all defined groups. Default: False
`-D, --distribute`
	Run the given scenarios by queueing them independently. If one of the scenarios is a baseline, it is queued first and the remaining scenarios are queued as dependent on the completion of the baseline job. Default: False
`-f, --projectFile`
	The XML file describing the project. If set, command-line argument takes precedence. Otherwise, value is taken from config file variable GCAM.ProjectXmlFile, if defined, otherwise the default is ‘./project.xml’.
`-g, --group`	The name of the scenario group to process. If not specified, the group with attribute default=”1” is processed.
`-G, --listGroups`
	List the scenario groups defined in the project file and exit. Default: False
`-k, --skipStep`	Steps to skip. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -k flag can be repeated to indicate additional steps. By default, all steps are run.
`-K, --skipScenario`
	Scenarios to skip. Multiple scenarios can be given in a single (comma-delimited) argument, or the -K flag can be repeated to indicate additional scenarios. By default, all scenarios are run.
`-l, --listSteps`
	List the steps defined for the given project and exit. Dynamic variables (created at run-time) are not displayed. Default: False
`-L, --listScenarios`
	List the scenarios defined for the given project and exit. Dynamic variables (created at run-time) are not displayed. Default: False
`-n, --noRun`	Display the commands that would be run, but don’t run them. Default: False
`-q, --noQuit`	Don’t quit if an error occurs when processing a scenario, just move on to processing the next scenario, if any. Default: False
`-s, --step`	The steps to run. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -s flag can be repeated to indicate additional steps. By default, all steps are run.
`-S, --scenario`	Which of the scenarios defined for the given project should be run. Multiple scenarios can be given in a single (comma-delimited) argument, or the -S flag can be repeated to indicate additional scenarios. By default, all active scenarios are run.
`--vars`	List variables and their values Default: False
`-x, --sandboxDir`
	The directory in which to create the run-time sandbox workspace. Defaults to value of {GCAM.SandboxProjectDir}/{scenarioGroup}.

sandbox¶

The sandbox sub-command allows you to create, delete, show the path of, or run a shell command in a workspace. If the --scenario argument is given, the operation is performed on a scenario-specific workspace within a project directory. If --scenario is not specified, the operation is performed on the project directory that contains individual scenario workspaces. Note that the gcam sub-command automatically creates workspaces as needed.

N.B. You can run sandbox with the --path option before performing any operations to be sure of the directory that will be operated on, or use the --noExecute option to show the command that would be executed by --run.

gt sandbox [-h] [--create] [--delete] [--recreate] [-g NAME] [-n] [-p]
           [-r CMD] [-s SCENARIO]

Named Arguments¶

`--create`	Create the identified sandbox. If used with –delete, the deletion occurs first. Default: False
`--delete`	Delete the identified sandbox’ If used with –create, the deletion occurs first. Default: False
`--recreate`	Recreate the identified sandbox. Equivalent to using the –delete and –create options together. Default: False
`-g, --groupDir`	The name of the scenario group subdir Default: “”
`-n, --noExecute`
	Print the command that would be executed by –run, but don’t execute it. Default: False
`-p, --path`	Print the absolute path to the identified sandbox. Default: False
`-r, --run`	Run the given command in the identified sandbox.
`-s, --scenario`	The scenario for the computed sandbox root. Default: “”

setup¶

The setup sub-command automates modification to copies of GCAM’s input XML files and construction of a corresponding configuration XML file. See GCAM XML-Setup for a detailed description.

gt setup [-h] [-b BASELINE] [-d] [-f] [-g GROUP] [-G SRCGROUPDIR]
         [-m MODULEPATH] [-M MODULESPEC] [-p period-or-year] [-r REFWORKSPACE]
         [-R RESULTSDIR] [-s SCENARIO] [-S SUBDIR] [--setupXml SETUPXML] [-T]
         [-u] [-x XMLSOURCEDIR] [-X XMLOUTPUTROOT] [-w WORKSPACE] [-y YEARS]

Named Arguments¶

`-b, --baseline`	Identify the baseline the selected scenario is based on. Note: at least one of –baseline (-b) / –scenario (-s) must be used.
`-d, --dynamicOnly`
	Generate only dynamic XML for dyn-xml: don’t create static XML. Default: False
`-f, --forceCreate`
	Re-create the workspace, even if it already exists. Default: False
`-g, --group`	The scenario group to process. Defaults to the group labeled default=”1”.
`-G, --srcGroupDir`
	A sub-directory under xmlsrc in which to find scenario dirs for this group. Use this to consolidate static XML files shared by multiple scenario groups. If –useGroupDir is specified, srcGroupDir defaults to the scenario group name. Using –srcGroupDir implies –useGroupDir.
`-m, --modulePath`
	The path to a scenario definition module. See -M flag for more info.
`-M, --moduleSpec`
	The “dot spec” for the Python module holding the setup classes and a function called ‘scenarioMapper’ or a dictionary called ‘ClassMap’ which map scenario names to classes. If the function ‘scenarioMapper’ exists, it is used. If not, the ‘ClassMap’ is used. Default is “{xmlsrc}/subdir/scenarios.py” (if subdir is defined) or “{xmlsrc}/scenarios.py” (if subdir is undefined) under the current ProjectRoot.
`-p, --stop`	The number of the GCAM period or the year to stop after
`-r, --refWorkspace`
	A reference workspace to use instead of the value of GCAM.RefWorkspace Default: “”
`-R, --resultsDir`
	The parent directory holding the GCAM output workspaces
`-s, --scenario`	Identify the scenario to run. Note: at least one of –baseline (-b) / –scenario (-s) must be used.
`-S, --subdir`	A sub-directory to use instead of scenario name Default: “”
`--setupXml`	An XML scenario definition file. Overrides configuration variable GCAM.ScenarioSetupFile.
`-T, --staticOnly`
	Generate only static XML for local-xml: don’t create dynamic XML. Default: False
`-u, --useGroupDir`
	Use the group name as a sub directory below xmlsrc, local-xml, and dyn-xml Default: False
`-x, --xmlSourceDir`
	The location of the xmlsrc directory.
`-X, --xmlOutputRoot`
	The root directory into which to generate XML files.
`-w, --workspace`
	The pathname of the workspace to operate on.
`-y, --years`	Years to generate constraints for. Must be of the form XXXX-YYYY. Default is “2015-2100” Default: “2015-2100”

Extending gt using plug-ins¶

The gt script will load any python files whose name ends in _plugin.py, found in any of the directories indicated in the config file variable GCAM.PluginPath. The value of GCAM.PluginPath must be a sequence of directory names separated by colons (:) on Unix-like systems or by semi-colons (;) on Windows.

See pygcam.subcommand for documentation of the plug-in API.

GCAM tool (gt)¶

Usage¶

Named Arguments¶

Subcommands¶

Sub-commands:¶

building¶

Named Arguments¶

chart¶

Positional Arguments¶

Named Arguments¶

compare¶

Positional Arguments¶

Named Arguments¶

config¶

Positional Arguments¶

Named Arguments¶

diff¶

Positional Arguments¶

Named Arguments¶

gcam¶

Named Arguments¶

gui¶

Named Arguments¶

init¶

Named Arguments¶

mcs¶

Positional Arguments¶

mi¶

Named Arguments¶

new¶

Positional Arguments¶

Named Arguments¶

protect¶

Named Arguments¶

query¶

Positional Arguments¶

Named Arguments¶

res¶

Named Arguments¶

run¶

Named Arguments¶

sandbox¶

Named Arguments¶

setup¶

Named Arguments¶

Extending gt using plug-ins¶

GCAM tool (`gt`)¶