Configuration System¶
The pygcam
scripts and libraries rely on a configuration file to:
- define the location of essential and optional files,
- allow the user to set defaults for many command-line arguments to scripts, and
- define both global default and project-specific values for all parameters.
The configuration file and variables are described below.
See also
Use the gt init command to initialize your configuration file.
Usage of the config
sub-command is described on the gt config
page. See pygcam.config for documentation of the API to the configuration system.
pygcam
uses the Python ConfigParser
package. See the documentation
there for more details.
Configuration file sections¶
The configuration file is divided into sections indicated by a name within square brackets. All variable declarations following a section declaration, until the next section declaration (if any) appear in the declared section.
Default section¶
Default values are defined in the [DEFAULT]
section. When pygcam
requests the value
of a variable from a project section (see below), the default value is returned if the
variable is not defined in the project section. Variables whose values apply to multiple
projects can be defined conveniently in the [DEFAULT]
section.
All pre-defined pygcam
variables are defined in the [DEFAULT]
section,
allowing them to be overridden on a project-by-project basis.
Project sections¶
Each project must have its own section. For example, to setup a project called,
say, “myproj”, I would create the section [myproj]
. Following this, I would define
variables particular to this project, e.g., where to find the files defining scenarios,
queries, and so on.
Note that the new sub-command will set up the structure for a new project and (optionally) add a section to the user’s config file for the named project.
The configuration files¶
There are up to 4 configuration files read, two of which are user-modifiable:
- First,
pygcam/etc/system.cfg
is read from within thepygcam
package. This defines all known config variables and provides their default values as described below. The values in this file are the appropriate values for Linux and similar systems. This file should not be modified by the user.- Next, a platform-specific file is read, if it exists. Currently, the only such files are
pygcam/etc/Windows.cfg
andpygcam/etc/Darwin.cfg
, read on Windows and Macintosh systems, respectively. (N.B. “Darwin” is the official platform name for the Macintosh operating system.) These files should not be modified by the user.- Next, if the environment variable
PYGCAM_SITE_CONFIG
is defined, it should refer to a configuration file that defines site-specific settings. This file is optional; it allows an administrator to consolidate site-specific values to simplify configuration for users.- Finally, the user’s configuration file is read if it exists; otherwise the file is created with the initial contents being a commented-out version of
pygcam/etc/system.cfg
. This provides a handy reference to the available parameters and their default values.
- On Linux and OS X, the user’s configuration file is found in
$HOME/.pygcam.cfg
- On Windows, the file
.pygcam.cfg
will be stored in the directory identified by the first of the following environment variables defined to have a non-empty value:PYGCAM_HOME
,HOMESHARE
, andHOMEPATH
. The first variable,PYGCAM_HOME
is known only to pygcam, while at least one of the other two should be set by Windows.- In all cases, the directory in which the configuration file is located is assigned to the pygcam configuration variable
Home
.
The values in each successive configuration file override default values for
variables of the same name that are set in files read earlier. Values can also be set in
project-specific sections whose names should match project names defined in the
project.xml file. Thus when a user specifies a project to operate on, either on the
command-line to GCAM tool (gt) or as the value of GCAM.DefaultProject
in $HOME/.pygcam.cfg
,
the project-specific values override any values set in [DEFAULT]
sections.
For example, consider the following values in $HOME/.pygcam.cfg
:
[DEFAULT]
GCAM.RefWorkspace = %(Home)s/GCAM/gcam-v4.3
[Project1]
GCAM.RefWorkspace = /other/location/GCAM/gcam-v4.4
[OtherProject]
# no value set here for GCAM.RefWorkspace
The default value for GCAM.RefWorkspace
is %(Home)s/GCAM/gcam-v4.3
. This value is
used for the project OtherProject
since no project-specific value is defined, but the project
Project1
overrides this with the value /other/location/GCAM/gcam-v4.4
.
The available parameters and their default values are described below.
Editing the user configuration file¶
You can edit the configuration file, $HOME/.pygcam.cfg
, with any editor capable of
working with plain text, i.e., not a word-processor such as Word. Use the command
gt config -e
to invoke an editor on the configuration file.
The command invoked by gt config -e
to edit the config file is the value of the
configuration parameter GCAM.TextEditor
, which defaults to a system-appropriate
value shown in the table below. Set this value in the configuration file to invoke
your preferred editor.
For example, if you prefer the emacs
editor on a Mac, you can add this line to
~/.pygcam.cfg
to cause the Finder to open the file using the emacs application:
GCAM.TextEditor = open -a emacs
Or, to edit the config file using the PyCharm app, use this:
GCAM.TextEditor = open -a PyCharm
If the editor command is not found on your execution PATH
, you can specify the
full pathname. Use quotes around the path if it includes spaces, as in the examples
below.
To use Notepad++ on Windows, use the following (adjusted as necessary for your installation location):
GCAM.TextEditor = "C:/Program Files/Notepad++/notepad++.exe"
To use PyCharm, use the following – again, adjusted to match your installation location:
GCAM.TextEditor = "C:/Program Files/JetBrains/PyCharm 2018.1.4/bin/pycharm64.exe"
Invoking the command:
gt config -e
will cause the editor to be invoked on your configuration file.
Referencing configuration variables¶
A powerful feature of the configuration system is that variables can be defined in
terms of other variables. The syntax for referencing the value of a variable is to
precede the variable name with %(
and follow it with )s
. Thus to reference
variable GCAM.QueryDir
, you would write %(GCAM.QueryDir)s
.
Note
When referencing a variable in the config file, you must include the
trailing s
after the closing parenthesis, or a Python exception will be raised.
Also note that variable names are case-sensitive.
Variable values are substituted when a variable’s value is requested, not
when the configuration file is read. The difference is that if variable A
is
defined in terms of variable B
, (e.g., A = %(B)s/something/else
), you can
subsequently change B
and the value of A
will reflect this when A
is
accessed by pygcam
.
All known variables are given default values in the pygcam system files. Users can create variables in any of the user controlled config files, if desired.
Environment variables¶
All defined environmental variables are loaded into the config parameter space before
reading any configuration files, and are accessible with a prefix of $
, as in a
UNIX shell. For example, to reference the environment variable SCRATCH
, you can
use %($SCRATCH)s
.
Validating configuration settings¶
pygcam
requires that certain configuration variables be set. The table
below shows key variables, indicating whether they are required or optional,
and whether their value must be a file or directory.
Variable name | Required | Type |
---|---|---|
GCAM.SandboxRoot | yes | directory |
GCAM.ProjectRoot | yes | directory |
GCAM.QueryDir | yes | directory |
GCAM.MI.Dir | yes | directory |
GCAM.RefWorkspace | yes | directory |
GCAM.TempDir | yes | directory |
GCAM.ProjectXmlFile | yes | file |
GCAM.RefConfigFile | yes | file |
GCAM.MI.JarFile | yes | file |
GCAM.UserTempDir | no | directory |
GCAM.RegionMapFile | no | file |
GCAM.RewriteSetsFile | no | file |
The config
sub-command provides a limited amount of validation by checking
that all required and optional variables are set to reasonable values. You can
do a basic (not foolproof) check that the required files and directories exist
using the command:
gt config -t
which will print out a listing of files and their status.
You can also specify a project to check that project’s variables. For example,
I can test the values set for project myproj
with the following command,
shown with command output:
$ gt +P myproj config -t
OK: GCAM.SandboxRoot = /people/plev920/ws
OK: GCAM.SandboxDir = /people/plev920/ws/myproj/
OK: GCAM.ProjectRoot = /people/plev920/bitbucket
OK: GCAM.ProjectDir = /people/plev920/bitbucket/myproj
OK: GCAM.QueryDir = /people/plev920/bitbucket/myproj/queries
OK: GCAM.MI.Dir = /people/plev920/GCAM/current/input/gcam-data-system/_common/ModelInterface/src
OK: GCAM.RefWorkspace = /people/plev920/GCAM/current
OK: GCAM.TempDir = /pic/scratch/plev920/tmp
OK: GCAM.UserTempDir = /people/plev920/tmp
OK: GCAM.ProjectXmlFile = /people/plev920/bitbucket/myproj/etc/project.xml
OK: GCAM.RefConfigFile = /people/plev920/GCAM/current/exe/configuration_ref.xml
OK: GCAM.MI.JarFile = /people/plev920/GCAM/current/input/gcam-data-system/_common/ModelInterface/src/ModelInterface.jar
OK: GCAM.RewriteSetsFile = /people/plev920/bitbucket/myproj/etc/rewriteSets.xml
Location of GCAM program and data files¶
The configuration variable GCAM.RefWorkspace
must point to a directory
structured like the standard GCAM workspace, with sub-directories
for input
, output
, libs
, and exe
. These files are the reference
files used by GCAM tool (gt) to set up “sandbox” workspaces in which to run GCAM.
pygcam
does not run GCAM in the reference workspace’s exe
directory;
it uses the files there to create new workspaces as required. Creating separate
workspaces for each scenario allows multiple scenarios to be run simultaneously
without contention for the XML database which is created at the end of the model
run. This is essential when running on a computing cluster.
The variable GCAM.MI.Dir
should point to a directory holding the ModelInterface
program. This is used to execute batch queries to extract results from GCAM. By
default, this location is computed from the GCAM.RefWorkspace
, but you can
change it if necessary, e.g., if you’re using a customized version of ModelInterface.
Default configuration variables and values¶
The system default values are provided in the pygcam
package in the file
pygcam/etc/system.cfg
, which is listed below. In addition to these values,
several values are read from platform-specific files, as noted above. These
values are shown below.
For Windows:¶
# Windows-specific default values
[DEFAULT]
# Windows requires semi-colon path-delimiter
PATHSEP = ;
GCAM.Executable = Objects-Main.exe
GCAM.TextEditor = notepad.exe
GCAM.TempDir = %(Home)s/temp
# Add this DLL for Windows
GCAM.SysRequiredFiles = exe/xerces-c_3_1.dll libs/jars
# Add "libs" for Windows
GCAM.WorkspaceFilesToLink = %(GCAM.InputFiles)s libs/jars
GCAM.SandboxFilesToLink = %(GCAM.InputFiles)s libs/jars exe/%(GCAM.Executable)s
# Set this to False to suppress the warning message
GCAM.SymlinkWarning = True
For MacOS:¶
# Macintosh-specific default values
[DEFAULT]
GCAM.Executable = Release/objects
GCAM.TextEditor = open -e
# Additional required files for Mac
GCAM.SysRequiredFiles = libs
# Add "libs" for the Mac
GCAM.WorkspaceFilesToLink = %(GCAM.InputFiles)s libs
GCAM.SandboxFilesToLink = %(GCAM.InputFiles)s libs exe/%(GCAM.Executable)s
Default configuration variable dependencies¶
The following figure shows variable dependencies according to the default definitions. Variables lower in the figure depend on those above them. Thus, if you change a variable with “descendants”, you affect the definition of everything below it in the figure.
The system defaults file¶
[DEFAULT]
GCAM.VersionNumber = 5.1.3
# This project is used if '-P' flag not given to the 'gt' command
GCAM.DefaultProject =
# Linux default is defined here. Can be overridden in
# platform-specific config file (e.g., etc/Darwin.cfg,
# etc/Windows.cfg).
GCAM.Executable = gcam.exe
# v5.1 changed gcam-data-system to gcamdata, but we don't know which
# version a project is using until run-time, so we compute this based
# on the version number set by the project.
GCAM.DataDir =
# This defines the variable for documentation purposes. The value
# is set automatically in each project section to the name of that
# section -- unless a non-blank value already exists in the section.
GCAM.ProjectName =
# Root directory for where the user keeps project folders
GCAM.ProjectRoot = %(Home)s/projects
GCAM.ProjectDir = %(GCAM.ProjectRoot)s/%(GCAM.ProjectName)s
# Where to find plug-ins. Internal plugin directory is added
# automatically. Use this to add custom plug-ins outside the pygcam
# source tree. The value is a semicolon-delimited (on Windows) or
# colon-delimited (on Unix) string of directories to search for files
# matching the pattern '*_plugin.py'
GCAM.PluginPath = %(GCAM.ProjectDir)s/plugins
# The location of the GCAM installation to use.
GCAM.RefWorkspace = %(Home)s/GCAM/gcam-v%(GCAM.VersionNumber)s
# Allows user to decide where to create the lockfile, depending on
# the characteristics of the available file systems. By default it
# is created in the newly created copy of the Workspace dir.
GCAM.WorkspaceSemaphoreDir =
# Files to copy or link when creating workspaces and sandboxes. Starting
# with v4.3, many more files appear under input that we do not want to copy.
# Note that input/gcamdata/_common is required since in-memory db
# queries set class path to reference the ModelInterface.jar found under this.
GCAM.InputFiles.all = input/climate input/magicc input/policy input/solution input/%(GCAM.DataDir)s/xml
GCAM.InputFiles.4.3 = %(GCAM.InputFiles.all)s input/gcam-data-system/_common
GCAM.InputFiles.4.4 = %(GCAM.InputFiles.all)s input/gcam-data-system/_common
GCAM.InputFiles.5.0 = %(GCAM.InputFiles.all)s input/gcam-data-system/_common
GCAM.InputFiles.5.1 = %(GCAM.InputFiles.all)s exe/restart
# Placeholder until 5.2 is released. It *might* just work...
GCAM.InputFiles.5.2 = %(GCAM.InputFiles.all)s exe/restart
# Set at run-time based on the current project's GCAM version number
GCAM.InputFiles =
# Allow user to add to the list
User.RequiredFiles =
# Hook for Windows and Mac to specify additional files (e.g., Windows DLLs)
GCAM.SysRequiredFiles =
GCAM.RequiredFiles = %(User.RequiredFiles)s exe/%(GCAM.Executable)s %(GCAM.InputFiles)s exe/log_conf.xml exe/XMLDBDriver.jar %(GCAM.SysRequiredFiles)s
# Files to link from the reference workspace to run-time workspace.
# If linking fails on Windows or GCAM.CopyAllFiles = True, files are
# copied instead. Note that unrecognized files here are ignored, with
# a WARNING message identifying them.
GCAM.WorkspaceFilesToLink = %(GCAM.InputFiles)s
# Same as above, for files to link from the run-time workspace to sandboxes.
# See OS-specific files for possible override.
GCAM.SandboxFilesToLink = %(GCAM.InputFiles)s exe/%(GCAM.Executable)s
# The reference config file to use as a starting point for "setup"
GCAM.RefConfigFile = %(GCAM.RefWorkspace)s/exe/configuration_ref.xml
# The path separator defaults to ":" but is overridden in Windows.cfg
# to be ";" since ":" appears after drive specifiers in Windows paths.
PATHSEP = :
# QueryPath is string with one or more colon-delimited elements that
# identify directories or XML files in which to find batch query
# definitions.
GCAM.QueryDir = %(GCAM.ProjectDir)s/queries
GCAM.QueryPath = %(GCAM.QueryDir)s%(PATHSEP)s%(GCAM.RefWorkspace)s/output/queries/Main_queries.xml
# File that defines query rewrites by name for use by query command.
# GCAM.RewriteSetsFile = %(GCAM.ProjectDir)s/etc/rewriteSets.xml
GCAM.RewriteSetsFile =
# Where query output is written if no output dir is specified on
# command-line to "query" sub-command.
GCAM.OutputDir = %(Home)s/tmp
# The location of GCAM source code (for the purpose of reading
# the .csv file that defines the current regional aggregation.
GCAM.SourceWorkspace =
# If using the XML "setup" system, this is the root folder for
# setup source files
GCAM.XmlSrc = %(GCAM.ProjectDir)s/xmlsrc
# The default input file for the runProj sub-command
GCAM.ProjectXmlFile = %(GCAM.ProjectDir)s/etc/project.xml
# Whether GCAM should generate a debug file (no value => no change)
GCAM.WriteDebugFile =
# Whether GCAM should generate a price file (no value => no change)
GCAM.WritePrices =
# Whether GCAM should generate the large XML file with the combined data
# from all input files. (no value => no change)
GCAM.WriteXmlOutputFile =
# Whether GCAM should generate outFile.csv (no value => no change)
GCAM.WriteOutputCsv = False
# Whether to write out restart files
GCAM.WriteRestartFiles = True
# Path to an XML file describing land protection scenarios
GCAM.LandProtectionXmlFile =
# Path to an XML file describing an RES policy, an input to the "res" sub-command
# If it's a relative path, it's treated as relative to %(GCAM.ProjectDir)s/etc.
GCAM.RESDescriptionXmlFile = RES_description.xml
# Path to an XML file that implements an RES policy, the output of the "res"
# sub-command. If it's a relative path, it's treated as relative to
# %(GCAM.SandboxRefWorkspace)s/local-xml/{scenario}, where the scenario must
# be given as an argument to the "res" sub-command.
GCAM.RESImplementationXmlFile = RES_implementation.xml
# Default location in which to look for scenario directories
GCAM.ScenariosDir =
# The pathname of the XML scenario setup file.
GCAM.ScenarioSetupFile = %(GCAM.ProjectDir)s/etc/scenarios.xml
# Where to save expanded XML when using "iterators" in XML setup.
# This is optional and provide to aid in debugging setups.
GCAM.ScenarioSetupOutputFile =
# Set this to identify the subclass of XMLEditor to use as a
# superclass to generate a class to process your XML-based
# scenario setup. Uses XMLEditor as a superclass by default.
# The format of this option is:
# {path to module directory}:{module.dot.specification}, e.g.
#
# GCAM.ScenarioSetupClass = %(Home)s/somewhere/pygcam:pygcam.sectorEditors.BioenergyEditor
#
GCAM.ScenarioSetupClass =
# ModelInterface directory for the version to use. This is now set dynamically in
# tool.py after the project is set, so we know which version of GCAM is in use. The
# subdir is 'input/gcam-data-system/_common/ModelInterface/src' through v5.0. Starting
# with 5.1 it is 'output/modelinterface'.
GCAM.MI.Subdir =
GCAM.MI.Dir = %(GCAM.RefWorkspace)s/%(GCAM.MI.Subdir)s
GCAM.MI.JarFile = %(GCAM.MI.Dir)s/ModelInterface.jar
# This is set dynamically to True on Linux (deprecated for GCAM 4.3)
GCAM.MI.UseVirtualBuffer = False
# The location of the libraries needed by ModelInterface. This is correct for v4.3 and v4.4.
GCAM.MI.ClassPath = %(GCAM.RefWorkspace)s/libs/jars/*%(PATHSEP)s%(GCAM.MI.JarFile)s%(PATHSEP)sXMLDBDriver.jar
# Arguments to java to ensure that ModelInterface has enough heap space.
GCAM.MI.JavaArgs = -Xms512m -Xmx2g
# Command to run ModelInterface interactively
GCAM.MI.Command = java %(GCAM.MI.JavaArgs)s -cp %(GCAM.MI.ClassPath)s ModelInterface/InterfaceMain
# Command to run batch queries
GCAM.MI.BatchCommand = %(GCAM.MI.Command)s -b "{batchFile}"
# Query file to use for interactive use of ModelInterface. If this file
# doesn't exist, GCAM.MI.RefQueryFile is used.
GCAM.MI.QueryFile = %(GCAM.QueryDir)s/Main_Queries.xml
# The query file to use if GCAM.MI.QueryFile does not point to an actual file
GCAM.MI.RefQueryFile = %(GCAM.RefWorkspace)s/output/queries/Main_queries.xml
# Name of log file to catch verbose ModelInterface output. If specified
# as a relative path, is interpreted as relative to the batch output dir.
# An absolute pathname is used as is. Set this to /dev/null on Unix to
# trash the output.
GCAM.MI.LogFile = mi.log
# The name of the database file (or directory, for BaseX)
GCAM.DbFile = database_basexdb
# Columns to drop when processing results of XML batch queries
GCAM.ColumnsToDrop = scenario,Notes,Date
# Change this if desired to increase or decrease diagnostic messages.
# A default value can be set here, and a project-specific value can
# be set in the project's config file section.
# Possible values (from most to least verbose) are:
# DEBUG, INFO, WARNING, ERROR, CRITICAL
GCAM.LogLevel = INFO
# The default location in which to find or create GCAM runtime sandboxes
GCAM.SandboxRoot = %(Home)s/GCAM/sandbox
GCAM.SandboxProjectDir = %(GCAM.SandboxRoot)s/%(GCAM.ProjectName)s
# Identifies the location of the workspace copy used to create
# new sandboxes. The workspace is created on demand.
GCAM.SandboxRefWorkspace = %(GCAM.SandboxProjectDir)s/Workspace
# N.B. These are set at run-time in project.py and are available
# with the "run" sub-command. SandboxDir is the directory under
# which sandboxes are created
# ScenarioGroup is set in project.py if <ScenarioGroup> sets
# useGroupDir="1". Thus the variable is available only via the
# run sub-command.
GCAM.ScenarioGroup =
# Directory in which new sandboxes are created for current project
# and scenario group. ScenarioGroup may be empty, but this doesn't
# affect path construction.
GCAM.SandboxDir = %(GCAM.SandboxProjectDir)s/%(GCAM.ScenarioGroup)s
# If set, application logger messages are written here. Note that
# this is different than the GCAM.BatchLogFile for batch job output.
GCAM.LogFile = %(GCAM.SandboxRoot)s/log/gt.log
# Show log messages on the console (terminal)
GCAM.LogConsole = True
# Format strings for log files and console messages. Note doubled
# '%%' required here around logging parameters to avoid attempted
# variable substitution within the config system.
GCAM.LogFileFormat = %%(asctime)s %%(levelname)s %%(name)s:%%(lineno)d %%(message)s
GCAM.LogConsoleFormat = %%(levelname)s %%(name)s: %%(message)s
# If GCAM.BatchLogFile or the --logFile arg to gt is not an absolute
# path (i.e., the path portion of the logFile does not start with '/',
# with Windows paths converted to Unix format), then batch log files
# are created relative to this directory.
GCAM.BatchLogDir = %(GCAM.SandboxDir)s/log
# Save batch log messages in the indicated file. Default is set up
# for SLURM, which replaces "%j" with the jobid. Note that "%" must
# be doubled (%%) so that it's not interpreted by configparser.
GCAM.BatchLogFile = gt-%%j.out
# The directory above the 'queryResults' and 'diffs' dir where CSV
# files are written to as a result of queries and diff commands.
# GCAM.BatchOutputRoot = %(GCAM.SandboxDir)s
# Set to True to keep XML database in memory, running queries defined
# in the generated XMLDBDriver.properties file based on list of queries
# defined in project. Setting GCAM.InMemoryDatabase to True implies that
# both GCAM.MultipleBatchQueries and GCAM.RunQueriesInGCAM are True.
GCAM.InMemoryDatabase = False
# Set to False to use old method of running ModelInterface anew for each query.
# Default creates a single batch file with multiple queries, invoking MI once.
# Setting GCAM.InMemoryDatabase to True implies that GCAM.MultipleBatchQueries
# is True since this is the only way to extract results from GCAM.
GCAM.BatchMultipleQueries = True
# If True, we expect GCAM to run batch queries before exiting. This is
# typically used with the in-memory database, but works otherwise, too.
# When False, an XMLDBDriver.properties file is written with an empty
# batch-query element. Setting GCAM.InMemoryDatabase to True implies
# that GCAM.RunQueriesInGCAM is True, but queries can still be run in
# GCAM when the XML database is written to disk. Setting this parameter
# to True implies that GCAM.BatchMultipleQueries is True since this is
# the only way to run multiple queries internally in GCAM.
GCAM.RunQueriesInGCAM = False
# The name of an STX:template filter file to use when writing the database
# See http://jgcri.github.io/gcam-doc/user-guide.html#controling-the-level-of-xml-db-output
# for details
GCAM.FilterFile =
# The name of the queue used for submitting batch jobs on a cluster.
# On SLURM, you can request multiple queue, taking first one available.
GCAM.DefaultQueue = slurm
# Notes:
# - "bsub -Is bash" for interactive jobs
# - "bqueues -u $USER" for queues avail to user
# - "bsub -P project_name ..." to specify an "account"
# See https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_welcome/lsf_welcome.html
LSF.BatchCommand = bsub -q {queueName} -n 1 -J {jobName} -W {walltime} -R "rusage[mem=6000]" -eo {logFile} {dependencies} %(GCAM.OtherBatchArgs)s {scriptFile}
LSF.JOB_ID_VAR =
PBS.BatchCommand = qsub -q {queueName} -N {jobName} -l walltime={walltime} -e {logPath} -m n -j oe {dependencies} %(GCAM.OtherBatchArgs)s {scriptFile}
PBS.JOB_ID_VAR = PBS_JOBID
SLURM.BatchCommand = sbatch -p {partition} -J {jobName} -t {walltime} -e {logFile} -o {logFile} --nodes=1 --get-user-env=10L {dependencies} %(GCAM.OtherBatchArgs)s {scriptFile}
SLURM.JOB_ID_VAR = SLURM_JOB_ID
# Arbitrary arguments to add to the selected batch command
GCAM.OtherBatchArgs =
# Known values currently are SLURM, PBS, and LSF
GCAM.BatchSystem = SLURM
GCAM.BatchCommand = %(SLURM.BatchCommand)s
# For qsub, the default number of minutes to allocate per task.
GCAM.Minutes = 20
# A file that maps GCAM regions to rename them or to aggregate
# them. Each line consists of a GCAM region name, some number of
# tabs, and the name to map the region to.
GCAM.RegionMapFile =
# Where to create temporary files
GCAM.TempDir = /tmp
# Where to create temporary batch scripts to run gcamtool.py on a
# compute node. Note that this should not be set to a directory
# that is machine-specific (such as "/tmp"), since the file needs
# to be visible from compute nodes. The directory is created if needed.
GCAM.UserTempDir = %(Home)s/tmp
# For Windows users without permission to create symlinks
GCAM.CopyAllFiles = False
# For debugging purposes: gcamtool.py can show a stack trace on error
GCAM.ShowStackTrace = False
# If set, this format is applied to columns holding values for years
# when combining CSV files into XLSX files.
GCAM.ExcelNumberFormat = 0.000
# TextEditor to open via the --edit option to the 'config' sub-command
GCAM.TextEditor = vi
# Logging level for the Flash web server underlying dash-based GUIs
Flask.LogLevel = WARN
Configuring the logging system¶
Setting logging verbosity¶
When the GCAM tool (gt) runs, or when pygcam functions are called from your
own code, diagnostic and informational messages are printed. You can control
the level of output by setting the GCAM.LogLevel
in your .pygcam.cfg
file. (See logging
for further details.)
The simplest setting is just one of the following values, in order of
decreasing verbosity: DEBUG
, INFO
, WARNING
, ERROR
, and FATAL
.
This will apply to all pygcam modules.
You can also specify verbosity by module, by specifying a module name and the level for that module as a comma-separated list of “module:level” strings, e.g.,:
GCAM.LogLevel = WARN, .utils:INFO, .scenarioSetup:DEBUG, CI_plugin:INFO, \
.mcs.worker:DEBUG,, myProj.writeFuncs:DEBUG
In this example, the default level is set to WARN
, and three pygcam modules
have their levels set: pygcam.utils is set to INFO, pygcam.scenarioSetup is set
to DEBUG, and pygcam.mcs.worker is set to DEBUG. A user’s plugin can also use
the logging system. This example sets logging levels for the user’s
CI_Plugin
and myProj.writeFuncs
modules.
Console / file logs and message formatting¶
Note that the module name is shown in the console log messages. Setting
GCAM.LogLevel
to DEBUG produces the maximum number of log messages;
setting it to FATAL minimizes message verbocity.
Other relevant variables are shown here with their default values:
# If set, application logger messages are written here. Note that
# this is different than the GCAM.BatchLogFile for batch job output.
GCAM.LogFile = %(GCAM.SandboxRoot)s/log/gt.log
# Show log messages on the console (terminal)
GCAM.LogConsole = True
# Format strings for log files and console messages. Note doubled
# '%%' required here around logging parameters to avoid attempted
# variable substitution within the config system.
GCAM.LogFileFormat = %%(asctime)s %%(levelname)s %%(name)s:%%(lineno)d %%(message)s
GCAM.LogConsoleFormat = %%(levelname)s %%(name)s: %%(message)s