pygcam.utils

This module contains constants, functions, and classes that are used by various other modules of the pygcam toolkit.

API

class pygcam.utils.ParseCommaList(option_strings, dest, nargs=None, **kwargs)
pygcam.utils.chunkify(lst, chunks)

Iterator to turn a list into the number of lists given by chunks. In the case that len(lst) % chunksize != 0, all chunks are made as close to the same size as possible.

Parameters:
  • lst – (list) a list of values
  • chunks – (int) the number of chunks to create
Returns:

iterator that returns one chunk at a time
pygcam.utils.coercible(value, type, raiseError=True)

Attempt to coerce a value to type and raise an error on failure.

Parameters:
  • value – any value coercible to type
  • type – any Python type
Returns:

(type) the coerced value, if it’s coercible, otherwise None if raiseError is False
Raises:

PygcamException – if not coercible and raiseError is True
pygcam.utils.copyFileOrTree(src, dst)

Copy src to dst, where the two can both be files or directories. If src and dst are directories, dst must not exist yet.

Parameters:
  • src – (str) path to a source file or directory
  • dst – (str) path to a destination file or directory.
Returns:

none
pygcam.utils.copyResource(relpath, dest, overwrite=True)

Copy a resource from the ‘pygcam’ package to the given destination.

Parameters:
  • relpath – (str) a path relative to the pygcam package
  • dest – (str) the pathname of the file to create by copying the resource.
  • overwrite – (bool) if False, raise an error if the destination file already exists.
Returns:

none
pygcam.utils.createTrialString(lst)

Assemble a list of numbers into a compact list using hyphens to identify ranges. This reverses the operation of parseTrialString

pygcam.utils.deleteFile(filename)

Delete the given filename, but ignore errors, like “rm -f”

Parameters:filename – (str) the file to remove
Returns:none
pygcam.utils.digitColumns(df, asInt=False)

Get a list of columns with integer names (as strings, e.g., “2007”) in df. If asInt is True return as a list of integers, otherwise as strings.

pygcam.utils.ensureCSV(file)

Ensure that the file has a ‘.csv’ extension by replacing or adding the extension, as required.

Parameters:file – (str) a filename
Returns:(str) the filename with a ‘.csv’ extension.
pygcam.utils.ensureExtension(filename, ext)

Force a filename to have the given extension, ext, adding it to any other extension, if present. That is, if filename is foo.bar, and ext is baz, the result will be foo.bar.baz. If ext doesn’t start with a “.”, one is added.

Parameters:
  • filename – filename
  • ext – the desired filename extension
Returns:

filename with extension ext
pygcam.utils.flatten(listOfLists)

Flatten one level of nesting given a list of lists. That is, convert [[1, 2, 3], [4, 5, 6]] to [1, 2, 3, 4, 5, 6].

Parameters:listOfLists – a list of lists, obviously
Returns:the flattened list
pygcam.utils.getBatchDir(scenario, resultsDir)

Get the name of the directory holding batch query results..

Parameters:
  • scenario – (str) the name of a scenario
  • resultsDir – (str) the directory in which the batch results directory should be created
Returns:

(str) the pathname to the batch results directory
pygcam.utils.getBooleanXML(value)

Get a value from an XML file and convert it into a boolean True or False.

Parameters:value – any value (it’s first converted to a string)
Returns:True if the value is in [‘true’, ‘yes’, ‘1’], False if the value is in [‘false’, ‘no’, ‘0’]. An exception is raised if any other value is passed.
Raises:PygcamException
pygcam.utils.getRegionList(workspace=None, states='withGlobal')

Set the list of the defined region names from the data system, if possible, otherwise use the built-in list of 32 regions.

Parameters:
  • workspace – (str) the path to a Main_User_Workspace directory that has the file input/gcamdata/inst/extdata/common/GCAM_region_names.csv, or None, in which case the value of config variable GCAM.SourceWorkspace (if defined) is used. If workspace is empty or None, and the config variable GCAM.SourceWorkspace is empty (the default value), the built-in default 32-region list is returned.
  • states – (str) One of {‘together’, ‘only’, ‘none’}. Default is ‘together’.
Returns:

a list of strings with the names of the defined regions
pygcam.utils.getResource(relpath)

Extract a resource (e.g., file) from the given relative path in the pygcam package.

Parameters:relpath – (str) a path relative to the pygcam package
Returns:the file contents
pygcam.utils.getYearCols(years, timestep=5)

Generate a list of names of year columns in GCAM result files from a string indicating a year range.

Parameters:
  • years – (str) a string of the form “2020-2050”
  • timestep – (int) the number of years between timesteps
Returns:

(list of strings) the names of the corresponding columns
pygcam.utils.get_path(pathname, defaultDir)

Return pathname if it’s an absolute pathname, otherwise return the path composed of pathname relative to the given defaultDir.

pygcam.utils.importFrom(modname, objname, asTuple=False)

Import modname and return reference to objname within the module.

Parameters:
  • modname – (str) the name of a Python module
  • objname – (str) the name of an object in module modname
  • asTuple – (bool) if True a tuple is returned, otherwise just the object
Returns:

(object or (module, object)) depending on asTuple
pygcam.utils.importFromDotSpec(spec)

Import an object from an arbitrary dotted sequence of packages, e.g., “a.b.c.x” by splitting this into “a.b.c” and “x” and calling importFrom().

Parameters:spec – (str) a specification of the form package.module.object
Returns:none
Raises:PygcamException – if the import fails
pygcam.utils.is_abspath(pathname)

Return True if pathname is an absolute pathname, else False.

pygcam.utils.loadModuleFromPath(modulePath, raiseOnError=True)

Load a module from a ‘.py’ or ‘.pyc’ file from a path that ends in the module name, i.e., from “foo/bar/Baz.py”, the module name is ‘Baz’.

Parameters:
  • modulePath – (str) the pathname of a python module (.py or .pyc)
  • raiseOnError – (bool) if True, raise an error if the module cannot be loaded
Returns:

(module) a reference to the loaded module, if loaded, else None.
Raises:

PygcamException
pygcam.utils.mkdirs(newdir, mode=504)

Try to create the full path newdir and ignore the error if it already exists.

Parameters:newdir – the directory to create (along with any needed parent directories)
Returns:nothing
pygcam.utils.model_years()

Return the defined model years by parsing xml/modeltime.xml. This is used by the setup plugin to convert years to model period for setting the stop year.

Returns:(list of int) the defined model years.
pygcam.utils.parseTrialString(string)

Converts a comma-separated list of ranges into a list of numbers. Ex. 1,3,4-6,2 becomes [1,3,4,5,6,2]. Duplicates are deleted. This function is the inverse of createTrialString().

Parameters:string – (str) comma-separate list of ints or int ranges indicated by two ints separated by a hyphen.
Returns:(list) a list of ints
pygcam.utils.printSeries(series, label, header='', asStr=False)

Print a series of values, with a give label.

Parameters:
  • series – (convertible to pandas Series) the values
  • label – (str) a label to print for the data
Returns:

none
pygcam.utils.pushd(directory)

Context manager that changes to the given directory and then returns to the original directory. Usage is with pushd('/foo/bar'): ...

Parameters:directory – (str) a directory to chdir to temporarily
Returns:none
pygcam.utils.queueForStream(stream)

Create a thread to read from a non-socket file descriptor and its contents to a socket so non-blocking read via select() works on Windows. (Since Windows doesn’t support select on pipes.)

Parameters:stream – (file object) the input to read from, presumably a pipe from a subprocess
Returns:(int) a file descriptor for the socket to read from.
pygcam.utils.readScenarioName(configFile)

Read the file configFile and extract the scenario name.

Parameters:configFile – (str) the path to a GCAM configuration file
Returns:(str) the name of the scenario defined in configFile
pygcam.utils.removeFileOrTree(path, raiseError=True)

Remove a file or an entire directory tree. Handles removal of symlinks on Windows, as these are treated differently in that system.

Parameters:
  • path – (str) the pathname of a file or directory.
  • raiseError – (bool) if True, re-raise any error that occurs during the file operations, else errors are ignored.
Returns:

none
pygcam.utils.resourceStream(relpath)

Return a stream on the resource found on the given path relative to the pygcam package.

Parameters:relpath – (str) a path relative to the pygcam package
Returns:(file-like stream) a file-like buffer opened on the desired resource.
pygcam.utils.saveToFile(txt, dirname='', filename='')

Save the given text to a file in the given directory.

Parameters:
  • txt – (str) the text to save
  • dirname – (str) path to a directory
  • filename – (str) the name of the file to create
Returns:

none
pygcam.utils.shellCommand(command, shell=True, raiseError=True)

Run a shell command and optionally raise PygcamException error.

Parameters:
  • command – the command to run, with arguments. This can be expressed either as a string or as a list of strings.
  • shell – if True, run command in a shell, otherwise run it directly.
  • raiseError – if True, raise ToolError on command failure.
Returns:

exit status of executed command
Raises:

ToolError
pygcam.utils.simpleFormat(s, varDict)

Simple version of str.format that does not treat ‘.’ as an attribute reference.

Parameters:
  • s – (str) string with args in curly braces
  • varDict – (dict) dictionary of var names and values
Returns:

(str) formatted string
pygcam.utils.symlinkOrCopyFile(src, dst)

Symlink a file unless GCAM.CopyAllFiles is True, in which case, copy the file.

Parameters:
  • src – (str) filename of original file
  • dst – (dst) filename of copy
Returns:

none
pygcam.utils.systemOpenFile(path)

Ask the operating system to open a file at the given pathname.

Parameters:path – (str) the pathname of a file to open
Returns:none
pygcam.utils.writeXmldbDriverProperties(outputDir='.', inMemory=True, filterFile='', batchFile='', batchLog='')

Write a XMLDBDriver.properties file using the values passed in the arguments.

Parameters:
  • outputDir – (str) where to write the file
  • inMemory – (bool) if True, the in-memory attribute is set to True
  • filterFile – (str) a file that filters GCAM query output to limit what’s written to the database
  • batchFile – (str) the path to an XML batch query file
  • batchLog – (str) the path to a log file into which to direct batch query messages (queries can be pretty verbose…)
Returns:

none