desiutil API

desiutil

This package provides low-level utilities for general use by DESI.

desiutil.annotate

Tools for adding units and comments to FITS files.

exception desiutil.annotate.FITSUnitWarning[source]

Warnings related to invalid FITS units.

desiutil.annotate._options()[source]

Parse command-line options.

desiutil.annotate.annotate_fits(filename, extension, output, units=None, comments=None, validate=True, truncate=False, overwrite=False, verbose=False)[source]

Add annotations to HDU extension in FITS file filename.

HDU extension must be a astropy.io.fits.BinTableHDU.

If units or comments is an empty dictionary, it will be ignored.

Parameters:
  • filename (str) – Name of FITS file.

  • extension (str or int) – Name or number of extension in filename.

  • output (str) – Name of file to write to.

  • units (dict, optional) – Mapping of table columns to units.

  • comments (dict, optional) – Mapping of table columns to comments.

  • validate (bool, optional) – By default, units will be checked for compliance with the FITS standard. Setting this to False disables errors, although warnings will still be printed.

  • truncate (bool, optional) – Allow long comments to be truncated when written out. The default is to raise an error if a comment is too long.

  • overwrite (bool, optional) – Pass this keyword to astropy.io.fits.HDUList.writeto().

  • verbose (bool, optional) – Include debug-level logging

Returns:

An updated version of the file, equivalent to the data in output.

Return type:

astropy.io.fits.HDUList

Raises:
  • IndexError – If the HDU specified (numerically) by extension does not exist.

  • KeyError – If the HDU specified (as an EXTNAME) by extension does not exist.

  • TypeError – If the HDU specified is not supported by this function.

  • ValueError – If neither units nor comments are specified.

Notes

  • Due to the way astropy.io.fits.open() manages memory, changes have to be written out while filename is still open, hence the mandatory output argument.

  • A FITS HDU checksum will always be added to the output, even if it was not already present.

desiutil.annotate.annotate_table(table, units, inplace=False, validate=True)[source]

Add annotations to table.

Parameters:
  • table (astropy.table.Table or astropy.table.QTable) – A data table.

  • units (dict) – Mapping of table columns to units.

  • inplace (bool, optional) – If True, modify table directly instead of returning a copy.

  • validate (bool, optional) – If True raise an error if any unit does not follow the FITS standard.

Returns:

An updated version of table.

Return type:

astropy.table.Table

Raises:

ValueError – If validate is True and an invalid unit is detected.

Notes

Currently Table does not support the concept of column-specific comments, especially in a way where the comments could be written to a file. If this ever changes, this function could be extended to add comments.

desiutil.annotate.check_comment_length(comments, error=True)[source]

Ensure keyword comments are short enough that they will not be truncated.

By experiment, astropy.io.fits will truncate comments longer than 46 characters, however the warning it emits when it does so is not very informative.

Parameters:
  • comments (dict) – Mapping of table columns to comments.

  • error (bool, optional) – If False just warn about long comments instead of raising an exception.

Returns:

The number of long comments detected, although the value is only relevant if error is False.

Return type:

int

Raises:

ValueError – If any comment is too long.

desiutil.annotate.find_column_name(columns, prefix=('unit',))[source]

Find the column that contains unit descriptions, or comments.

Parameters:
  • columns (iterable) – The column names from a CSV or similar file.

  • prefix (tuple, optional) – Search for a column matching one or more items in prefix. The search is case-insensitive.

Returns:

The index of columns that matches.

Return type:

int

Raises:

IndexError – If no match was found.

desiutil.annotate.find_key_name(data, prefix=('unit',))[source]
Parameters:
  • data (dict) – A dictionary resulting from a parsed YAML file.

  • prefix (tuple, optional) – Search for a column matching one or more items in prefix. The search is case-insensitive.

Returns:

The key of data that matches.

Return type:

str

Raises:

KeyError – If no match was found.

desiutil.annotate.load_csv_units(filename)[source]

Parse a CSV file that contains column names and units and optionally comments.

Table column names are assumed to be in the first column of the CSV file. Any column with the name “Unit(s)” (case-insensitive) is assumed to contain FITS-style units. Any column with the name “Comment(s)” (case-insensitive) or “Description(s)” is assumed to be the comment.

Parameters:

filename (str or pathlib.Path) – Read column definitions from filename.

Returns:

A tuple containing two dict objects for units and comments. If no comments are detected, the comments dict will be empty.

Return type:

tuple

Raises:

ValueError – If filename does not at least contain a “unit” column.

desiutil.annotate.load_yml_units(filename)[source]

Parse a YAML file that contains column names and units and optionally comments.

The YAML file should contain a dictionary with a keyword like ‘units’ and, optionally, a keyword like ‘comments’ or ‘description.

For backwards-compatibility, the YAML file can be simply a dictionary containing column names.

Parameters:

filename (str or pathlib.Path) – Read column definitions from filename.

Returns:

A tuple containing two dict objects for units and comments. If no comments are detected, the comments dict will be empty.

Return type:

tuple

desiutil.annotate.main()[source]

Entry-point for command-line scripts.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desiutil.annotate.validate_unit(unit, error=False)[source]

Check units for consistency with FITS standard, while allowing some special exceptions.

Parameters:
  • unit (str) – The unit to parse.

  • error (bool, optional) – If True, failure to interpret the unit raises an exception.

Returns:

If a special exception is detected, the name of the unit is returned. Otherwise, None.

Return type:

str

Raises:

ValueError – If error is set and unit can’t be parsed.

desiutil.api

This package contains code for creating API files for use with Sphinx documentation. The resulting api file simply has a Sphinx-readable link to every *.py file.

desiutil.api._test_file(d, f)[source]

Do not generate an API entry for test files.

desiutil.api.find_modules(name)[source]

Find *.py files in the package directory corresponding to name.

Parameters:

name (str) – The name of the package.

Returns:

The modules found in the package.

Return type:

list

desiutil.api.main()[source]

Entry-point for command-line scripts.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desiutil.api.write_api(modules, options)[source]

Write out a file containing the modules found.

Parameters:
  • modules (list) – The names of the modules found in the package.

  • options (Namespace) – The command-line options.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desiutil.bitmask

Mask bits for the spectro pipeline.

Individual packages will define their own mask bits and use this as a utility access wrapper. Typical users will get their bitmasks pre-made from those packages, not from here.

Stephen Bailey, Lawrence Berkeley National Lab Fall 2015

Examples

desispec could create a ccdmask like this:

>>> from desiutil.bitmask import BitMask
>>> import yaml
>>> _bitdefs = yaml.safe_load('''
... ccdmask:
...     - [BAD,       0, "Pre-determined bad pixel (any reason)"]
...     - [HOT,       1, "Hot pixel"]
...     - [DEAD,      2, "Dead pixel"]
...     - [SATURATED, 3, "Saturated pixel from object"]
...     - [COSMIC,    4, "Cosmic ray"]
... ''')
...
>>> ccdmask = BitMask('ccdmask', _bitdefs)

Users would then access this mask with:

>>> from desispec.bitmasks import ccdmask
>>> ccdmask.COSMIC | ccdmask.SATURATED  #- 2**4 + 2**3
24
>>> ccdmask.mask('COSMIC')     # 2**4, same as ccdmask.COSMIC
16
>>> ccdmask.mask(4)            # 2**4, same as ccdmask.COSMIC
16
>>> ccdmask.COSMIC             # 2**4, same as ccdmask.mask('COSMIC')
16
>>> ccdmask.bitnum('COSMIC')
4
>>> ccdmask.bitname(4)
'COSMIC'
>>> ccdmask.names()
['BAD', 'HOT', 'DEAD', 'SATURATED', 'COSMIC']
>>> ccdmask.names(3)
['BAD', 'HOT']
>>> ccdmask.comment(0)
'Pre-determined bad pixel (any reason)'
>>> ccdmask.comment('COSMIC')
'Cosmic ray'
class desiutil.bitmask.BitMask(name, bitdefs)[source]

BitMask object to represent bit names, masks, and comments.

Typical users are not expected to create BitMask objects directly; other packages like desispec and desitarget will have used this to pre-create the bitmasks for them using definition files in those packages.

Parameters:
  • name (str) – Name of this mask, must be key in bitdefs.

  • bitdefs (dict) – Dictionary of different mask bit definitions; each value is a list of [bitname, bitnum, comment]. A 4th entry is optional, which must be a dictionary.

bitname(bitnum)[source]

Return bit name (string) for this bitnum (integer).

Parameters:

bitnum (int) – The number of the bit.

Returns:

The name of the bit.

Return type:

str

bitnum(bitname)[source]

Return bit number (int) for this bitname (string).

Parameters:

bitname (str) – The bit name.

Returns:

The bit value.

Return type:

int

comment(bitname_or_num)[source]

Return comment for this bit name or bit number.

Parameters:

bitname_or_num (int or str) – Name of number of the mask.

Returns:

The comment string.

Return type:

str

mask(name_or_num)[source]

Return mask value.

Parameters:

name_or_num (int or str) – Name of number of the mask.

Returns:

The value of the mask.

Return type:

int

Examples

>>> bitmask.mask(3)         #  2**3
8
>>> bitmask.mask('BLAT')
>>> bitmask.mask('BLAT|FOO')
names(mask=None)[source]

Return list of names of masked bits.

Parameters:

mask (int, optional) – The mask integer to convert to names. If not supplied, return names of all known bits.

Returns:

The list of names contained in the mask.

Return type:

list

class desiutil.bitmask._MaskBit(name, bitnum, comment, extra={})[source]

A single mask bit.

Subclasses int to act like an int, but allows the ability to extend with blat.name, blat.comment, blat.mask, blat.bitnum.

name

The name of the bit.

Type:

str

bitnum

The number of the bit. The value of the bit is 2**bitnum.

Type:

int

mask

The value of the bit, 2**bitnum.

Type:

int

comment

A comment explaining the meaning of the bit.

Type:

str

desiutil.brick

Code for calculating bricks, which are a tiling of the sky with the following properties:

  • bricks form rows in dec like a brick wall; edges are constant RA or dec

  • they are rectangular with longest edge shorter or equal to bricksize

  • circles at the poles with diameter=bricksize

  • there are an even number of bricks per row

Use this with caution! In most cases you should be propagating brick info from input targeting, not recalculating brick locations and names.

Note that this code was originally in desispec, so earlier commit information is in the desispec repository.

class desiutil.brick.Bricks(bricksize=0.25)[source]

The Bricks object describes bricks of a certain size.

Parameters:

bricksize (float, optional) – Brick size in degrees. Default 0.25 degrees.

_array_radec(ra, dec)[source]

Convert (ra, dec) to arrays and clean up the data.

_row_col(ra, dec)[source]

Determine the brick row and column, given ra, dec.

brick_radec(ra, dec)[source]

Return center (ra,dec) of brick that contains input (ra, dec) [deg]

Parameters:
Returns:

The centers of the bricks at the locations of interest.

Return type:

ndarray

brick_tan_wcs_size()[source]

Compute required angular size needed for WCS transformation.

Returns the minimum required angular size (pixel scale x number of pixels) for a TAN WCS tiling on these brick centers, so that RA1, RA2, DEC1, DEC2 land within the tile.

Returns:

The angular size in degrees.

Return type:

float

brickarea(ra, dec)[source]

Return the area of the brick for a given location.

Parameters:
Returns:

The areas of the bricks at the locations of interest.

Return type:

ndarray

brickid(ra, dec)[source]

Return the BRICKID for a given location.

Parameters:
Returns:

The legacysurvey BRICKID at the locations of interest.

Return type:

ndarray

brickname(ra, dec)[source]

Return brick name of brick covering (ra, dec).

Parameters:
Returns:

An array of strings containing the names.

Return type:

ndarray

brickq(ra, dec)[source]

Return the BRICKQ for a given location.

Parameters:
Returns:

The legacysurvey BRICKQ at the locations of interest.

Return type:

ndarray

property bricksize

Size of a brick in degrees.

brickvertices(ra, dec)[source]

Return the vertices in RA/Dec of the brick that given locations lie in

Parameters:
Returns:

The 4 vertices of the bricks at the locations of interest (an array with 4 columns of (RA, Dec) and len(ra) rows).

Return type:

ndarray

Notes

The vertices are ordered counter-clockwise from the minimum (RA, Dec).

to_table()[source]

Convert Bricks object into a Table.

Returns:

A table containing the brick data.

Return type:

astropy.table.Table

desiutil.brick.brickname(ra, dec, bricksize=0.25)[source]

Return brick name of brick covering (ra, dec).

Parameters:
  • ra (float or ndarray) – Right Ascension in degrees.

  • dec (float or ndarray) – Declination in degrees.

  • bricksize (float, optional) – Brick size in degrees. Default 0.25 degrees.

Returns:

An array of strings containing the names.

Return type:

ndarray

Notes

This function is a convenience wrapper on desiutil.brick.Bricks.brickname(). It will cache the brick computation to speed up repeated calls.

desiutil.census

Determine the number of files and size in DESI data file systems.

As of 2022, this module is deprecated, and the command-line interface, desi_data_census has been removed. It was superseded by NERSC-provided metadata files that are much faster to parse.

Notes

  • Directories to check:

    • Imaging raw & reduced.

    • spectro raw & reduced.

    • Work directories.

    • Non-Footprint image data.

  • Check group id, readability.

  • Count number of files and size.

  • Extract year from mtime. Shift to fiscal year. FY starts in October.

  • Don’t record filenames, just high-level directories.

  • Treat projecta as same system, follow symlinks to projecta

  • If a symlink is followed to another filesystem, os.walk() can’t get back to the original filesystem.

  • Symlinks to another subdirectory should only count as the symlink. The file itself belongs to the other subdirectory.

  • Physical directories count toward inode and size total.

class desiutil.census.ScannedFile(filename, size, year)[source]

Simple object to store results of a file scan.

filename

Name of the file.

Type:

str

size

Size in bytes of the file.

Type:

int

year

Year the file was modified.

Type:

int

Is the file a symbolic link?

Type:

bool

isexternal

If the file is a symbolic link, does it link outside the tree being scanned?

Type:

bool

linkname

If the file is a symbolic link, it points to this file.

Type:

str

linksize

If the file is a symbolic link, this is the size of the link itself, and the size attribute is the size of the file it points to.

Type:

int

linkyear

Year the link itself was modified.

Type:

int

desiutil.census.get_options(test_args=None)[source]

Parse command-line options.

Parameters:

test_args (list) – Override command-line arguments for testing purposes.

Returns:

A simple object containing the parsed options.

Return type:

argparse.Namespace

desiutil.census.in_path(root, path)[source]

Check if path is in the same directory hierarchy as root.

Parameters:
  • root (str) – Root directory.

  • path (str) – Filename, could be a file or a directory.

Returns:

True if path is in root.

Return type:

bool

desiutil.census.main()[source]

Entry point for the desi_data_census script.

Returns:

Exit status that will be passed to sys.exit().

Return type:

int

desiutil.census.output_csv(summary, filename)[source]

Convert data into CSV file.

Parameters:
  • summary (list) – A data structure.

  • filename (str) – Name of the file to write to.

Returns:

The data written to the CSV file, as a list of rows.

Return type:

list

desiutil.census.scan_directories(conf, data)[source]

Scan the directories specified by the configuration file.

Parameters:
  • conf (dict) – The configuration that applies to all directories.

  • data (list) – The specific directories to scan.

Returns:

A list containing data structures summarizing data found.

Return type:

list

desiutil.census.scan_directory(dirpath, dirnames, filenames, gid)[source]

Count number and size of files in a single directory hierarchy.

Parameters:
  • dirpath (str) – Current directory, returned by os.walk().

  • dirnames (list) – List of directories in dirpath.

  • filenames (list) – List of files in dirpath.

  • gid (int) – Group ID number that should be associated with this directory.

Returns:

A tuple containing two dictionaries: the summary results organized by year, and a summary of links to external directories.

Return type:

tuple()

desiutil.census.scan_file(dirpath, filename, gid)[source]

Analyze a single file or directory.

Parameters:
  • dirpath (str) – Current directory, returned by os.walk().

  • filename (str) – Base name of current file.

  • gid (int) – Group ID number that should be associated with this directory.

Returns:

A simple object containing the metadata relating to the file.

Return type:

ScannedFile

desiutil.census.walk_error(e)[source]

Handle errors reported by os.walk().

Parameters:

e (OSError) – The exception reported.

desiutil.census.year(mtime, fy=True)[source]

Convert a file’s modification time into a year.

Parameters:
  • mtime (int or float) – File modification time as reported by os.stat().

  • fy (bool, optional) – If True use Fiscal Year (FY) instead of calendar year. FY is defined to begin 1 October.

Returns:

The year to which a file belongs.

Return type:

int

desiutil.depend

Utilities for working with code dependencies stored in FITS headers.

The code name and the code version are stored in pairs of keywords, similar to how table columns are defined. e.g.:

DEPNAM00 = 'numpy'
DEPVER00 = '1.11'
DEPNAM01 = 'desiutil'
DEPVER01 = '1.4.1'

The functions and Dependencies class provide convenience wrappers to loop over they keywords looking for a particular dependency and adding a new dependency version to next available DEPNAMnn/DEPVERnn.

Examples:

>>> import desiutil
>>> from desiutil import depend
>>> import astropy
>>> from astropy.io import fits
>>>
>>> hdr = fits.Header()
>>> depend.setdep(hdr, 'desiutil', desiutil.__version__)
>>> depend.setdep(hdr, 'astropy', astropy.__version__)
>>> depend.getdep(hdr, 'desiutil')
'1.4.1.dev316'
>>> depend.hasdep(hdr, 'astropy')
True
>>> hdr
DEPNAM00= 'desiutil'
DEPVER00= '1.4.1.dev316'
DEPNAM01= 'astropy '
DEPVER01= '1.1.1   '

There is also an object wrapper that gives a header dict-like semantics to update or view dependencies. This directly updates the input header object so that it can be used in subsequent I/O

>>> codever = depend.Dependencies(hdr)
>>> codever['blat'] = '1.2'
>>> codever['foo'] = '3.4'
>>> for codename, version in codever.items():
...     print(codename, version)
...
('desiutil', '1.4.1.dev316')
('astropy', u'1.1.1')
('blat', '1.2')
('foo', '3.4')
class desiutil.depend.Dependencies(header=None)[source]

Dictionary-like object to track dependencies.

Parameters:

header (dict-like, optional) – A dict-like object. If not provided, a OrderedDict will be used.

items()[source]

Returns iterator over (name, version).

desiutil.depend.add_dependencies(header, module_names=None, long_python=False, envvar_names=None)[source]

Adds DEPNAMnn, DEPVERnn keywords to header for imported modules.

Parameters:
  • header (dict-like) – A dict-like object, e.g. astropy.io.fits.Header.

  • module_names (list, optional) – List of of module names to check; if None, checks desiutil.depend.possible_dependencies.

  • long_python (bool, optional) – If True use the full, verbose sys.version string for the Python version. Otherwise, use a short version, e.g., 3.5.2.

  • envvar_names (list, optional) – List of of environment variables to check; if None, checks desiutil.depend.possible_envvars.

Notes

Only adds the dependency keywords if the module has already been previously loaded in this python session. Uses module.__version__ if available, otherwise unknown (/path/to/module/).

desiutil.depend.getdep(header, name)[source]

Get dependency version for code name.

Parameters:
Returns:

The version string for name.

Return type:

str

Raises:

KeyError – If name not tracked in header.

desiutil.depend.hasdep(header, name)[source]

Check if name is defined in header.

Parameters:
Returns:

True if version for name is tracked in header, otherwise False.

Return type:

bool

desiutil.depend.iterdep(header)[source]

Returns iterator over (codename, version) tuples.

Parameters:

header (dict-like) – A dict-like object, e.g. astropy.io.fits.Header.

desiutil.depend.mergedep(srchdr, dsthdr, conflict='src')[source]

Merge dependencies from srchdr into dsthdr

Parameters:
  • srchdr (dict-like) – source dict-like object, e.g. astropy.io.fits.Header, with dependency keywords DEPNAMnn, DEPVERnn

  • dsthdr (dict-like) – destination dict-like object

  • conflict (str, optional) – ‘src’ or ‘dst’ or ‘exception’; see notes

Notes

Dependencies in srchdr are added to dsthdr, modifying it in-place, adjusting DEPNAMnn/DEPVERnn numbering as needed. If the same dependency appears in both headers with different versions, conflict controls the behavior:

  • if ‘src’, the srchdr value replaces the dsthdr value

  • if ‘dst’, the dsthdr value is retained unchanged

  • if ‘exception’, raise a ValueError exception

Raises:

ValueError – If conflict == 'exception' and the same dependency name appears in both headers with different values; or if conflict isn’t one of ‘src’, ‘dst’, or ‘exception’.

desiutil.depend.remove_dependencies(header)[source]

Remove all DEPNAMnn/DEPVERnn dependencies from a header

Parameters:

header (dict-like) – header with DEPNAMnn/DEPVERnn keywords to remove

Notes

Updates header in-place

desiutil.depend.removedep(header, name)[source]

Removed dependency name from header.

Parameters:
  • header (dict-like) – header object with DEPNAMnn/DEPVERnn keywords

  • name (str) – name of dependency to remove

Notes

Modifies header in-place

Raises:

ValueError – If name is not present in any of the DEPNAMnn keys

desiutil.depend.setdep(header, name, version)[source]

Set dependency version for code name.

Parameters:
  • header (dict-like) – A dict-like object, e.g. astropy.io.fits.Header.

  • name (str) – Code name string.

  • version (str) – Code version string.

Raises:

IndexError – If there are more than 100 dependencies to track.

desiutil.dust

Get \(E(B-V)\) values from the Schlegel, Finkbeiner & Davis (1998; SFD98) dust map.

class desiutil.dust.SFDMap(mapdir=None, north='SFD_dust_4096_ngp.fits', south='SFD_dust_4096_sgp.fits', scaling=1.0)[source]

Map of E(B-V) from Schlegel, Finkbeiner and Davis (1998).

Use this class for repeated retrieval of E(B-V) values when there is no way to retrieve all the values at the same time: It keeps a reference to the FITS data from the maps so that each FITS image is read only once.

Parameters:
  • mapdir (str, optional, defaults to DUST_DIR`+`/maps``.) – Directory in which to find dust map FITS images, named SFD_dust_4096_ngp.fits and SFD_dust_4096_sgp.fits. If not specified, the map directory is derived from the value of the DUST_DIR environment variable, otherwise an empty string is used.

  • north (str, optional) – Names of north and south galactic pole FITS files. Defaults are SFD_dust_4096_ngp.fits and SFD_dust_4096_sgp.fits respectively.

  • south (str, optional) – Names of north and south galactic pole FITS files. Defaults are SFD_dust_4096_ngp.fits and SFD_dust_4096_sgp.fits respectively.

  • scaling (float, optional, defaults to 1) – Scale all E(B-V) map values by this multiplicative factor. Pass scaling=0.86 for the recalibration from Schlafly & Finkbeiner (2011).

Notes

Modified from https://github.com/kbarbary/sfdmap/

ebv(*args, **kwargs)[source]

Get E(B-V) value(s) at given coordinate(s).

Parameters:
  • coordinates (SkyCoord or ndarray) – If one argument is passed, assumed to be an SkyCoord instance, in which case the frame and unit keyword arguments are ignored. If two arguments are passed, they are treated as latitute, longitude (can be scalars or arrays or a tuple), in which case the frame and unit are taken from the passed keywords.

  • frame (str, optional, defaults to 'icrs') – Coordinate frame, if two arguments are passed. Allowed values are any SkyCoord frame, and 'fk5j2000' and 'j2000'.

  • unit (str, optional, defaults to 'degree') – Any SkyCoord unit.

  • interpolate (bool, optional, defaults to True) – Interpolate between the map values using bilinear interpolation.

Returns:

Specific extinction E(B-V) at the given locations.

Return type:

ndarray

Notes

Modified from https://github.com/kbarbary/sfdmap/

class desiutil.dust._Hemisphere(fname, scaling)[source]

Represents one of the hemispheres (in a single file).

Parameters:
  • fname (str) – File name containing one hemisphere of the dust map.

  • scaling (float) – Multiplicative factor by which to scale the dust map.

data

Pixelated array of dust map values.

Type:

ndarray

crpix1, crpix2

World Coordinate System: Represent the 1-indexed X and Y pixel numbers of the poles.

Type:

float

lam_scal

Number of pixels from b=0 to b=90 deg.

Type:

int

lam_nsgp

+1 for the northern hemisphere, -1 for the south.

Type:

int

Notes

Taken in full from https://github.com/kbarbary/sfdmap/

ebv(l, b, interpolate)[source]

Project Galactic longitude/latitude to lambert pixels (See SFD98).

Parameters:
  • l (numpy.ndarray) – Galactic longitude and latitude.

  • b (numpy.ndarray) – Galactic longitude and latitude.

  • interpolate (bool) – If True use bilinear interpolation to obtain values.

Returns:

Reddening values.

Return type:

ndarray

desiutil.dust._bilinear_interpolate(data, y, x)[source]

Map a two-dimensional integer pixel-array at float coordinates.

Parameters:
  • data (ndarray) – Pixelized array of values.

  • y (float or ndarray) – y coordinates (each integer y is a row) of location in pixel-space at which to interpolate.

  • x (float or ndarray) – x coordinates (each integer x is a column) of location in pixel-space at which to interpolate.

Returns:

Interpolated data values at the passed locations.

Return type:

float or ndarray

Notes

Taken in full from https://github.com/kbarbary/sfdmap/

desiutil.dust._get_ext_coeff(temp, photsys, band, ebv_sfd, rv=3.1)[source]

Obtain extinction coeffiecient A_X/E(B-V)_SFD for black body spectrum with a given temperature observed with photsys and band and extinction ebv_sfd

Parameters:
  • temp (float) – temperature in Kelvin

  • photsys (str) – N, S, or G (gaia)

  • band (str) – g,r,z (if photsys=N or S); G, BP, or RP if photsys=G

  • ebv_sfd (float) – E(B-V) from SFD dust map

  • Rv (float) – Value of extinction law R_V; default is 3.1

Returns extinction coefficient A_X / E(B-V)_SFD

Note: this is currently a _hidden function due to its speclite dependency, but it could be promoted if needed by external libraries.

desiutil.dust.dust_transmission(wave, ebv_sfd, Rv=3.1)[source]

Return the dust transmission [0-1] vs. wavelength.

Parameters:
  • wave – 1D array of vacuum wavelength [Angstroms]

  • ebv_sfd – E(B-V) as derived from the map of Schlegel, Finkbeiner and Davis (1998)

  • Rv – total-to-selective extinction ratio, defaults to 3.1

Returns:

1D array of dust transmission (between 0 and 1)

The routine does internally multiply ebv_sfd by dust.ebv_sfd_scaling. The Fitzpatrick dust extinction law is used, given R_V (default 3.1).

Also see mwdust_transmission which return transmission within a filter

desiutil.dust.ebv(*args, **kwargs)[source]

Convenience function, equivalent to SFDMap().ebv(*args).

desiutil.dust.ext_ccm(wave, Rv=3.1)[source]

Return extinction curve from CCM (1989), defined in the wavelength range [1250,33333] Angstroms.

Parameters:
  • wave – 1D array of vacuum wavelength [Angstroms]

  • Rv – Value of R_V (scalar); default is 3.1

Returns:

1D array of A(lambda)/A(V)

desiutil.dust.ext_fitzpatrick(wave, R_V=3.1, avglmc=False, lmc2=False, x0=None, gamma=None, c1=None, c2=None, c3=None, c4=None)[source]

Return extinction curve from Fitzpatrick (1999).

Parameters:

wave – 1D array of vacuum wavelength [Angstroms]

Returns:

1D array of A(lambda)/A(V)

OPTIONAL INPUT KEYWORDS
R_V - scalar specifying the ratio of total to selective extinction

R(V) = A(V) / E(B - V). If not specified, then R = 3.1 Extreme values of R(V) range from 2.3 to 5.3

avglmc - if set, then the default fit parameters c1,c2,c3,c4,gamma,x0

are set to the average values determined for reddening in the general Large Magellanic Cloud (LMC) field by Misselt et al. (1999, ApJ, 515, 128)

lmc2 - if set, then the fit parameters are set to the values determined

for the LMC2 field (including 30 Dor) by Misselt et al. Note that neither /AVGLMC or /LMC2 will alter the default value of R_V which is poorly known for the LMC.

The following five input keyword parameters allow the user to customize the adopted extinction curve. For example, see Clayton et al. (2003, ApJ, 588, 871) for examples of these parameters in different interstellar environments.

x0 - Centroid of 2200 A bump in microns

(default = 4.596)

gamma - Width of 2200 A bump in microns

(default = 0.99)

c3 - Strength of the 2200 A bump

(default = 3.23)

c4 - FUV curvature

(default = 0.41)

c2 - Slope of the linear UV extinction component

(default = -0.824 + 4.717 / R)

c1 - Intercept of the linear UV extinction component

(default = 2.030 - 3.007 * c2)

Notes

  1. The following comparisons between the FM curve and that of Cardelli, Clayton, & Mathis (1989), (see ccm_unred.pro):

      • In the UV, the FM and CCM curves are similar for R < 4.0, but diverge for larger R

      • In the optical region, the FM more closely matches the monochromatic extinction, especially near the R band.

  2. Many sightlines with peculiar ultraviolet interstellar extinction

    can be represented with the FM curve, if the proper value of R(V) is supplied.

REQUIRED MODULES:

scipy, numpy

REVISION HISTORY:

Written W. Landsman Raytheon STX October, 1998 Based on FMRCurve by E. Fitzpatrick (Villanova) Added /LMC2 and /AVGLMC keywords, W. Landsman August 2000 Added ExtCurve keyword, J. Wm. Parker August 2000 Assume since V5.4 use COMPLEMENT to WHERE W. Landsman April 2006 Ported to Python, C. Theissen August 2012

desiutil.dust.ext_odonnell(wave, Rv=3.1)[source]

Return extinction curve from Odonnell (1994), defined in the wavelength range [3030,9091] Angstroms. Outside this range, use CCM (1989).

Parameters:
  • wave – 1D array of vacuum wavelength [Angstroms]

  • Rv – Value of R_V (scalar); default is 3.1

Returns:

1D array of A(lambda)/A(V)

desiutil.dust.extinction_total_to_selective_ratio(band, photsys, match_legacy_surveys=False)[source]

Return the linear coefficient R_X = A(X)/E(B-V) where A(X) = -2.5*log10(transmission in X band), for band X in ‘G’,’R’ or ‘Z’ when photsys = ‘N’ or ‘S’ specifies the survey (BASS+MZLS or DECALS), or for band X in ‘G’, ‘BP’, ‘RP’ when photsys = ‘G’ (when gaia dr2) or for band X in ‘W1’, ‘W2’, ‘W3’, ‘W4’ when photsys is either ‘N’ or ‘S’ E(B-V) is interpreted as SFD.

Parameters:
  • band – ‘G’, ‘R’, ‘Z’, ‘BP’, ‘RP’, ‘W1’, ‘W2’, ‘W3’, or ‘W4’

  • photsys – ‘N’ or ‘S’

Returns:

scalar, total extinction A(band) = -2.5*log10(transmission(band))

desiutil.dust.mwdust_transmission(ebv, band, photsys, match_legacy_surveys=False)[source]

Convert SFD E(B-V) value to dust transmission 0-1 for band and photsys

Parameters:
  • ebv (float or array-like) – SFD E(B-V) value(s)

  • band (str) – ‘G’, ‘R’, ‘Z’, ‘W1’, ‘W2’, ‘W3’, or ‘W4’

  • photsys (str or array of str) – ‘N’ or ‘S’ imaging surveys photo system

Returns:

scalar or array (same as ebv input), Milky Way dust transmission 0-1

If photsys is an array, ebv must also be array of same length. However, ebv can be an array with a str photsys.

Also see dust_transmission which returns transmission vs input wavelength

desiutil.funcfits

Module for fitting simple functions to 1D arrays

J. Xavier Prochaska, UC Santa Cruz Fall 2015

desiutil.funcfits.func_fit(x, y, func, deg, xmin=None, xmax=None, w=None, **kwargs)[source]

Simple function fit to 2 arrays.

Modified code originally from Ryan Cooke (PYPIT).

Parameters:
  • x (ndarray) – Independent data values.

  • y (ndarray) – Dependent data to fit.

  • func (str) – Name of the fitting function: polynomial, legendre, chebyshev.

  • deg (int or dict) – Order of the fit.

  • xmin (float, optional) – Minimum value in the array (or the left limit for a legendre/chebyshev polynomial).

  • xmax (float, optional) – Maximum value in the array (or the left limit for a legendre/chebyshev polynomial).

  • w (ndarray, optional) – Weights to be used in the fitting (weights = 1/sigma).

Returns:

Dictionary describing the Fit including the coefficients.

Return type:

dict

desiutil.funcfits.func_val(x, fit_dict)[source]

Get values from a fit_dict.

Modified code originally from Ryan Cooke (PYPIT).

Parameters:

x (ndarray) – Evaluate the fit at these coordinates.

Returns:

Array containing the values.

Return type:

ndarray

desiutil.funcfits.iter_fit(xarray, yarray, func, order, weights=None, sigma=None, max_rej=None, maxone=True, sig_rej=3.0, initialmask=None, forceimask=False, xmin=None, xmax=None, niter=999, **kwargs)[source]

A “robust” fit with iterative rejection is performed to the xarray, yarray pairs.

Modified code originally from Ryan Cooke (PYPIT).

Parameters:
  • xarray (ndarray) – Independent variable values.

  • yarray (ndarray) – Dependent variable values.

  • func (str) – Name of the fitting function: polynomial, legendre, chebyshev.

  • order (int) – The order of the function to be used in the fitting.

  • sigma (ndarray, optional) – Error in the yvalues. Used only for rejection.

  • weights (ndarray, optional) – Weights to be used in the fitting (weights = 1/sigma).

  • maxone (bool, optional [True]) – If True, only the most deviant point in a given iteration will be removed.

  • sig_rej (float, optional [3.0]) – Confidence interval for rejection.

  • max_rej (int, optional [None]) – Maximum number of points to reject.

  • initialmask (ndarray) – A mask can be supplied as input, these values will be masked for the first iteration. 1 = value masked.

  • forceimask (bool, optional [False]) – If True, the initialmask will be forced for all iterations.

  • niter (int, optional [999]) – Maximum number of iterations.

  • xmin (float) – Minimum value in the array (or the left limit for a legendre/chebyshev polynomial).

  • xmax (float) – Maximum value in the array (or the right limit for a legendre/chebyshev polynomial).

Returns:

The tuple contains a dict containing the fit and a mask array containing masked values.

Return type:

tuple()

desiutil.funcfits.mk_fit_dict(coeff, order, func, xmin=None, xmax=None, **kwargs)[source]

Generate a dict that is formatted for using func_val.

Parameters:
  • coeff (array) – Coefficients of the fit

  • order (int) – The order of the function to be used in the fitting.

  • func (str) – Name of the fitting function: polynomial, legendre, chebyshev.

  • xmin (float) – Minimum value in the array (or the left limit for a legendre/chebyshev polynomial).

  • xmax (float) – Maximum value in the array (or the right limit for a legendre/chebyshev polynomial).

Returns:

The formatted dictionary.

Return type:

dict

desiutil.git

This package contains code for interacting with DESI git products.

desiutil.git.last_tag(owner, repo)[source]

Scan GitHub tags and return the most recent tag.

Parameters:
  • owner (str) – The owner or group in GitHub

  • repo (str) – Name of the product.

Returns:

The most recent tag found on GitHub.

Return type:

str

desiutil.git.version(git='git')[source]

Use git describe to generate a version string.

Parameters:

git (str, optional) – Path to the git executable, if not in PATH.

Returns:

A PEP 386-compatible version string.

Return type:

str

Notes

The version string should be compatible with PEP 386 and PEP 440.

desiutil.iers

Utilities for overriding astropy IERS functionality (astropy.utils.iers), especially for preventing unnecessary downloads of IERS data files in a high performance computing environment.

desiutil.iers.freeze_iers(name='iers_frozen.ecsv', ignore_warnings=True)[source]

Use a frozen IERS table saved with this package.

This should be called at the beginning of a script that calls astropy time and coordinates functions which refer to the UT1-UTC and polar motions tabulated by IERS. The purpose is to ensure identical results across systems and astropy releases, to avoid a potential network download, and to eliminate some astropy warnings.

After this call, the loaded table will be returned by astropy.utils.iers.IERS_Auto.open() and treated like a a normal IERS table by all astropy code. Specifically, this method registers an instance of a custom IERS_Frozen class that inherits from IERS_B and overrides astropy.utils.iers.IERS._check_interpolate_indices() to prevent any IERSRangeError being raised.

See http://docs.astropy.org/en/stable/utils/iers.html for details.

This function returns immediately after the first time it is called, so it it safe to insert anywhere that consistent IERS models are required, and subsequent calls with different args will have no effect.

The desiutil.plots.plot_iers() function is useful for inspecting IERS tables and how they are extrapolated to DESI survey dates.

Parameters:
  • name (str, optional) – Name of the file to load the frozen IERS table from. Should normally be relative and then refers to this package’s data/ directory. Must end with the .ecsv extension.

  • ignore_warnings (bool, optional) – Ignore ERFA and IERS warnings about future dates generated by astropy time and coordinates functions. Specifically, ERFA warnings containing the string “dubious year” are filtered out, as well as AstropyWarnings related to IERS table extrapolation.

desiutil.iers.update_iers(save_name='iers_frozen.ecsv', num_avg=1000)[source]

Update the IERS table used by astropy time, coordinates.

Downloads the current IERS-A table, replaces the last entry (which is repeated for future times) with the average of the last num_avg entries, and saves the table in ECSV format.

This should only be called every few months, e.g., with major releases. The saved file should then be copied to this package’s data/ directory and committed to the git repository.

Requires a network connection in order to download the current IERS-A table. Prints information about the update process.

The desiutil.plots.plot_iers() function is useful for inspecting IERS tables and how they are extrapolated to DESI survey dates.

Parameters:
  • save_name (str, optional) – Name where frozen IERS table should be saved. Must end with the .ecsv extension.

  • num_avg (int, optional) – Number of rows from the end of the current table to average and use for calculating UT1-UTC offsets and polar motion at times beyond the table.

desiutil.install

This package contains code for installing DESI software products.

class desiutil.install.DesiInstall[source]

Code and data that drive the desiInstall script.

baseproduct

The bare name of the product, e.g. “desiutil”.

Type:

str

baseversion

The bare version, without any branches/ qualifiers.

Type:

str

default_nersc_dir_template

The default code and Modules install directory for every NERSC host.

Type:

str

fullproduct

The path to the product including its URL, e.g., “https://github.com/desihub/desiutil”.

Type:

str

github

True if the selected product lives on GitHub.

Type:

bool

is_branch

True if a branch has been selected.

Type:

bool

log

Logging object.

Type:

logging.Logger

nersc

Holds the value of NERSC_HOST, or None if not defined.

Type:

str

options

The parsed command-line options.

Type:

argparse.Namespace

product_url

The URL that will be used to download the code. This differs from fullproduct in that it includes the tag or branch name.

Type:

str

anaconda_version()[source]

Try to determine the exact DESI+Anaconda version from the environment.

Returns:

The DESI+Anaconda version.

Return type:

str

property build_type

Determine the build type.

Returns:

A set containing the detected build types.

Return type:

set

cleanup()[source]

Clean up after the install.

Returns:

Returns True

Return type:

bool

compile_branch()[source]

Certain packages need C/C++ code compiled even for a branch install.

default_nersc_dir(nersc_host=None)[source]

Set the directory where code will reside.

Parameters:

nersc_host (str, optional) – Specify a NERSC host that might be different from the current host.

Returns:

Path to the host-specific install directory.

Return type:

str

get_code()[source]

Actually download the code.

Following the standard order of execution, this is the first method that might actually modify the system (by downloading code).

Raises:

DesiInstallException – If any download errors are detected.

get_extra()[source]

Download any additional data not included in the code repository.

This is done here so that INSTALL_DIR is defined and exists.

get_options(test_args=None)[source]

Parse command-line arguments passed to the desiInstall script.

Parameters:

test_args (list) – Normally, this method is called without arguments, and sys.argv is parsed. Arguments should only be passed for testing purposes.

Returns:

A simple object containing the parsed options. Also, the attribute options is set.

Return type:

argparse.Namespace

get_product_version()[source]

Determine the base product and version information.

Returns:

A tuple containing the base product name and version.

Return type:

tuple()

Raises:

DesiInstallException – If the product and version inputs didn’t make sense.

identify_branch()[source]

If this is not a tag install, determine the branch identity.

Returns:

The full path to the branch code.

Return type:

str

install()[source]

Run setup.py, etc.

install_module()[source]

Process the module file.

Returns:

The text of the processed module file.

Return type:

str

module_dependencies()[source]

Figure out the dependencies and load them.

Returns:

The list of dependencies.

Return type:

list

property nersc_module_dir

The directory that contains Module directories at NERSC.

permissions()[source]

Set permissions on installed software.

prepare_environment()[source]

Prepare the environment for the install.

Returns:

The current working directory. Because we’re about to change it.

Return type:

str

run()[source]

This method wraps all the standard steps of the desiInstall script.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

sanity_check()[source]

Sanity check the options.

Returns:

True if there were no problems.

Return type:

bool

Raises:

DesiInstallException – If any options don’t make sense.

set_install_dir()[source]

Decide on an install directory.

Returns:

The directory selected for installation.

Return type:

str

start_modules()[source]

Set up the modules infrastructure.

Returns:

True if the modules infrastructure was initialized successfully.

Return type:

bool

unlock_permissions()[source]

Unlock installed directories to allow their removal.

Returns:

Status code returned by chmod.

Return type:

int

verify_bootstrap()[source]

Make sure that desiutil/desiInstall was installed with an explicit Python executable path.

For anything besides an initial bootstrap install of desiutil, this function does nothing.

Returns:

Returns True if everything is OK.

Return type:

bool

verify_url(svn='svn')[source]

Ensure that the download URL is valid.

Parameters:

svn (str, optional) – The path to the subversion command.

Returns:

True if everything checked out OK.

Return type:

bool

Raises:

DesiInstallException – If the subversion URL could not be found.

exception desiutil.install.DesiInstallException[source]

The methods of DesiInstall should raise this exception to indicate that the command-line script should exit immediately.

desiutil.install.dependencies(modulefile)[source]

Process the dependencies for a software product.

Parameters:

modulefile (str) – Name of the module file containing dependencies.

Returns:

Returns the list of dependencies. If the module file is not found or there are no dependencies, the list will be empty.

Return type:

list

Raises:

ValueError – If modulefile can’t be found.

desiutil.install.main()[source]

Entry point for the desiInstall script.

Returns:

Exit status that will be passed to sys.exit().

Return type:

int

desiutil.io

Module for I/O related code.

desiutil.io._dtype_size(dtype)[source]

Parse dtype to find its size.

For example, <U14 returns 14.

Parameters:

dtype (numpy.dtype) – Dtype object.

Returns:

The size of the type.

Return type:

int

Notes

This is different from dtype.itemsize, which is number of bytes.

desiutil.io._pick_encoding(table, encoding)[source]

Pick which encoding to use; giving warning if options are in conflict.

Parameters:
  • table (astropy.table.Table) – Table object.

  • encoding (str) – Encoding to use. If None, use table.meta['ENCODING'].

Returns:

The chosen encoding.

Return type:

str

Raises:

UnicodeError – If no enoding could be found at all.

Notes

encoding trumps `table.meta['ENCODING'].

desiutil.io.combine_dicts(dict1, dict2)[source]

Combine two dict objects into one, respecting common keys.

If dict1 and dict2 both have key a, then dict1[a] and dict2[a] must both be dictionaries to recursively merge.

Parameters:
  • dict1 (dict) – First dictionary.

  • dict2 (dict) – Second dictionary.

Returns:

The combined dictionary.

Return type:

dict

Raises:

ValueError – If the values for a common key are not both dict.

desiutil.io.decode_table(data, encoding='ascii', native=True)[source]

Decode byte strings in a table into unicode strings.

Parameters:
  • data (numpy structured array or Table) – Data for conversion.

  • encoding (str, optional) – Encoding to use for converting bytes into unicode; default ‘ascii’; if None, try ENCODING keyword in data instead.

  • native (bool, optional) – If True (default), only decode if native str type is unicode (i.e. python3 but not python2)

Returns:

Decoded data.

Return type:

Table

Notes

encoding option overides data.meta['ENCODING']; use encoding=None to use data.meta['ENCODING'] instead.

desiutil.io.encode_table(data, encoding='ascii')[source]

Encode unicode strings in a table into bytes using numpy.char.encode.

Parameters:
  • data (numpy structured array or Table) – Data for conversion.

  • encoding (str, optional) – Encoding to use for converting unicode to bytes; default ‘ascii’ (FITS and HDF5 friendly); if None, try ENCODING keyword in data instead.

Returns:

Table with unicode columns converted to bytes.

Return type:

Table

Raises:
  • UnicodeEncodeError – If any input strings cannot be encoded using the specified encoding.

  • UnicodeError – If no encoding is given as argument or in table metadata.

Notes

encoding option overides data.meta['ENCODING']; use encoding=None to use data.meta['ENCODING'] instead.

desiutil.io.unlock_file(*args, **kwargs)[source]

Unlock a read-only file, return a file-like object, and restore the read-only state when done. Arguments are the same as open().

Returns:

A file-like object, as returned by open().

Return type:

file-like

Notes

  • This assumes that the user of this function is also the owner of the file. os.chmod() would not be expected to work in any other circumstance.

  • Technically, this restores the original permissions of the file, it does not care what the original permissions were.

  • If the named file does not exist, this function effectively does not attempt to guess what the final permissions of the file would be. In other words, it just does whatever open() would do. In this case it is the user’s responsibilty to change permissions as needed after creating the file.

Examples

>>> with unlock_file('read-only.txt', 'w') as f:
...     f.write(new_data)
desiutil.io.yamlify(obj, debug=False)[source]

Recursively process an object so it can be serialised for yaml. Based on jsonify in linetools.

Note: All string-like keys in dict s are converted to str.

Parameters:
  • obj (object) – Any object.

  • debug (bool, optional) – Print extra information if requested.

Returns:

An object suitable for yaml serialization. For example numpy.ndarray is converted to list, numpy.int64 is converted to int, etc.

Return type:

object

desiutil.log

DESI-specific utility functions that wrap the standard logging module.

This module is intended to support three different logging use patterns:

  1. Just get an easy-to-use, pre-configured logging object.

  2. Easily change the log level temporarily within a function. This is provided by a context manager.

  3. Change the default log level on the command-line. This can actually be accomplished in two ways: the command-line interpreter can call get_logger() with the appropriate level, or the environment variable DESI_LOGLEVEL can be set.

In addition, it is possible to add timestamps and change the delimiter of log messages as needed. See the optional arguments to get_logger().

Examples

Simplest possible use:

>>> from desiutil.log import log
>>> log.info('This is some information.')

This is exactly equivalent to:

>>> from desiutil.log import get_logger
>>> log = get_logger()
>>> log.info('This is some information.')

Temporarily change the log level with a context manager:

>>> from desiutil.log import get_logger, DesiLogContext, DEBUG
>>> log = get_logger()  # defaults to INFO
>>> log.info('This is some information.')
>>> log.debug("This won't be logged.")
>>> with DesiLogContext(log, DEBUG):
...     log.debug("This will be logged.")
>>> log.debug("This won't be logged.")

Create the logger with a different log level:

>>> from desiutil.log import get_logger, DEBUG
>>> if options.debug:
...     log = get_logger(DEBUG)
>>> else:
...     log = get_logger()
class desiutil.log.DesiLogContext(logger, level=None)[source]

Provides a context manager to temporarily change the log level of an existing logging object.

Parameters:
  • logger (logging.Logger) – Logging object.

  • level (int, optional) – The logging level to set. If it is not set, this whole class does nothing.

exception desiutil.log.DesiLogWarning[source]

Warnings related to misconfiguration of the DESI logging object.

desiutil.log._configure_root_logger(timestamp=False, delimiter=':')[source]

Configure a root logger.

Parameters:
  • timestamp (bool, optional) – If True, add a timestamp to the log message.

  • delimiter (str, optional) – Use delimiter to separate fields in the log message (default :).

Returns:

The name of the root logger, suitable for input to logging.getLogger().

Return type:

str

desiutil.log.get_logger(level=None, timestamp=False, delimiter=':')[source]

Returns a default DESI logger.

Parameters:
  • level (int or str, optional) – Set the logging level (default INFO).

  • timestamp (bool, optional) – If True, add a timestamp to the log message.

  • delimiter (str, optional) – Use delimiter to separate fields in the log messages (default :).

Returns:

A logging object configured with the DESI defaults.

Return type:

logging.Logger

Notes

  • If level is not None, that sets the log level, overriding anything else.

  • If level is not set, and if the environment variable DESI_LOGLEVEL exists and has value DEBUG, INFO, WARNING, ERROR or CRITICAL (upper or lower case), that is used to set the log level.

  • If DESI_LOGLEVEL is not set and level is None, the default level is set to INFO.

desiutil.modules

This package contains code for processing and installing Module files.

desiutil.modules._write_module_data(filename, data)[source]

Write and permission-lock Module file data. This is intended to consolidate some duplicated code.

Parameters:
  • filename (str) – The module file to write.

  • data (str) – The data to be written to filename.

desiutil.modules.configure_module(product, version, product_root, working_dir=None, dev=False)[source]

Decide what needs to go in the Module file.

Parameters:
  • product (str) – Name of the product.

  • version (str) – Version of the product.

  • product_root (str) – Directory that contains the installed code.

  • working_dir (str, optional) – The directory to examine. If not set, the current working directory will be used.

  • dev (bool, optional) – If True, interpret the directory as a ‘development’ install, e.g. a trunk or branch install.

Returns:

A dictionary containing the module configuration parameters.

Return type:

dict

desiutil.modules.default_module(module_keywords, module_dir)[source]

Install or update a .version file to set the default Module.

Parameters:
  • module_keywords (dict) – The parameters to use for Module file processing.

  • module_dir (str) – The directory where the Module file should be installed.

Returns:

The text of the processed .version file.

Return type:

str

Note

.version files are always installed with world-read permissions.

desiutil.modules.init_modules(moduleshome=None, method=False, command=False)[source]

Set up the Modules infrastructure.

Parameters:
  • moduleshome (str, optional) – The path containing the Modules init code. If not provided, MODULESHOME will be used.

  • method (bool, optional) – If True the function returned will be suitable for converting into an instance method.

  • command (bool, optional) – If True, return the command used to call Modules, rather than a function.

Returns:

A function that wraps the module() function, and deals with setting sys.path. Returns None if no Modules infrastructure could be found.

Return type:

callable

desiutil.modules.main()[source]

Entry-point for command-line scripts.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desiutil.modules.process_module(module_file, module_keywords, module_dir)[source]

Process a Module file.

Parameters:
  • module_file (str) – A template Module file to process.

  • module_keywords (dict) – The parameters to use for Module file processing.

  • module_dir (str) – The directory where the Module file should be installed.

Returns:

The text of the processed Module file.

Return type:

str

Note

Module files are always installed with world-read permissions.

desiutil.names

This package contains functions for naming ‘things’ in DESI or decoding those names.

desiutil.names.radec_to_desiname(target_ra, target_dec)[source]

Convert the right ascension and declination of a DESI target into the corresponding “DESINAME” for reference in publications. Length of target_ra and target_dec must be the same if providing an array or list. Note that these names are not unique for roughly one percent of DESI targets, so also including TARGETID in publications is highly recommended for uniqueness.

Parameters:
  • target_ra (array of float64) –

    Right ascension in degrees of target object(s). Can be float, double,

    or array/list of floats or doubles

  • target_dec (array of float64) –

    Declination in degrees of target object(s). Can be float, double,

    or array/list of floats or doubles

Returns:

The DESI names referring to the input target RA and DEC’s. Array is the same length as the input arrays.

Return type:

array of str

desiutil.plots

Module for code plots.

class desiutil.plots.MaskedArrayWithLimits(*args, **kwargs)[source]

Masked array with additional vmin, vmax attributes.

This class accepts the same arguments as MaskedArray.

This is not a general-purpose subclass and is only intended to simplify passing vmin, vmax limits from prepare_data() to the plotting utility methods defined in this module.

Parameters:
  • vmin (float, optional) – Minimum value when used for clipping or masking.

  • vmax (float, optional) – Maximum value when used for clipping or masking.

vmin

Minimum value when used for clipping or masking.

Type:

float

vmax

Maximum value when used for clipping or masking.

Type:

float

desiutil.plots.init_sky(projection='mollweide', ra_center=120, galactic_plane_color='red', ecliptic_plane_color='red', ax=None)[source]

Initialize matplotlib axes with a projection of the full sky.

Parameters:
  • projection (str, optional) – Projection to use. Defaults to ‘mollweide’. To show the available projections, call matplotlib.projections.get_projection_names().

  • ra_center (float, optional) – Projection is centered at this RA in degrees. Default is +120°, which avoids splitting the DESI northern and southern regions.

  • galactic_plane_color (color name, optional) – Draw a solid curve representing the galactic plane using the specified color, or do nothing when None.

  • ecliptic_plane_color (color name, optional) – Draw a dotted curve representing the ecliptic plane using the specified color, or do nothing when None.

  • ax (Axes, optional) – Axes to use for drawing this map, or create new axes if None.

Returns:

A matplotlib Axes object. Helper methods projection_ra() and projection_dec() are added to the object to facilitate conversion to projection coordinates.

Return type:

Axes

Notes

If requested, the ecliptic and galactic planes are plotted with zorder set to 20. This keeps them above most other plotted objects, but legends should be set to a zorder higher than this value, for example:

leg = ax.legend(ncol=2, loc=1)
leg.set_zorder(25)
desiutil.plots.plot_grid_map(data, ra_edges, dec_edges, cmap='viridis', colorbar=True, label=None, ax=None, **kwargs)[source]

Plot an array of 2D values using an all-sky projection.

Pass the data array through prepare_data() to select a subset to plot and clip the color map to specified values or percentiles.

This function is similar to plot_healpix_map() but is generally faster and has better handling of RA wrap around artifacts.

Additional keyword parameters will be passed to init_sky().

Parameters:
  • data (array or masked array) – 2D array of data associated with each grid cell, with shape \((N_{\mathrm{RA}}, N_{\mathrm{Dec}})\). Use the output of prepare_data() as a convenient way to specify data cuts and color map clipping.

  • ra_edges (array) – 1D array of \(N_{\mathrm{RA}} + 1\) RA grid edge values in degrees, which must span the full circle, i.e., ra_edges[0] == ra_edges[-1] - 360. The RA grid does not need to match the edges of the projection, in which case any wrap-around cells will be duplicated on both edges.

  • dec_edges (array) – 1D array of \(N_{\mathrm{Dec}} + 1\) Dec grid edge values in degrees. Values are not required to span the full range [-90, +90].

  • cmap (colormap name or object, optional) – Matplotlib colormap to use for mapping data values to colors.

  • colorbar (bool, optional) – Draw a colorbar below the map when True.

  • label (str, optional) – Label to display under the colorbar. Ignored unless colorbar is True.

  • ax (Axes, optional) – Axes to use for drawing this map, or create default axes using init_sky() when None.

Returns:

The axis object used for the plot.

Return type:

Axes

desiutil.plots.plot_healpix_map(data, nest=False, cmap='viridis', colorbar=True, label=None, ax=None, **kwargs)[source]

Plot a healpix map using an all-sky projection.

Pass the data array through prepare_data() to select a subset to plot and clip the color map to specified values or percentiles.

This function is similar to plot_grid_map() but is generally slower at high resolution and has less elegant handling of pixels that wrap around in RA, which are not drawn.

Requires that matplotlib and healpy are installed.

Additional keyword parameters will be passed to init_sky().

Parameters:
  • data (array or masked array) – 1D array of data associated with each healpix. Must have a size that exactly matches the number of pixels for some NSIDE value. Use the output of prepare_data() as a convenient way to specify data cuts and color map clipping.

  • nest (bool, optional) – If True, assume NESTED pixel ordering. Otheriwse, assume RING pixel ordering.

  • cmap (colormap name or object, optional) – Matplotlib colormap to use for mapping data values to colors.

  • colorbar (bool, optional) – Draw a colorbar below the map when True.

  • label (str, optional) – Label to display under the colorbar. Ignored unless colorbar is True.

  • ax (Axes, optional) – Axes to use for drawing this map, or create default axes using init_sky() when None.

Returns:

The axis object used for the plot.

Return type:

Axes

desiutil.plots.plot_iers(which='auto', num_points=500, save=None)[source]

Plot IERS data from 2015-2025.

Plots the UT1-UTC time offset and polar x,y motions over a 10-year period that includes the DESI survey, to demonstrate the time ranges when different sources (IERS-A, IERS-B) are used and where values are predicted then fixed.

This function is primarily intended to document and debug the desiutil.iers.freeze_iers() function.

Requires that the matplotlib package is installed.

Parameters:
  • which ({'auto', 'A', 'B', 'frozen'}) – Select which IERS table source to use. The default ‘auto’ matches the internal astropy default. Use ‘A’ or ‘B’ to force the source to be either the latest IERS-A table (which will be downloaded), or the IERS-B table packaged with the current version of astropy. The ‘frozen’ option calls freeze_iers().

  • num_points (int, optional) – The number of times covering 2015-25 to calculate and plot.

  • save (str, optional) – Name of file where plot should be saved. Format is inferred from the extension.

Returns:

Tuple (figure, axes) returned by plt.subplots().

Return type:

tuple()

desiutil.plots.plot_sky_binned(ra, dec, weights=None, data=None, plot_type='grid', max_bin_area=5, clip_lo=None, clip_hi=None, verbose=False, cmap='viridis', colorbar=True, label=None, ax=None, return_grid_data=False, **kwargs)[source]

Show objects on the sky using a binned plot.

Bin values either show object counts per unit sky area or, if an array of associated data values is provided, mean data values within each bin. Objects can have associated weights.

Requires that matplotlib is installed. When plot_type is “healpix”, healpy must also be installed.

Additional keyword parameters will be passed to init_sky().

Parameters:
  • ra (array) – Array of object RA values in degrees. Must have the same shape as dec and will be flattened if necessary.

  • dec (array) – Array of object Dec values in degrees. Must have the same shape as ra and will be flattened if necessary.

  • weights (array, optional) – Optional of weights associated with each object. All objects are assumed to have equal weight when this is None.

  • data (array, optional) – Optional array of scalar values associated with each object. The resulting plot shows the mean data value per bin when data is specified. Otherwise, the plot shows counts per unit sky area.

  • plot_type ({'grid', 'healpix'}) – Must be either ‘grid’ or ‘healpix’, and selects whether data in binned in healpix or in (sin(Dec), RA).

  • max_bin_area (float, optional) – The bin size will be chosen automatically to be as close as possible to this value but not exceeding it.

  • clip_lo (float or str, optional) – Clipping is applied to the plot data calculated as counts / area or the mean data value per bin. See prepare_data() for details.

  • clip_hi (float or str, optional) – Clipping is applied to the plot data calculated as counts / area or the mean data value per bin. See prepare_data() for details.

  • verbose (bool, optional) – Print information about the automatic bin size calculation.

  • cmap (colormap name or object, optional) – Matplotlib colormap to use for mapping data values to colors.

  • colorbar (bool, optional) – Draw a colorbar below the map when True.

  • label (str, optional) – Label to display under the colorbar. Ignored unless colorbar is True.

  • ax (Axes, optional) – Axes to use for drawing this map, or create default axes using init_sky() when None.

  • return_grid_data (bool, optional) – If True, return (ax, grid_data) instead of just ax.

Returns:

The axis object used for the plot, and the grid_data if return_grid_data is True.

Return type:

Axes or (ax, grid_data)

desiutil.plots.plot_sky_circles(ra_center, dec_center, field_of_view=3.2, data=None, cmap='viridis', facecolors='skyblue', edgecolor='none', colorbar=True, colorbar_ticks=None, label=None, ax=None, **kwargs)[source]

Plot circles on an all-sky projection.

Pass the optional data array through prepare_data() to select a subset to plot and clip the color map to specified values or percentiles.

Requires that matplotlib is installed.

Additional keyword parameters will be passed to init_sky().

Parameters:
  • ra_center (array) – 1D array of RA in degrees at the centers of each circle to plot.

  • dec_center (array) – 1D array of DEC in degrees at the centers of each circle to plot.

  • field_of_view (array) – Full sky openning angle in degrees of the circles to plot. The default is appropriate for a DESI tile.

  • data (array, optional) – 1D array of data associated with each circle, used to set its facecolor.

  • cmap (colormap name, optional) – Matplotlib colormap to use for mapping data values to colors. Ignored unless data is specified.

  • facecolors (matplotlib color or array of colors, optional) – Ignored when data is specified. An array must have one entry per circle or a single value is used for all circles.

  • edgecolor (matplotlib color, optional) – The edge color used for all circles. Use ‘none’ to hide edges.

  • colorbar (bool, optional) – Draw a colorbar below the map when True and data is provided.

  • colorbar_ticks (list, optional) – Use the specified colorbar ticks or determine them automatically when None.

  • label (str) – Label to display under the colorbar. Ignored unless a colorbar is displayed.

  • ax (Axes, optional) – Axes to use for drawing this map, or create default axes using init_sky() when None.

Returns:

The axis object used for the plot.

Return type:

Axes

desiutil.plots.plot_slices(x, y, x_lo, x_hi, y_cut, num_slices=5, min_count=100, axis=None, set_ylim_from_stats=True, scatter=True)[source]

Scatter plot with 68, 95 percentiles superimposed in slices. Modified from code written by D. Kirkby

Requires that the matplotlib package is installed.

Parameters:
  • x (array of float) – X-coordinates to scatter plot. Points outside [x_lo, x_hi] are not displayed.

  • y (array of float) – Y-coordinates to scatter plot. Y values are assumed to be roughly symmetric about zero.

  • x_lo (float) – Minimum value of x to plot.

  • x_hi (float) – Maximum value of x to plot.

  • y_cut (float) – The target maximum value of \(|y|\). A dashed line at this value is added to the plot, and the vertical axis is clipped at \(|y|\) = 1.25 * y_cut (but values outside this range are included in the percentile statistics).

  • num_slices (int, optional) – Number of equally spaced slices to divide the interval [x_lo, x_hi] into.

  • min_count (int, optional) – Do not use slices with fewer points for superimposed percentile statistics.

  • axis (matplotlib.axes.Axes, optional) – Uses the current axis if this is not set.

  • set_ylim_from_stats (bool, optional) – Set ylim of plot from 95% stat.

  • scatter (bool, optional) – Show the data as a scatter plot. Best to limit to small datasets

Returns:

The Axes object used in the plot.

Return type:

matplotlib.axes.Axes

desiutil.plots.prepare_data(data, mask=None, clip_lo=None, clip_hi=None, save_limits=False)[source]

Prepare array data for color mapping.

Data is clipped and masked to be suitable for passing to matplotlib routines that automatically assign colors based on input values.

Parameters:
  • data (array or masked array) – Array of data values to assign colors for.

  • mask (array of bool or None) – Array of bools with same shape as data, where True values indicate values that should be ignored when assigning colors. When None, the mask of a masked array will be used or all values of an unmasked array will be used.

  • clip_lo (float or str) – Data values below clip_lo will be clipped to the minimum color. If clip_lo is a string, it should end with “%” and specify a percentile of un-masked data to clip below.

  • clip_hi (float or str) – Data values above clip_hi will be clipped to the maximum color. If clip_hi is a string, it should end with “%” and specify a percentile of un-masked data to clip above.

  • save_limits (bool) – Save the calculated lo/hi clip values as attributes vmin, vmax of the returned masked array. Use this flag to indicate that plotting functions should use these vmin, vmax values when mapping the returned data to colors.

Returns:

Masked numpy array with the same shape as the input data, with any input mask applied (or copied from an input masked array) and values clipped to [clip_lo, clip_hi].

Return type:

masked array

Examples

If no optional parameters are specified, the input data is returned with only non-finite values masked:

>>> data = np.arange(5.)
>>> prepare_data(data)
masked_array(data = [0.0 1.0 2.0 3.0 4.0],
             mask = [False False False False False],
       fill_value = 1e+20)

Any mask selection is propagated to the output:

>>> prepare_data(data, data == 2)
masked_array(data = [0.0 1.0 -- 3.0 4.0],
             mask = [False False  True False False],
       fill_value = 1e+20)

Values can be clipped by specifying any combination of percentiles (specified as strings ending with “%”) and numeric values:

>>> prepare_data(data, clip_lo='25%', clip_hi=3.5)
masked_array(data = [1.0 1.0 2.0 3.0 3.5],
             mask = [False False False False False],
       fill_value = 1e+20)

Clipped values are also masked when the clip value or percentile is prefixed with “!”:

>>> prepare_data(data, clip_lo='!25%', clip_hi=3.5)
masked_array(data = [-- 1.0 2.0 3.0 3.5],
             mask = [ True False False False False],
       fill_value = 1e+20)

An input masked array is passed through without any copying unless clipping is requested:

>>> masked = numpy.ma.arange(5)
>>> masked is prepare_data(masked)
True

Use the save_limits option to store the clipping limits as vmin, vmax attributes of the returned object:

>>> d = prepare_data(data, clip_lo=1, clip_hi=10, save_limits=True)
>>> d.vmin, d.vmax
(1.0, 10.0)

These attributes can then be used by plotting routines to fix the input range used for colormapping, independently of the actual range of data.

desiutil.redirect

Utilities for redirecting stdout / stderr to files.

desiutil.redirect._get_libc()[source]

Helper function to import libc once.

desiutil.redirect.stdouterr_redirected(to=None, comm=None)[source]

Redirect stdout and stderr to a file.

The general technique is based on:

http://stackoverflow.com/questions/5081657 http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

If the optional communicator is specified, then each process redirects to a different temporary file. Upon exit from the context the rank zero process concatenates these in order to the final file result.

If the enclosing code raises an exception, the traceback is printed to the log file.

Parameters:
  • to (str) – The output file name.

  • comm (mpi4py.MPI.Comm) – The optional MPI communicator.

desiutil.setup

This module supplies desi_update_version, which simplifies setting and updating version strings in Python packages.

This module also supports deprecated python setup.py <command> actions.

For historical reasons, this module is retains an outdated name setup.py.

class desiutil.setup.DesiAPI(dist, **kw)[source]

Generate an api.rst file.

finalize_options()[source]

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options()[source]

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

run()[source]

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

class desiutil.setup.DesiModule(dist, **kw)[source]

Allow users to install module files with python setup.py module_file.

finalize_options()[source]

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options()[source]

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

run()[source]

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

class desiutil.setup.DesiTest(dist, **kw)[source]

Add coverage to test commands.

finalize_options()[source]

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options()[source]

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

class desiutil.setup.DesiVersion(dist, **kw)[source]

Allow users to easily update the package version with python setup.py version.

finalize_options()[source]

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options()[source]

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

run()[source]

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

desiutil.setup.find_version_directory(productname)[source]

Return the name of a directory containing version information.

Looks for files in the following places:

  • py/productname/_version.py

  • productname/_version.py

Parameters:

productname (str) – The name of the package.

Returns:

Name of a directory that can or does contain version information.

Return type:

str

Raises:

IOError – If no valid directory can be found.

desiutil.setup.get_version(productname)[source]

Get the value of __version__ without having to import the module.

Parameters:

productname (str) – The name of the package.

Returns:

The value of __version__.

Return type:

str

desiutil.setup.main()[source]

Entry-point for command-line scripts.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desiutil.setup.update_version(productname, tag=None)[source]

Update the _version.py file.

Parameters:
  • productname (str) – The name of the package.

  • tag (str, optional) – Set the version to this string, unconditionally.

Raises:

IOError – If the repository type could not be determined.

desiutil.sklearn

Useful functions copied from the scikit-learn Python package.

class desiutil.sklearn.GaussianMixtureModel(weights, means, covars, covtype='full')[source]

Read and sample from a pre-defined Gaussian mixture model.

Parameters:
  • weights (numpy.ndarray) – A 1D array of weights. The length of the array is the number of components

  • means (numpy.ndarray) – A 2D array of means. The number of rows is the number of components. The number of columns is the number of dimensions.

  • covars (numpy.ndarray) – A 3D array of covariances. The first dimension is the number of components. Each component has a 2D array with size given by the number of dimensions.

  • covtype (str, optional) – Type of covariance. Defaults to ‘full’.

static load(filename)[source]

Load a model from a file.

Parameters:

filename (str) – The name of the file to load from.

Returns:

The model that was in filename.

Return type:

desiutil.sklearn.GaussianMixtureModel

sample(n_samples=1, random_state=None)[source]

Sample from a model.

Parameters:
  • n_samples (int, optional) – Number of samples to return, default 1.

  • random_state (numpy.random.RandomState, optional) – A random state object.

Returns:

An array containing the samples.

Return type:

numpy.ndarray

Raises:

ValueError – If the covariance type is unknown.

static save(model, filename)[source]

Save a model to a file.

Parameters:

desiutil.stats

Just contains a dead-simple wrapper on numpy.percentile().

desiutil.stats.perc(x, per=68.2)[source]

Calculate the percentile bounds of a distribution, i.e. for per=68, the code returns the upper and lower bounds that encompass 68 percent of the distribution.

Uses simple interpolation.

Parameters:
  • x (numpy.ndarray) – numpy array of values

  • per (float, optional) – Percentile for the calculation [0-100].

Returns:

Value at lower, value at upper.

Return type:

numpy.ndarray

desiutil.svn

This package contains code for interacting with DESI svn products.

desiutil.svn.last_revision()[source]

Get the svn revision number.

Returns:

The latest svn revision number. A revision number of 0 indicates an error of some kind.

Return type:

str

Notes

This assumes that you’re running python setup.py version in an svn checkout directory.

desiutil.svn.last_tag(tags, username=None)[source]

Scan an SVN tags directory and return the most recent tag.

Parameters:
  • tags (str) – A URL pointing to an SVN tags directory.

  • username (str, optional) – If set, pass the value to SVN’s --username option.

Returns:

The most recent tag found in tags.

Return type:

str

desiutil.svn.version(productname, url=None)[source]

Returns the version of a package.

Parameters:
  • productname (str) – The name of the package.

  • url (str, optional) – If the product is not defined in the known_products file, the URL can be set this way.

Returns:

A PEP 386-compatible version string.

Return type:

str

Notes

The version string should be compatible with PEP 386 and PEP 440.

desiutil.timer

Provides Timer class to standardize reporting of algorithm and I/O timing.

The timer.Timer class is intended for reporting timing of high-level events that take seconds to minutes, e.g. reporting the timing for the input, calculation, and output steps of a spectroscopic pipeline script. It is not intended for detailed profiling of every possible function for the purposes of optimization studies (use python cProfile for that); nor is it intended for timing small functions deep in the call stack that could result in N>>1 messages when run in parallel.

Example:

from desiutil.timer import Timer
t = Timer()
t.start('blat')

#- Use context manager for timing simple steps, e.g. one-liners
with t.time('blat.input'):
    stuff = io.read(filename)

#- Or use full start/stop if you prefer
t.start('blat.algorithm')
results = calculate(stuff)
t.stop('blat.algorithm')

with t.time('blat.output'):
    io.write(results)

#- Stop outer timer; Note that named timers can be nested or interleaved
t.stop('blat')

#- Print a json report of the timing
print(t.report())

This module has the philosophy that adding timing information should not crash your code, even if the timer is used incorrectly, e.g. starting or stopping a timer multiple times, stopping a timer that was never started, or forgetting to stop a timer before asking for a report. These print additional warnings or error messages, but don’t raise exceptions.

class desiutil.timer.Timer(silent=False)[source]

A basic timer class for standardizing reporting of algorithm and I/O timing

TIMER:<START|STOP>:<filename>:<lineno>:<funcname>: <message>

_prefix(step)[source]

Return standard prefix string for timer reporting

Parameters:

step (str) – timing step, e.g. “START” or “STOP”

_print(level, message)[source]

Print message with timing level prefix if not self.silent

Parameters:
  • level (str) – timing level (‘START’, ‘STOP’, ‘WARNING’, …)

  • message (str) – message to print after standardized prefix

cancel(name)[source]

Cancel timer name and remove from timing log

report()[source]

Return a json dump of self.timers, with start/stop as ISO-8601 strings

Implicitly stops any timers that are still running

Use Timer.timers for access as a dictionary, where start/stop are seconds elapsed since the epoch (1970-01-01T00:00:00 UTC on Unix).

start(name, starttime=None)[source]

Start a timer name (str); prints TIMER-START message

Parameters:

name (str) – name of timer to stop

Options:

starttime (str or float): start time to use instead of now

If name is started multiple times, the last call to .start is used for the timestamp.

Tries to parse starttime in this order:

  1. (float) Unix time-since epoch

  2. (str) ISO-8601

  3. (str) Unix date cmd, e.g. “Mon Sep 21 20:09:48 PDT 2020”

stop(name, stoptime=None)[source]

Stop timer name (str); prints TIMER-STOP message

Parameters:

name (str) – name of timer to stop

Options:

stoptime (str or float): stop time to use instead of now

Returns time since start in seconds, or -1.0 if name wasn’t started

Note: this purposefully does not raise an exception if called with a name that wasn’t started, under the philosophy that adding timing statements shouldn’t crash code, even if used incorrectly.

If a timer is stopped multiple times, the final stop and duration are timestamped as the last call to Timer.stop.

Tries to parse stoptime in this order:

  1. (float) Unix time-since epoch

  2. (str) ISO-8601

  3. (str) Unix date cmd, e.g. “Mon Sep 21 20:09:48 PDT 2020”

stopall()[source]

Stop any timers that have not yet been individually stopped

time(name)[source]

Context manager for timing a code snippet.

Usage:

t = Timer()
with t.time('blat'):
    blat()

is equivalent to:

t = Timer()
t.start('blat')
blat()
t.stop('blat')
timer_seconds2iso8601()[source]

Return copy self.timers with start/stop as ISO-8601 strings

Does not stop any running timers.

desiutil.timer.compute_stats(timerlist)[source]

Compute timer min/max/mean/median stats

Parameters:

timerlist – list of Timer objects

Returns: dict[timername][…] with keys for start/stop/duration.min/max/mean/median

Different Timers can have different named subtimers

desiutil.timer.parsetime(t)[source]

Parse time as int,float,str(int),str(float),ISO-8601, or Unix date

If t is None, return time.time()

Returns t as float Unix seconds since epoch timestamp

desiutil.timer.timestamp2isotime(timestamp)[source]

Return seconds since epoch timestamp as ISO-8601 string