Dataclasses

A set of classes that holds dataset configuration data.

Each class is based on a dictionary and only contains simple elements for easy serialization to json (or other formats). A full dataset configuration can be had by serializing the Job class.

The convert method of each class will turn any regular dictionary objects into special dataclasses objects.

The valid method of each class will test the validity of the data to be an actual dataset config.

The output method of each class will create json with info on each dataclass, to be used in javascript.

Configuration Objects

The building blocks of how datasets, jobs, and tasks are configured.

class iceprod.core.dataclasses.Job(*args, **kwargs)

Holds all information about a running job.

If the options are empty, this is the same as a dataset configuration.

Variables:

• version – 3

• options – {} – a dict of parameters to pass to the task runner

• steering – None

• tasks – []

• difplus – None

• description –

  ””

• categories – []

plural = 'Jobs'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Steering(*args, **kwargs)

Holds all information that goes in the steering section of a configuration.

Variables:

• parameters – {}

• batchsys – None

• system – {} – just specialized parameters

• resources – []

• data – []

plural = 'Steering'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses._TaskCommon(*args, **kwargs)

Holds common attributes used by task, tray, module.

Variables:

• name –

  ’’

• resources – []

• data – []

• classes – []

• parameters – {}

convert()

valid()

class iceprod.core.dataclasses.Task(*args, **kwargs)

Bases: _TaskCommon

Holds all information about a task.

Variables:

• depends – [] – a list of task names

• batchsys – None

• trays – []

• requirements – {} – a dict of requirements

• task_files – False – whether to use the task files API

plural = 'Tasks'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Tray(*args, **kwargs)

Bases: _TaskCommon

Holds all information about a tray.

Variables:

• iterations – 1

• modules – []

plural = 'Trays'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Module(*args, **kwargs)

Bases: _TaskCommon

Holds all information about a module.

Variables:

• running_class – None – the python class or function to call

• src – None – src of class or script

• args – None – args to give to class or src if not an iceprod module

• env_shell – None – src of script which sets env and calls arg

• env_clear – True – clear the env before calling the module (calls env_shell after clearing, if defined)

• configs – None – any json config files that should be written (format is {filename: data})

Note that env_clear should be used carefully, as it wipes out any loaded classes.

plural = 'Modules'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Class(*args, **kwargs)

A class object, downloaded from a url.

Variables:

• name – None – required

• src – None – if downloaded from url

• resource_name – None – if present in resource object

• recursive – False

• libs – None – if more than default lib directory

• env_vars – None

plural = 'Classes'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses._ResourceCommon(*args, **kwargs)

Holds common attributes used by Resource and Data.

Variables:

• remote –

  ’’

• local –

  ’’

• compression – False

compression_options = [False, True, 'none', 'gzip', 'gz', 'bzip', 'bz2', 'lzma']

convert()

valid()

do_transfer()

Test if we should actually transfer the file.

class iceprod.core.dataclasses.Resource(*args, **kwargs)

Bases: _ResourceCommon

A resource object, representing a file to download.

Variables:

arch – None

plural = 'Resources'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Data(*args, **kwargs)

Bases: _ResourceCommon

A data object, representing input and/or output of data.

Variables:

• type – ‘permanent’ – required

• movement – ‘both’ – required

plural = 'Data'

type_options = ['permanent', 'tray_temp', 'task_temp', 'job_temp', 'dataset_temp', 'site_temp']

movement_options = ['input', 'output', 'both']

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

storage_location(env)

Get storage location from the environment.

Parameters:

env – environment

Returns:

storage location as a string, or raises Exception

filecatalog(env)

Get filecatalog from the environment.

Parameters:

env – environment

Returns:

FileCatalog object, or raises Exception

Metadata Objects

Metadata for NSF requirements

class iceprod.core.dataclasses.DifPlus(*args, **kwargs)

A DifPlus object.

Variables:

• dif – None

• plus – None

plural = 'DifPlus'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Dif(*args, **kwargs)

A Dif object.

Variables:

• entry_id – None

• entry_title – None

• parameters – ‘ ‘

• iso_topic_category – ‘geoscientificinformation’

• data_ceter – None

• summary – ‘ ‘

• metadata_name – ‘[CEOS IDN DIF]’

• metadata_version – ‘9.4’

• personnel – []

• sensor_name – ‘ICECUBE’

• source_name – ‘SIMULATION’

• dif_creation_date – time.strftime(“%Y-%m-%d”)

plural = 'Dif'

valid_parameters = ['SPACE SCIENCE > Astrophysics > Neutrinos', 'SPACE SCIENCE > Astrophysics > Neutrinos > Atmospheric', 'SPACE SCIENCE > Astrophysics > Neutrinos > Extraterrestrial Point Source', 'SPACE SCIENCE > Astrophysics > Neutrinos > Gamma Ray Burst', 'SPACE SCIENCE > Astrophysics > Neutrinos > WIMPS', 'SPACE SCIENCE > Astrophysics > Neutrinos > Diffuse Source', 'SPACE SCIENCE > Astrophysics > Neutrinos > Extreme High Energy', 'SPACE SCIENCE > Astrophysics > Neutrinos > Super Nova', 'SPACE SCIENCE > Astrophysics > Neutrinos > Cosmic Ray Muon Component', 'SPACE SCIENCE > Astrophysics > Neutrinos > Tau', 'SPACE SCIENCE > Astrophysics > Neutrinos > Cascades', 'SPACE SCIENCE > Astrophysics > Neutrinos > Galactic Plane', 'SPACE SCIENCE > Astrophysics > Cosmic Rays', 'SPACE SCIENCE > Astrophysics > Cosmic Rays > Composition', 'SPACE SCIENCE > Astrophysics > Cosmic Rays > Air Shower', 'SPACE SCIENCE > Astrophysics > Cosmic Rays > Cosmic Ray Muons', 'SPACE SCIENCE > Astrophysics > Cosmic Rays > Moon Shadow', 'SPACE SCIENCE > Engineering > Sensor Characteristics', 'SPACE SCIENCE > Engineering > Sensor Characteristics > Photomultiplier Tubes', 'SPACE SCIENCE > Engineering > Sensor Characteristics > Digital Optical Modules', 'EARTH SCIENCE > Cryosphere > Glaciers/Ice Sheets', 'EARTH SCIENCE > Cryosphere > Glaciers/Ice Sheets > Hot Water Drilling', 'EARTH SCIENCE > Cryosphere > Glaciers/Ice Sheets > Hot Water Drilling > Hole Drilling', 'EARTH SCIENCE > Cryosphere > Glaciers/Ice Sheets > Hot Water Drilling > Hole Refreeze']

valid_source_name = {'EXPERIMENTAL': 'Data with an instrumentation based source', 'SIMULATION': 'Data which are numerically generated'}

valid_sensor_name = {'AMANDA': 'Antarctic Muon and Neutrino Detector Array', 'AMANDA-A': 'Prototype Antarctic Muon and Neutrino Detector Array', 'EHWD': 'Enhanced Hot Water Drill', 'GASP': 'Gamma Astronomy-South Pole', 'ICECUBE': 'IceCube', 'ICETOP': 'IceTop', 'RICE': 'Radio Ice Cherenkov Experiment', 'RPSC-MET': 'Raytheon Polar Services Corporation Meteorology', 'SPASE-1': 'South Pole Air Shower Experiment 1', 'SPASE-2': 'South Pole Air Shower Experiment 2', 'SPTR': 'South Pole TDRSS Relay', 'VULCAN': 'South Pole Air-Cherenkov Telescope'}

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Plus(*args, **kwargs)

A Plus object.

Variables:

• start – None

• end – None

• category – None

• subcategory – None

• run_number – None

• i3db_key – None

• simdb_key – None

• project – [] – [{name: version}]

• steering_file – None

• log_file – None

• command_line – None

plural = 'Plus'

valid_category = ['unclassified', 'generated', 'propagated', 'unbiased', 'filtered', 'calibration', 'monitoring', 'webcam', 'hole', 'TestDAQ', 'FAT', 'log', 'upmu', 'minbias', 'cascades', 'high-energy', 'wimp', 'GRB']

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.Personnel(*args, **kwargs)

A Personnel object.

Variables:

• role – None

• first_name – None

• last_name – None

• email – None

plural = 'Personnel'

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()

class iceprod.core.dataclasses.DataCenter(*args, **kwargs)

A Data Center object.

Variables:

• name – None

• personnel – []

plural = 'DataCenter'

valid_names = ['UWI-MAD/A3RI > Antarctic Astronomy and Astrophysics Research Institute, University of Wisconsin, Madison']

output()

Output dict with values and (optionally) the object name for new objects.

convert()

valid()