Tray segments¶
In addition to services and modules, IceTray has a concept of a tray segment, which is an automatic configuration function that adds one or more modules, services and/or other segments to a I3Tray. These segments would be defined by module experts, live within the project source tree and be released with project and meta-projects. This is designed to reduce configuration duplication (and remove the need to cut-paste from other scripts) and to simplify scripts in several cases (by building massive data processing scripts from segments of segments). With this in mind, segments provide a way to:
Logically connect sets of services and modules that together perform a single task (e.g. Gulliver-based reconstructions or pulse extraction with NFE)
Build large segments of standard processing (e.g. modularization of bulk filtering and processing scripts)
Support modules with widely used alternate set of defaults for some specific use (e.g. when different settings are required for real data vs MC data)
Note
When creating a segment for use in production, adhere to the “Give one entity, one cohesive responsibility.” standard. We’ve seen several cases of segments taking on too much responsibility.
A Basic Tray Segment Example¶
Tray segments are defined by a python function, such as the following:
@icetray.traysegment
def small_segment(tray, name, NDumps=1):
for i in range(0, NDumps):
tray.AddModule('Dump', name + ('_dump_%d' % i))
This can then be added to a tray by using the following in your python scripts:
tray.AddSegment(small_segment, 'dumps', NDumps=4)
A more realistic example from the payload-parsing project:
@icetray.traysegment
def I3DOMLaunchExtractor(tray, name, OMKey2MBIDXML=os.path.expandvars("$I3_SRC") + "/phys-services/resources/mainboard_ids.xml", BufferID="I3DAQData", ExceptionId="I3DAQDecodeException"):
if OMKey2MBIDXML != None:
tray.AddService("I3XMLOMKey2MBIDFactory", name + "_OmKey2MbId", InFile=OMKey2MBIDXML)
tray.AddService("I3PayloadParsingEventDecoderFactory", name + "_EventDecoder")
tray.AddModule("I3FrameBufferDecode", name + "_fbdecoder", BufferID=BufferID, ExceptionId=ExceptionId)
The decorator (@icetray.traysegment) marks the function as a tray segment, allowing it to be found later (for example, by I3Tray and icetray-inspect). Note that this also allows you to have multiple tray segments in one file. The arguments of the segment are specified as keywords (e.g. OMKey2MBIDXML), with default values following the = sign. This segment can be added to your tray like so:
tray.AddSegment(payload_parsing.I3DOMLaunchExtractor, "extract")
How to find Tray Segments¶
For tasks connected to a single module, relevant standard configurations should be added to the module’s python directory (more details below).
Since tray segments are not stored in a single place but distributed throughout the source projects, finding them could be an issue. To address this, icetray-inspect also reports all available tray segments as well:
blaufuss@squeak[build]% icetray-inspect --no-modules --no-services payload-parsing
-----------------------------------------------------------------------------
icecube.payload_parsing.I3DOMLaunchExtractor (I3Tray segment)
Extract DOMLaunches and triggers from raw DAQ data.
Parameters:
BufferID
Description : Name of the raw data buffer in the frame.
Default : 'I3DAQData'
ExceptionId
Description : Name of the boolean to put in the frame when an error occurs.
Default : 'I3DAQDecodeException'
OMKey2MBIDXML
Description : Path to XML file mapping mainboard IDs to OMKeys.
Default : '/data/i3home/blaufuss/icework/offline-software/trunk/src/phys-services/resources/mainboard_ids.xml'
-----------------------------------------------------------------------------
You can get a list of all available tray-segments from icetray-inspect:
blaufuss@squeak[build]% icetray-inspect --no-modules --no-services -a ./lib
Alternative configurations in Segments¶
For case 3 above (modules with alternate defaults), although still
a tray segment, can be handled very quickly using module_altconfig()
.
For example, adding an alternate configuration for I3WaveCalibrator to be
used with DOMSimulator-based simulation can be done as follows:
DOMSimulatorCalibrator = icetray.module_altconfig('I3WaveCalibrator',
DOMsimulatorWorkArounds=True, FADCSaturationMargin=1)
This can then be added to a tray using I3Tray.AddSegment()
in the same
way:
tray.AddSegment(WaveCalibrator.DOMSimulatorCalibrator, 'calibrator')
When using module_altconfig()
., those parameters specified in the arguments to module_altconfig()
. override the module’s defaults. These can be overridden again in I3Tray.AddSegment()
and any other arguments to the original module can also be specified there.
These alternative configurations also are reported by icetray-inspect:
blaufuss@squeak[build_deb(I3)]% icetray-inspect --no-modules --no-services WaveCalibrator
*** WaveCalibrator ***
-------------------------------------------------------------------------------
icecube.WaveCalibrator.DOMSimulatorCalibrator (I3Tray segment)
Alternate configuration for I3WaveCalibrator
Overridden defaults:
DOMsimulatorWorkArounds=True
FADCSaturationMargin=1
Writing a Segment for your Modules¶
Segments can be put in any location accessible to python, but those that are part of Icetray projects typically live in the module’s pybdindings. They are then imported from the module’s namespace.
Modules without any python component will need to add a PYTHON_DIR directive to their project’s CMakeLists.txt:
i3_project(myproject PYTHON_DIR python)
and an __init__.py file to a new python subdirectory like this:
from icecube import icetray
import os
icetray.load('myproject', False)
Short segments can be added directly to this file, but longer ones should be added to another .py file, and then brought into the project’s namespace with an import command in __init__.py.
There is one additional variant of the icetray.traysegment decorator that can be useful in some circumstances. This is an example from the hdfwriter project:
@icetray.traysegment_inherit(tableio.I3TableWriter)
def I3HDFWriter(tray, name, Output=None, **kwargs):
"""Tabulate data to an HDF5 file.
:param Output: Path to output file
"""
tabler = I3HDFTableService(Output)
tray.AddModule(tableio.I3TableWriter, name, TableService=tabler,
**kwargs)
This is a wrapper around I3TableWriter
that adds one additional service. Any options besides Output are passed through to I3TableWriter
via the kwargs parameter, but what those available options are would not ordinarily show up in the output of icetray-inspect. The use of traysegment_inherit()
here makes no functional changes, but causes the options taken by I3TableWriter
to be appended to the segment’s own options (in this case, Output) when shown in icetray-inspect
.
Segments of Segments¶
Segments can of course include tray.AddSegment()
calls within them, and in this way large processing chains can be built up. A working example from the IC86 L2 processing:
@icetray.traysegment
def RawDataToPulses(tray, name, superdstname = 'I3SuperDST',
pulses='UncleanedPulses'):
# Raw Data
tray.AddSegment(payload_parsing.I3DOMLaunchExtractor, name + '_launches')
tray.AddModule('I3WaveCalibrator', name + '_wavecal')
tray.AddModule('I3Wavedeform', name + '_wavedeform', Output=pulses)
tray.AddModule('Delete', name + '_delete', Keys=['CalibratedWaveforms'])
# Super DST
if superdstname != None:
tray.AddModule('Rename', name + '_sdstrename',
Keys=[superdstname, pulses], If=lambda fr: pulses not in fr)
For common processing chains like the L2 processing, such a segment would typically live in a pure python project. For small personal projects, they can live in any python file (including in the script from which they are being used).
Expanding segments in the I3Tray¶
There are several ways to see what is inside of a segment. The simplest is using the –expand-segments argument to icetray-inspect:
[nwhitehorn@wanderer ~/i3/offline/build]$ icetray-inspect --expand-segments --no-modules --no-services payload-parsing
*** payload-parsing ***
-------------------------------------------------------------------------------
icecube.payload_parsing.I3DOMLaunchExtractor (I3Tray segment)
Extract DOMLaunches and triggers from raw DAQ data.
Extra keyword arguments are passed to I3PayloadParsingEventDecoderFactory.
Equivalent to:
AddService('I3XMLOMKey2MBIDFactory', 'example_OmKey2MbId', InFile='/home/nwhitehorn/i3/offline/src/phys-services/resources/mainboard_ids.xml.gz')
AddService('I3PayloadParsingEventDecoderFactory', 'example_EventDecoder')
AddModule('I3FrameBufferDecode', 'example_fbdecoder', BufferID='I3DAQData', ExceptionId='I3DAQDecodeException')
It is also possible to print out the contents of an I3Tray or TrayInfo object using the Python print()
function to get the contents and configuration of the entire tray, with all segments expanded, in a human-readable form. The Python repr()
operator can also be used to get a more-tractable (and potentially executable) version of a tray or TrayInfo frame:
print(repr(tray))
gives:
tray = I3Tray.I3Tray()
tray.AddModule('I3Reader', 'reader', DropBuffers=False,
Filename='/tmp/bork.i3', FilenameList=[], SkipKeys=[])
tray.AddModule('Dump', 'dump', IcePickServiceKey='', If=None)
tray.AddModule('I3Writer', 'writer', CompressionLevel=-2,
DropOrphanStreams=[], Filename='bork-out.i3', IcePickServiceKey='',
If=None, SkipKeys=[], Streams=[])
You can also, of course, read the source code for the segment.
Default Parameters for Segments¶
- icecube.icetray.I3Default¶
The base class of the class hierarchy.
When called, it accepts no arguments and returns a new featureless instance that has no instance attributes and cannot be given any.
Often times segments will contain default parameters. When segments are layered
with default parameters which are passed to modules it can often be difficult to
track down where the parameter was actually set. There is a global singleton
I3Default
which if passed to a module will be equivalent to not setting
the parameter. Setting the segment to I3Default
will simplify the segment so
you don’t have to guess the correct default to pass the the module.
Example usage:
@icetray.traysegment
def MySegment(tray,name,
Param1 = icetray.I3Default,
Param1 = icetray.I3Default,
If = lambda f: True):
tray.AddModule("MyModule",
Param1=Param1,
Param2=Param2,
)