Booking data in IceTray scripts

The IceTray interface to the framework is provided by a service that manages the actual output file and a module that shuffles data between the frame and the service. A simple script could look something like this:

from icecube import icetray, dataio
from icecube.icetray import I3Tray
from icecube.hdfwriter import I3HDFWriter

tray = I3Tray()
tray.AddModule('I3Reader',
               filename = 'foo.i3.gz')

tray.AddSegment(I3HDFWriter,
               output="foo.hdf5",
               keys=['LineFit','InIceRawData'],
               SubEventStreams=['in_ice'],
              )

tray.Execute()

Here, I3HDFTableWriter is a tray segment that sets up a table service and an I3TableWriter module. The ‘keys’ parameter is a list of frame object names. Any objects in the file named ‘LineFit’ will be written to a table called ‘LineFit’ in the HDF5 file ‘foo.hd5’.

You may have noticed the parameter SubEventStreams above. In triggered data, each global trigger readout window is contained in a Q (“DAQ”) frame, followed by zero or more P (“Physics”) frames that represent different views into that event, each with its own pulse selection, reconstructions, etc. Each class of selection has a name (“in_ice” in the above example), and are usually based on a specific sub-trigger, e.g. SMT-8, excluding windows only joined into a global trigger due to long triggers like Fixed-Rate (FRT). There can be multiple P frames from a given stream attached to one Q frame; these are numbered with an index and are disjoint by convention. Since I3TableWriter treats frames as disjoint events, you usually want to give it a single SubEventStream to handle.

Booking untriggered simulation files

Since P frames are created as an artifact of trigger post-processing, they do not exist in low-level simulation files, e.g. immediately after I3CORSIKAReader. To convert these files to tables with tableio, use one of the special-purpose segments, e.g.:

from icecube.hdfwriter import I3SimHDFWriter
tray.AddSegment(I3SimHDFWriter,
               output="foo.hdf5",
               keys=['LineFit','InIceRawData'],
              )

Note that I3SimHDFWriter does not have a SubEventStreams parameter.

Specifying objects by key

Objects can be specified by key. In addition, you may specify which converter should be used and what name the table should have in the output file. If you pass a list of keys:

keys = ['LineFit','InIceRawData']

The frame objects called ‘LineFit’ and ‘InIceRawData’ will be written to the tables ‘LineFit’ and ‘InIceRawData’ using the default converters.

You can also pass a dictionary to specify which converters should be used:

keys = {'LineFit': dataclasses.converters.I3ParticleConverter(),
        'InIceRawData': dataclasses.converters.I3DOMLaunchSeriesMapConverter()}

For full control, you can pass a list of dicts, specifying the key, converter, and name of the output table:

keys = [dict(key='LineFit', converter=dataclasses.converters.I3ParticleConverter(), name='SuperSpecialLineFit'),
        dict(key='InIceRawData', converter=dataclasses.converters.I3DOMLaunchSeriesMapConverter(), name='DOMLaunches')]

Specifying objects by type

If you don’t know ahead of time which specific key-names you’d like to book, you can tell I3TableWriter to book all objects of a given type. Types are specified via their Python bindings:

types = [dataclasses.I3Particle, dataclasses.I3DOMLaunchSeriesMap]

You can also specify converters to use for each type:

types = {dataclasses.I3Particle: dataclasses.converters.I3ParticleConverter(),
         dataclasses.I3DOMLaunchSeriesMap: dataclasses.converters.I3DOMLaunchSeriesMapConverter()}

You can also pass a list of dicts specifying the type and converter:

types = [dict(type=dataclasses.I3Particle, converter=dataclasses.converters.I3ParticleConverter(),
         dict(type=dataclasses.I3DOMLaunchSeriesMap, dataclasses.converters.I3DOMLaunchSeriesMapConverter())]

Specifying a default booker

You may want to use a special converter for some types, and default converters for others. You can specify that particular keys or types should be processed with the default converter by passing in tableio.default.

Example

To use a custom converter named MyCustomConverter for the I3MCTree and the default/builtin converter for I3Particle:

types = { dataclasses.I3MCTree   : MyCustomConverter(),
          dataclasses.I3Particle : tableio.default }

similarly:

keys = { 'LineFit' : tableio.default,
         'I3MCTree' : MyCustomConverter() }

Finding out which converters exist

If you want to discover which converters are already defined, you can inspect tableio.I3ConverterRegistry.registry in ipython. This dictionary contains all the converters tableio knows about. The I3ConverterRegistry is also used by I3TableWriter to automatically find an appropriate converter. So your converter has to be listed there in order to be used by I3TableWriter.

Note

Please note that only those converters for which the library has been loaded are listed in the I3ConverterRegistry. This means that if you want to book a class I3Class that lives in library mylib, you have to do a from icecube import mylib first. The reason is that the converters are residing in the pybindings of the libraries.

Booking everything in the file

Warning

You should really avoid doing this whenever possible, as it’s likely to create very, very large files.

If you really want dump absolutely everything in an I3 file using the default converters, you can set BookEverything to True:

tray.AddModule(I3TableWriter,'writer', tableservice = table_service, BookEverything = True)

This gives you absolutely no control over how objects are booked. If you just want to discover which converters are defined, you can inspect tableio.I3ConverterRegistry.registry in ipython. This dictionary contains all the converters tableio knows about.

Booking to multiple files at once

You can also route output to multiple files in parallel by instantiating many I3TableServices and passing them as a list to tableio.I3TableWriter:

from icecube import icetray
from icecube.icetray import I3Tray
from icecube.tableio import I3TableWriter, I3CSVTableService
from icecube.hdfwriter import I3HDFTableService
from icecube.rootwriter import I3ROOTTableService

tray = I3Tray()
tray.AddModule('I3Reader',filename = 'foo.i3.gz')

hdf = I3HDFTableService('foo.hd5')
root = I3ROOTTableService('foo.root','master_tree')
csv = I3CSVTableService('foo_csv')

tray.AddModule(I3TableWriter,
               tableservice = [hdf, root, csv],
               keys         = ['LineFit','InIceRawData']
              )

tray.Execute()

Booking from files with Q-frames

Using files that contain Q-frames, you have to know that there may be different SubEventStreams and that you have to tell I3TableWriter which one you want to book. Otherwise, it will just produce empty files. If this happens, you should get a WARN message telling you that there were SubEventStreams present that were not booked. Book the ones you want to write out using parameter SubEventStreams. E.g. you can do:

tray.AddModule("I3NullSplitter", "fullevent")

tray.AddSegment(HDFWriter,
   output="foo.hdf5",
   keys=["LineFit", "MPEFit"],
   SubEventStreams=["fullevent"],
   )