Online Filterscripts

  • Maintainer: Erik Blaufuss (blaufuss AT umd.edu)

Overview

This is the project containing the new online filters for the 2023 season and beyond. Highlights include:

* Simplification of the pole filtering - data processing running at South pole
  in the PnF system (https://github.com/icecube/pnf) will generate SuperDST for all
  events, performing realtime event searches, and shipping all events passing some
  basic event selections north.
* New WG filter selections in the North, replacing the old L1/L2
  filters with selections aimed to get closer to "final level" in a
  simpler and more direct manner.  These will live in offline_filterscripts project

For simulation, these software are setup to mimic normal data, with a minimal changes to remove compression and space saving.

Important IceTray segments

  • pole_base_processing_and_filter - intended to run at pole on raw DAQ output

    • location: python/pole_base_processing.py

    • This segment does several core data processing and filter steps on data from the DAQ

      • base_processing

        • Decode raw DAQ data into dataclasses objects (Triggers, I3DOMLaunches, etc)

        • Remove known bad data producing DOM readouts

        • Calibrate I3DOMLaunches to I3Waveforms

        • Perform Feature Extraction on calibrated waveforms (I3RecoPulses)

        • Generate SuperDST object from I3RecoPulses

        • Extract Seatbelt waveform objects to send north

        • Extract VEMCAL and I3SLCCalibration objects for transmission North for IceTop Calibration

        • More details are available for these segments in a slide set online_filterscripts/resources/docs/New_L1_Baseprocessing.pdf

      • Online filtering - apply dedicated filters for realtime alert events, and selections for SuperDST inclusion

  • SuperDST criteria (indicated by a ‘Keep_SuperDST_23’ in the OnlineFilterMask):

    • IceTop, Fixed Rate, Slow Particle, DeepCore, and other special triggers are automatically included in the SuperDST collection from pole

    • InIce SMT8, Volume and String triggered events have a software retrigger applied to them at pole to SMT12 (‘SoftwareSMT12Filt_23’ in OnlineFilterMask)

      • Any event passing the OnlineL2Filter_23 is also saved as SuperDST, even if it fails the retriggering.

    • Additionally, a few filters keep All raw waveforms for potential offline use, in addition to the SuperDST:

      • HighQFilter - any event with more than 1000pe in an InIce split (‘HighQFilter_23’ in OnlineFilterMask)

      • FixedRateTriggerFilter - any event with a FRT minBias trigger active (‘FixedRateTriggerFilter’ in OnlineFilterMask)

      • MinBiasFilter - a random 1 in 1000 prescale selection of random DAQ triggers for low level studies (‘MinBiasFilter’ in OnlineFilterMask)

  • Online realtime alert filters include:

    • Muon Filter - refurbished version of old MuonFilter (‘MuonFilter_23’ in OnlineFilterMask)

    • OnlineL2Filter - refurbished version of old OnlineL2Filter (‘OnlineL2Filter_23’ in OnlineFilterMask)

    • GFUFilter - refurbished version of the old GFUFilter, selecting events for online point source searches and Gold/Bronze track alerts (‘GFUFilter_23’ in OnlineFilterMask)

    • HESEFilter - refurbished version of the HESEFilter, selecting high-charge contained events for track and cascade alerts (‘HESEFilter_23’ in OnlineFilterMask)

online_filterscripts is still a bit of a work in progress, Please don’t hesitate to contact me (@blaufuss on Slack) if you have more questions.

The PnF online filtering system (https://github.com/icecube/pnf) produces 3 distinct files while processing the DAQ data:

  • PFRaw files - Saving all DAQ encoded data (including all waveforms from all DOMs) for all events. Saved to archive at pole and shipped north annually.

  • PFDST files - Saving the I3SuperDST record for all DAQ events, as well as seatbelt waveforms. Saved to archive at pole and shipped north annually.

  • PFFilt files - Save output of online filter/SuperDST selections from online_filterscripts, tuned to available satellite bandwidth. Sent north within ~24 hr via TDRS.

A more detailed collection of what’s in each file type is available in What’s in files made at pole in PnF?.

Useful scripts

There are a few useful “driver” scripts where you can add filters while testing and process data. These are in resources/scripts

  • PFRaw_to_DST.py - Applies base_processing to a sample of raw DAQ data (PFRaw files from pole). Generates files that will look like those to be sent north via satellite.

  • calc_filter_rates.py - Helper script to calculate filter rates from a filtered file.

Usage information is included the resources/scripts/README file.

Guidelines

When writing filters and python code here, please follow these guidelines:

  • Make sure your filter IceTray segment cleans up after itself. Any stray objects created that are not intended for long term preservation should be deleted from the frame at the end of the segment.

  • Please check for formatting styles to match the other files. Online Filterscripts is already configured to support ruff (https://docs.astral.sh/ruff/). Please check your code with this tool.

  • Please make a branch of icetray/main, develop your filter and make a pull request (PR) (should be assigned to @blaufuss automatically) when ready for inclusion. It will be reviewed before merging.

  • For details on branching, PRs, etc see the Git Guide (https://github.com/icecube/icecube.github.io/wiki/GitGuide%3AGitHub-in-IceCube)

Resources