Offline processing of data sent north from the South Pole¶

Online processing at pole¶

Decisions stored in OnlineFilterMask, which is never altered by any offline processing. Note that the OnlineFilterMask will appear to be identical for all P frames under a single Q frame, from the viewpoint of offline processing.

Offline processing¶

Large scale offline processing is handled by IceCube M&O on the grid in IceProd.

How to process data offline yourself¶

There are 3 top level scripts that are called from the command line in sequence. The output file from the previous is the input file to the next:

offline_filterscripts/resources/scripts/production_run_filter.py
offline_filterscripts/resources/scripts/production_reco_cascade.py
offline_filterscripts/resources/scripts/production_reco_muon.py

The offline production batch jobs are split into these three steps to reduce resident memory use (allowing access to more job slots); production_reco_cascade.py requires the most ~5 GB.

Offline filters¶

The offline filters are applied in the top level script offline_filterscripts/resources/scripts/production_run_filter.py. Offline filter names are listed in offline_filterscripts/resources/offline_filter_config.json

Execute the offline filters:

$ echo $I3_DATA
/cvmfs/icecube.opensciencegrid.org/data
$ $I3_SRC/offline_filterscripts/resources/scripts/production_run_filter.py --qify \
-g $I3_DATA/i3-test-data-svn/trunk/2023data/Level2_IC86.2023_data_Run00138821_84_781_GCD_withSLCcal2.i3.zst \
-i /data/exp/IceCube/2024/filtered/PFFilt/0105/PFFilt_PhysicsFiltering_Run00138821_Subrun00000000_00000000.tar.bz2 \
-o filteredfile.i3.zst --gapsfile outfile_gaps.txt

The processing description follows. The top level script calls:

offline_process

read_superdst_files

Reader

Convert P to Q frame

Require I3SuperDST, I3EventHeader, and DSTTriggers in frame

Require OnlineFilterMask Keep_SuperDST_23 prescale passed

I3SuperDST to CleanSuperDST (bad doms)

CleanSuperDST to InIceDSTPulses

Trigger Splitter to create InIceSplit P frames

look for quiet time between these triggers: deep core SMT3, SMT8, Volume, String, Faint

DSTPulses to SplitInIcePulses ; DSTTriggers to I3TriggerHierarchy

ExtractIceTopTankPulses to create IceTopDSTPulses

IceTop cluster trigger splitter I3TopHLCClusterCleaning

IceTopHLCTankPulses to CleanedIceTopHLCTankPulses

Copy Q to NullSplit P frame (used by IceAct filter)

Create SLOPSplit P frames

Add TriggerHierarchy to all P frames (from DSTTriggers)

online basic recos for InIceSplit

standard_srt_config

I3SeededRTCleaning_RecoPulseMask_Module

SplitInIcePulses to SplitRTCleanedInIcePulses

I3TimeWindowCleaning 6 usec

SplitRTCleanedInIcePulses to CleanedInIcePulses

Flary DOM checker

linefit.simple

CleanedInIcePulses to PoleMuonLinefit

I3SinglePandelFitter

PoleMuonLinefit CleanedInIcePulses to PoleMuonLlhFit

CutsModule I3FirstPulsifier

CleanedInIcePulses to FirstPulseMuonPulses

direct_hits.I3DirectHitsCalculatorSegment

FirstPulseMuonPulses PoleMuonLlhFit to PoleMuonLlhFitDirectHitsBase

hit_multiplicity.I3HitMultiplicityCalculatorSegment

FirstPulseMuonPulses to MuonPulsesHitMultiplicityBase

I3mue

CleanedInIcePulses PoleMuonLlhFit to PoleMuonLlhFitMuE

online cascade filter if necessary

offline filters

See offline_filterscripts/resources/docs

Make (Q)OfflineFilterMask (Q)PassedAnyOfflineFilter, delete filter booleans

Create IceTop legacy tags

Cut frames that do not pass any offline filters

two writers for output files: i3.zst and _gaps.txt

Cascade filter reconstruction¶

The final reconstructions for P frames that pass the offline cascade filter are applied. The monopod reconstruction consumes the most time and memory ~5 GB.

$ echo $I3_DATA
/cvmfs/icecube.opensciencegrid.org/data
$ $I3_SRC/offline_filterscripts/resources/scripts/production_reco_cascade.py \
-g $I3_DATA/i3-test-data-svn/trunk/2023data/Level2_IC86.2023_data_Run00138821_84_781_GCD_withSLCcal2.i3.zst
-i filteredfile.i3.zst -o cascadefile.i3.zst

Muon filter reconstruction¶

The final reconstructions for P frames that pass the offline muon filter are applied.

$ echo $I3_DATA
/cvmfs/icecube.opensciencegrid.org/data
$ $I3_SRC/offline_filterscripts/resources/scripts/production_reco_muon.py \
-g $I3_DATA/i3-test-data-svn/trunk/2023data/Level2_IC86.2023_data_Run00138821_84_781_GCD_withSLCcal2.i3.zst \
-i cascadefile.i3.zst -o outfile.i3.zst

Analyzing output data processed offline¶

A script is provided to calculate the P-frame rate for offline filters and other quantities.

Use NullSplit P frames for OnlineFilterMask

NullSplit is empty P frame, inheriting all objects from Q frame (except when sometimes used by IceAct filter).

$ $I3_SRC/offline_filterscripts/resources/scripts/calc_rates.py outfile.i3.zst > outfile_calc_rates.txt 2> outfile_calc_rates.err
$ cat outfile_calc_rates.txt

['/data/exp/IceCube/2024/filtered/dev/dev.11/0904/Run00139834/Level2_IC86.2024_data_Run00139834_Subrun00000000_00000002.i3.zst']
OnlineFilterMask count:  {'Keep_SuperDST_23': 1961, 'OnlineDeepCoreFilter_24': 241, 'HighQFilter_23': 54, 'MuonFilter_23': 206, 'OnlineL2Filter_23': 126, 'SoftwareSMT12Filt_23': 961, 'CascadeFilter_24': 146, 'MinBiasFilter': 2, 'GrecoFilter_24': 1}
Weird count:  {}
Boolean count:  {'PassedAnyOfflineFilter': 2059, 'PassSMT8Trig': 386}
QBoolean count:  {'QPassedAnyOfflineFilter': 1961}
OfflineFilterMask count:  {'SLOPFilter_24': 302, 'OfflineSupDeepCoreFilter_24': 158, 'OfflineMu_24': 150, 'OfflineHQ_24': 54, 'IceTopSouthwestNHLCTank1_24': 189, 'FaintParticleFilter_24': 629, 'IceTopNHLCTank5_24': 302, 'IceTopTwoStation_24': 116, 'OfflineCscd_24': 34, 'MESEFilter_24': 181, 'MonopoleFilter_24': 34, 'ELOWEN_Filter_24': 3, 'IceActTriggerPresent_24': 1}
FilterMask count:  {}
QFilterMask count:  {}
Frames  4019
I3EventHeader  4019
Non-NullSplit frames  2058
I3SuperDST  4019
DSTTriggers  4019
header_I3SuperDST_DSTTriggers  4019
ml_suite_triton_output  0
Files cover:  75.0888485941  sec.
Overall frame rate:  53.52326044743463  Hz
Overall I3EventHeader rate:  53.52326044743463  Hz
Overall non-NullSplit frame rate:  27.407531724513678  Hz
Overall I3SuperDST_cnt rate:  53.52326044743463  Hz
Overall DSTTriggers_cnt rate:  53.52326044743463  Hz
Overall header_I3SuperDST_DSTTriggers_cnt rate:  53.52326044743463  Hz
Overall ml_suite_triton_output rate:  0.0  Hz
OnlineFilterMask
Filter:  Keep_SuperDST_23  Rate:  26.115728722920952  Hz
Filter:  OnlineDeepCoreFilter_24  Rate:  3.209531168905635  Hz
Filter:  HighQFilter_23  Rate:  0.7191480627423414  Hz
Filter:  MuonFilter_23  Rate:  2.743416683794858  Hz
Filter:  OnlineL2Filter_23  Rate:  1.6780121463987965  Hz
Filter:  SoftwareSMT12Filt_23  Rate:  12.798172005470187  Hz
Filter:  CascadeFilter_24  Rate:  1.944363280747812  Hz
Filter:  MinBiasFilter  Rate:  0.026635113434901533  Hz
Filter:  GrecoFilter_24  Rate:  0.013317556717450767  Hz
Boolean
Filter:  PassedAnyOfflineFilter  Rate:  27.420849281231128  Hz
Filter:  PassSMT8Trig  Rate:  5.140576892935996  Hz
QBoolean
Filter:  QPassedAnyOfflineFilter  Rate:  26.115728722920952  Hz
OfflineFilterMask
Filter:  SLOPFilter_24  Rate:  4.021902128670131  Hz
Filter:  OfflineSupDeepCoreFilter_24  Rate:  2.1041739613572212  Hz
Filter:  OfflineMu_24  Rate:  1.997633507617615  Hz
Filter:  OfflineHQ_24  Rate:  0.7191480627423414  Hz
Filter:  IceTopSouthwestNHLCTank1_24  Rate:  2.517018219598195  Hz
Filter:  FaintParticleFilter_24  Rate:  8.376743175276532  Hz
Filter:  IceTopNHLCTank5_24  Rate:  4.021902128670131  Hz
Filter:  IceTopTwoStation_24  Rate:  1.5448365792242889  Hz
Filter:  OfflineCscd_24  Rate:  0.4527969283933261  Hz
Filter:  MESEFilter_24  Rate:  2.4104777658585888  Hz
Filter:  MonopoleFilter_24  Rate:  0.4527969283933261  Hz
Filter:  ELOWEN_Filter_24  Rate:  0.0399526701523523  Hz
Filter:  IceActTriggerPresent_24  Rate:  0.013317556717450767  Hz
FilterMask
QFilterMask

Offline processing of data sent north from the South Pole¶

Online processing at pole¶