Python in Radcube¶
There are several tools, modules, segments, etc, that are written in python for the radcube project.
This is an overview of what is there.
The docmumentation here is useful, but the files also have extensive docstrings and can be found here: icecube.radcube
.
The python-based classes for this project can be found at Basic Classes. The tools for using machine learning within radcube can be found at: Machine Learning Tools. Pure python modules can be found at :ref`RadcubePythonModules`.
Python Tools¶
There are two tools to help do very common functions when analyzing radio data.
The first converts all types of I3AntennaWaveforms (time series and spectra) into/from numpy arrays, PythonListToRadTrace
and RadTraceToPythonList
.
There is another function, GetChunkSNR
, which is used to calculate SNR using the “chunk method”.
In this algorithm, the RMS amplitude of a time series is calculated in N non-overlapping windows and the SNR is calculated as the maximum-absolute amplitude in the whole waveform divided by the median RMS of the chunks, all squared.
See also the docstrings at icecube.radcube.python_tools
.
Basic Classes¶
Spike Filter¶
Applies a filter to remove frequency-domain RFI spikes. This class requires a properly formatted file with the weight for each frequency bin. Spike filters are applied per-antenna-channel to account for the different noise observed at different locations.
For more details on the theory of the method, see Sec 3 of this ICRC proceeding.
See also the docstrings at:
icecube.radcube.SpikeFilter
.
Star Interpolator¶
This class interpolates the simulated electric fields using CoREAS output. A detailed explaination of the algorithm is given in Star Interpolation Tool. See also the “radcube paper”.
See also the docstrings at:
icecube.radcube.StarInterpolator
.
TAXI Artifact Remover¶
General tool for removing the chip/bin artifacts from the TAXI waveforms. All functions are expected to be given numpy arrays corresponding to the amplitudes of the waveform bins. It is assumed that all amplitudes being given are in units of ADC bins. In general the functions require the waveforms to be NxM in shape where N is the number of copies of that waveform, depending on the cascading mode being used and M is the length of the waveform. To ensure that this class is properly used, use the
GetCascadeReshaped
, apply any corrections, and then useFlattenWaveforms
.There are three main corrections that are applied:
ReplaceMissingBins
- Fixes the missing bins and/or corrupted bins during readout
RemoveChipOffsets
- Removes the offsets of individual chips based on the median amplitude of each chip
BinSpikeRemover
- Identifies individual bins that are broken (e.g. by bit flips) and corrects them depending on how many copies of the waveform are recorded (AKA the “cascading mode”)See also the docstrings at:
icecube.radcube.TAXIArtifactRemover
.
TAXI Background Reader¶
Singleton object that can be pinged for background waveforms from measured radio waveforms. After supplying an/some I3File(s), the “soft triggered” waveforms from the files are read out and saved until either all the waveforms in the passed in frames have been read or until the specified number of waveforms (nTraces) has been read in.
Since these waveforms are read in in units of ADC, several operations can be done to them such that they will be in a sufficient state to add to, e.g. simulated waveforms. These include converting them into a voltage (convertToVoltage) and removing TAXI artifacts.
Internally, the waveforms are stored in a python dict, indexed by antenna and channel key. The radio data can then be index by requesting a spectrum or a time series. Alternatively, the
FFTDataContainer
can simply be requested to get both.For an example, see
$I3_BUILD/radcube/resources/examples/ExampleScript_AddingMeasuredNoise.py
See also the docstrings at:
icecube.radcube.TAXIBackgroundReader
.
Machine Learning Tools¶
Machine learning is used in radcube to waveform indification and cleaning. Classes have been included to handle the reading in of networks and their application to the IceCube data structures.
Please note that these will only be available if TensorFlow
is installed on the machine that you are running.
Additionally, the speed can largely vary depending on if GPUs are available or not, but the identification of this is left to the user.
All classes can be accessed through icecube.radcube.ml_tools
.
SingleChannelClassifier¶
Loads in trained model and has functions to return a classification score for a passed in waveform. The user is required to perform any preprocessing of the waveforms (e.g. upsampling, bandpassing) outside of a normalization. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long.
The user is expected to supply a .h5 file with the trained network for use.
See docs
TimeSeriesDenoiser¶
Applies a trained machine learning algorithm to the passed in waveforms in the frame and outputs a “denoised” version of that waveform. The models are trained on 4000 bins long waveform, so the input waveform sould also be 4000 bins long.
The user is expected to supply a .h5 file with the trained network for use.
Python Modules¶
There are several classes that are pure python modules. These are almost exclusively wrappers for the classes specified above and simply run the relevant functions on waveforms in the frames.
ApplySpikeFilter¶
Runs the Spike Filter class on obejects in the frame.
ApplySpikeFilter
includes the following parameters:
InputName: Name of the
I3AntennaDataMap
in the frame to apply the RFI filtering toOutputName: Name of the
I3AntennaDataMap
to output to the frame after filteringFilterFile: Name of the file containing the filter weights.
SpikePower: Weights are applied in the frequency domain \(A_{i}' = A_i * w_i^{\rm SpikePower}\)
ApplyInDAQ: If true, module will run in the Q Frame, otherwise on the P Frame
MeasuredNoiseAdder¶
Uses background waveforms measured in the real antennas to add to simulated radio waveforms to generate realistic simulations. This module is essentially just an
I3Module
that runs the TAXI Background Reader and adds these waveforms in the time domain.This class also can run the
icecube.radcube.SpikeFilter
on the read in data as well.This algorithm considers two cases:
You have noise from antennas that you want to add to simulations of those exact same antennas. i.e. you are simulating existing antennas
You have noise from a set of antennas that are not in your simulation, e.g. you are simulating a possible layout for an array on the moon
Set
MatchExactNoise
toTrue
for case 1 andFalse
for case 2. In case 1, the measured noise will only be added to antennas with matching keys. In case 2, a background waveform will be added and an object will be added to the frame which know the linkage between the AntennaKey for the measured/simulated antennas
MeasuredNoiseAdder
includes the following parameters:
InputName: Name of the input
I3AntennaDataMap
in the frameOutputName: Name of the output
I3AntennaDataMap
to output to put in the frameTaxiFile: List of the I3File(s) with the background TAXI waveforms to be read in and added later
NTimeBins: The number of time bins that are needed from each waveform.
NTraces: The number of waveforms to be read in from the I3File(s). If less are available, it will read in as many as it can.
ConvertToVoltage: If
False
, will keep waveforms in ADC, otherwise they will be converted to voltage.InsertNoiseOnly: If
True
, the pure noise will also be added to the frame.Overuse: If
True
, individual background waveforms will be used more than once. Otherwise, the tray will be terminated when the background waveforms are exhausted.FrequencyBand: If set with a list of low/high frequencies. All waveforms will first be filtered on that band.
RemoveBinSpikes: If
True
the TAXI Artifact Remover class will be run on the waveforms before they are internally stored (since some artifacts may not be correctable after adding to simulated pulses).SpikeFilter: If given an instance of a Spike Filter class, the spike filter will be applied to the background waveforms before being stored internally.
RadcubePlotter¶
Module which can be used to quickly makes plots of a single antenna/E-Field. The user gives a list of lists where each entry includes the following five things:
Name of the object in the frame to plot, e.g., “FilteredWaveform”
The fraction of the bins to plot. Plots bins 0 to specified value
Descriptive label to put on the plots for this entry, e.g., “After Bandpass Filter”
True if quantity is in ADC units, False if not
Method to plot the frequency spectrum “lin” for linear y-axis, “dB” to put it in dBm/Hz, “raw” for electric field
RadcubePlotter
includes the following parameters:
DataToPlot: List which includes the points listed above for each plot to be made
StationID: Id number of the station to plot
AntennaID: Id number of the antenna to plot
OutputDir: Directory where the output plots will be saved
ZoomToPulse: If non-zero, will zoom in on the pulse in the waveforms using the specified width
RemoveTAXIArtifacts¶
This class is a wrapper to run the TAXI Artifact Remover on the measured digitized waveforms in the frame. See the documentation for that class for more information on the underlying algorithms
Note that this code is only meant to be run on the raw ADC amplitudes. i.e. do not run it on waveforms that have already been filtered or otherwise modified.
RemoveTAXIArtifacts
includes the following parameters:
InputName: Name of the input
I3AntennaDataMap
in the frameOutputName: Name of the output
I3AntennaDataMap
to output to put in the frameBaselineValue: Value to set the baselines to in ADC bins
MedianOverCascades: Apply the median baselines over non-cascaded/semi-cascaded chunks
RemoveBinSpikes: Remove the single bin artifacts using an algorithm depending on the cascading mode
BinSpikeDeviance: Parameter which defines what constitutes a bin spike
RemoveNegativeBins: Removes the -1 bins (corrupted data) and replace it with average
SimulateFromStar¶
Simulates shower on a specified array using simulations on a star shaped pattern. This class does three things:
Loads up a star pattern simulation into an instance of the
icecube.radcube.StarInterpolator
classPuts the other meta-data into the frame (e.g. I3Particle)
Can randomize the core location or match the core location to an existing particle in the frame
The intended use cases for this module are:
Make a radio-only simulation, creating all frames Q frames, as needed
Add radio data to an existing simulation in the QFrame
In the first case set,
PiggybackMode = False
and choose a randomization scheme for the cores. You can either use theRandomCoreCenter
and aCoreRadius
about that center, sampled uniformly on the disk. The number of times you want to choose a random core is set byNThrows
. This needs it’s own RNG and due to limitations in IceTray, C++ implemented services cannot be used by python-implemented modules. So this class has it’s own RNG instance, provide it aRNGSeed
.Or, you can supply a list of I3Positions though which the axis will go.
In the second case, set
PiggybackModel = True
and set thePiggybackPrimary
to the name of the I3Particle in the frame that will be used as the core location, directly.
SimulateFromStar
includes the following parameters:
DirectoryList: List of paths to CORSIKA/CoREAS sims
NThrows: How many random cores per shower
CoreRadius: The radius from
RandomCoreCenter
within which the cores will be placedRandomCoreCenter: The core will be randomized about this point
RNGSeed: Seed for the randomization of the core location
PiggybackMode: Set to
True
if another module is also reading the CORSIKA file (will use the same core as the other module’s primary)PiggybackPrimary: The particle in the frame that will set the seed for the particle