Usage¶

This section currently only covers the standard usage of the module/service chain for IceCube, using a fully parameterized model (i.e. without Geant4).

Basic usage¶

For a simple simulation run, you should process your I3MCTree using MMC. Then, just use the supplied “tray segment” to add all relevant services and modules to the I3Tray:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits")

This reads the ‘I3MCTree’ and ‘MMCTrackList’ objects from each frame and adds hits named ‘MCHitSeriesMap’. All photons are generated in a way compatible to ppc.

By default, clsim uses only the GPU for photon propagation (i.e. CPU devices are skipped during OpenCL device enumeration). This behavior can be changed using the UseGPUs and UseCPUs boolean options. The following example would only use the CPU for simulation and skip all GPU devices:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
                UseGPUs=False, UseCPUs=True)

Generating Photons using Geant4¶

By default, clsim does apply photon yield parameterizations compatible to ppc. This can, however, be turned off to use a full Geant4 simulation run. In this case, full GPU simulation might not be necessary because the photon generation will become the bottleneck.

To use Geant4, just enable the UseGeant4 switch:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
                UseGeant4=True)

Even in this mode, Geant4 will not be used for muons that have a length assigned. These are assumed to have been generated by MMC. Both, neutrino-generator and Corsika generate muons without lengths, so the module should generally do the right thing. To make absolutely sure that no parameterizations are used, also set the MMCTrackListName option to None. This will apply Geant4 to all particles in the I3MCTree (which are InIce and not Dark), even if they already have a length assigned to them. (You should make sure that this is what you really want, as MMC might already have added cascades to the muon track that would be added a second time by Geant4.)

Low-Energy simulations¶

In order to simulate low energies in a more correct way, you might want to consider disabling the DOM oversizing optimization. It is set to an oversize factor of 5 (in radius), which gives you a 25-fold increase in simulation speed at the expense of accuracy in timing and for tracks very close to DOMs.

To disable DOM oversizing (which might be a good idea especially when using Geant4) use the DOMOversizeFactor switch:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
                DOMOversizeFactor=1., # disables oversizing (default is 5.)
                UseGeant4=True)       # enable or disable Geant4 as needed

Ice Models¶

By default, the ‘SPICE-Mie’ ice model is used. This can, however, easily be changed by supplying either a photonics-compatible ice description file or a ppc-compatible ice description directory using the ‘IceModelLocation’ parameter:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
                IceModelLocation=expandvars("$I3_BUILD/clsim/resources/ice/ppc_aha_0.80"))

Another example (using a photonics file) would be:

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
                IceModelLocation=expandvars("$I3_BUILD/clsim/resources/ice/photonics_wham/Ice_table.wham.i3coords.cos090.11jul2011.txt"))

Example Script¶

This is a short example script that reads an input .i3 file, applies MMC and clsim and writes the result to a second file. It also uses Geant4 for photon generation and is configured to run on the CPU only. In addition, adding a random number generator using a python object (instead of a I3Service) is demonstrated.

from icecube.icetray import I3Tray
from os.path import expandvars
import os, sys

from icecube import icetray, dataclasses, dataio, phys_services
from icecube import clsim

I3Seed  = 12345
# a random number generator
randomService = phys_services.I3SPRNGRandomService(
    seed = I3Seed,
    nstreams = 10000,
    streamnum = 1)
tray = I3Tray()

tray.AddModule("I3Reader","reader",
               Filename="input.i3")

tray.AddSegment(clsim.I3CLSimMakeHits, "makeCLSimHits",
    RandomService = randomService,
    UseGPUs=False,
    UseCPUs=True,
    UseGeant4=True,
    IceModelLocation=expandvars("$I3_BUILD/clsim/resources/ice/spice_mie"))

tray.AddModule("I3Writer","writer",
    Filename = "output.i3")

tray.Execute()

Cable Shadow¶

While the IceCube DOM has nearly uniform azimuthal acceptance, roughly 1/10th of the photocathode is shadowed by the communications and power cable. The clsim photon propagation kernel accounts for this by recording photon intersections with DOMs only if the photon path does not intersect the cable near the DOM. The cable is assumed to be a vertical, opqaue cylinder with 46 mm diameter and infinite vertical extent, flush with the DOM sphere.

This treatment, however, is disabled by default. It can be enabled by setting the CableOrientation parameter of I3CLSimMakePhotons (or a derived tray segment) to the path to a file giving the azimuthal orientation of the cable at each DOM. Examples include:

ice-models/resources/models/PPCTABLES/cable_position/orientation.led7.txt: positions derived from an analysis of the intensity at neighboring strings when LED 7 was flashed (source-oriented). This assumes that the angle between LED 7 and the cable is always 90 degrees, and constrains the positions to within a few degrees.
ice-models/resources/models/PPCTABLES/cable_position/orientation.shadow.txt: positions derived from an analysis of the received intensity compared to simulation as a function of the simulated cable position. This makes no assumptions on the rotation of the cable w.r.t. to the DOM mainboard, but only constrains the position to tens of degrees.

Either option reduces the average DOM acceptance by 10%, but in a directionally-specific way. The UnshadowedFraction parameter that has been used to account for the cable shadow in the past is _not_ automatically adjusted to compensate, so downstream analyzers will have to handle this themselves. The UnshadowedFraction needs to be set to 1 in this mode.

../../_images/First_Cable_Shadow_image.png

../../_images/Second_Cable_Shadow_Image.png

CUDA propagator¶

In addition to the OpenCL photon propagation kernel, there is an alternate implementation in native NVIDIA CUDA, enabled by passing UseCUDA=True to I3CLSimMakePhotons. This kernel significantly faster than the OpenCL implementation on 2020-era NVIDIA GPUs, and can run on platforms like POWER9 where NVIDIA does not provide an OpenCL implementation. Details on its optimization can be found in Hendrik Schwanekamp’s masters thesis.

There are a few operational restrictions to be aware of.

The clsim-cuda library must be compiled against a specific version of the CUDA toolkit (version >= 10). The propagator may fail if it encounters an unsupported GPU driver at runtime.
CMake >= 3.8 is required for CUDA support.
CUDA detection may fail if your C compiler is newer than the CUDA toolkit, e.g. CUDA 10 requires gcc < 8.
Because the kernel is compiled ahead of time, many parameters of the kernel are fixed. The ones that may be set at runtime are:
- Absorption and scattering coefficients for 171 ice layers
- Photon wavelength distribution
- DOM oversize factor
All other parameters will remain at their default values, even if set to another value in the propagator configuration. These include:
- The geometry (fixed to IC86)
- The form and parameters of the scattering function

ppc interoperability¶

I3CLSimMakePhotons can also use the propagator from ppc. This makes it possible to use experimental ice model features directly without first needing to port them to clsim, as well as comparing implementations of the propagator using the same input and output stages. To use this, supply a custom setup function to I3CLSimMakePhotons, e.g.:

from icecube.ppc import MakeCLSimPropagator
tray.AddSegment(clsim.I3CLSimMakePhotons,
   PropagatorSetup=MakeCLSimPropagator,
   ...
)

MakeCLSimPropagator will translate the configuration objects in DetectorParams into ppc’s preferred configuration format (text files and environment variables) and return configured instances of I3LightSourceToStepConverter. Note that since ppc stores its state in global variables, this converter is a singleton, and you should not use MakeCLSimPropagator more than once per process.

Non-spherical sensors¶

Previously, photon propagation required all sensors to be identically sized spheres. The OpenCL kernel has been updated to handle photon collisions with sensors of different radii and shapes (spheroids and cylinders). This should allow for usage with modules that would be included for the IceCube Upgrade and IceCube Gen2.

An upgrade/Gen2 GCD file is all that’s needed to run. Be aware that some existing files have prebuilt the I3ModuleGeoMap in them, and these will not work since I3GeometryDecomposer needs to rebuild those with the proper default sensor information. For an example of a test GCD file that can be used, see $I3TESTDATA/GCD/GeoCalibDetectorStatus_ICUpgrade.v58.mixed.cleaned.womhacked.V0.i3.zst. Do not use this for realistic MC though, unless you want to include WOMs on string 36.

Photon to MCPE conversion is currently left to other modules, so set MCPESeriesName=None when calling the clsim.I3CLSimMakePhotons module.