RSS

Allpix Squared 2.0 Released

After more than 3.5 years of continuous development and countless feature and patch releases of the Allpix Squared 1.x series, we are proud to announce the second major release of the framework, Allpix Squared 2.0. This version contains more than 1500 commits over the last feature release 1.6 and brings major improvements in many areas as detailed below. Among them is a major rework of the framework core, introducing an efficient multithreading functionality including full reproducibility, as well as a large number of new features and physics simulation models.

As usual, the release is available as Docker image, CVMFS installation, binary tarball and source code from the repository.

We would like to use this opportunity for a few laudatory words.

  • First of all, we would like to extend our gratitude to the by now 34 contributors and former developers as well as the countless users for their continued input to the development, be it through actual code or through fruitful discussions and suggestions on the implementation of different features.
  • Furthermore, we would like to thank the two Google Summer of Code students, Viktor Sonesten and Mohamed Moanis Ali, who have contributed significantly to the new multithreading capabilities. It took a while for your code to land in master, but here it is!
  • Last but not least, we would like to thank Koen Wolters, one of the bedrocks of the Allpix Squared development who laid the foundation of the framework and until today - despite leaving the field and taking up a full-time job - continues to contribute code, to review merge requests and to make suggestions for improvements of the tool in his spare time. Thank you very much!

We would also like to advertise our second User Workshop, which will be held fully virtually from 17 August 2021 to 19 August 2021. Please register and submit an abstract to present your use case of Allpix Squared.

Enough of the chatter, let’s cut to the new features shipped with Allpix Squared 2.0:

Multithreading

The biggest new feature is arguably the availability of full, event-based multithreading. With this, the simulation can make full use of all available CPU resources and can significantly speed up the simulation. The feature has been initially developed by two Google Summer of Code students and has been completed with lots of rework in the first few months of 2021. While the initial code was in place relatively quickly, quite some obstacles have been encountered on the way to a stable version.

Event Parallelisation

In versions 1.x, Allpix Squared already had the option to process the multiple module instances for different detectors but for the same module in parallel. While this was a very simple thing to implement, it does not scale very well due to multiple reasons. First of all, the usage of threads is always limited by the number of detectors in the simulation, limiting the simulation to a fully single-thread execution for simulations of individual detectors. Furthermore, detector modules which could be processed in parallel would be interleaved with unique modules that required a synchronous execution, and finally, individual module processing would be limited by the slowest detector instance.

To overcome these limitations, the parallelism architecture has been completely re-thought and re-written to follow the popular notion of event-based multithreading that is particularly useful in Monte Carlo simulations where events - by definition - represent fully independent entities. With the new architecture, a central thread pool coordinates the assignment of events to different worker threads, and all module instances within one event are processed sequentially by the assigned worker. This allows to scale linearly with the number of workers available on the system.

Event-based Seeding

One of the corner stones of Monte Carlo simulations is reproducibility. Also with multithreading, Allpix Squared guarantees strong reproducibility, meaning that a simulation can be fully reproduced when starting the program with the same configuration and the same starting seeds for the pseudo-random number generation. Especially in multithreaded environments, this places stringent requirements on the internal workings of the code base and the way, seeds are distributed and random numbers are used by the different parts. A common approach, also implemented in Allpix Squared is event-based seeding. For this, every event receives a seed from a central pseudo-random number generator (PRNG) upon its creation. This seed subsequently is used to set the initial state of a PRNG to be used throughout the lifetime of this event. In order to keep the memory footprint as low as possible, Allpix Squared does not allocate these PRNGs per event, but only per worker thread and seeds them whenever a new event is processed on the respective thread. In order to allow interruption of event processing and resumption at a later point in time, the state of the PRNG is stored whenever the event processing is halted, e.g. for buffering as described below, and is loaded again when the event is restarted.

Input/Output Module and Event Sequence

In order to create fully bit-compatible output files and to guarantee the same input to the simulation, matched to the same event seeds, some modules require to process events in their predefined sequence. This means the framework core has to take care of distributing workload accordingly and to buffer events which are not due for processing in the relevant modules, and to avoid data races. Allpix Squared provides a new class of modules, dubbed SequentialModule, for this purpose. Modules deriving from this class have the guarantee to receive events for processing in their natural sequence. The central thread pool holds a priority queue for partially processed events which are waiting to be further processed by a sequential module. This allows to temporarily buffer these events such that the workers can occupy themselves with another event, leading to a very efficient usage of available resources. An almost linear increase in event generation frequency with the number of worker threads has been observed for up to the tested maximum of 96 CPU cores that were available, despite the presence of SequentialModules in the simulation chain.

Event generation rate as function of used worker threads for a reference simulation

Event generation rate as function of used worker threads for a reference simulation

Sequential modules can waive the sequence requirement at run time if they have been configured in a way that strict order in event processing is not required by the user.

Interface to Geant4

The run managers provided by the Geant4 framework for multithreaded simulations start and manage their own threads, a solution that does not integrate with the Allpix Squared framework. It was therefore necessary to write our own implementation of run managers, both for single- and multithreaded execution of the simulation in order to be able to control the worker threads, but also to re-seed the Geant4 pseudo-random number generators with the event seeds obtained from the Allpix Squared core. In the end, the solution implemented is quite sleek and has less overhead than the initial Geant4 interface used up to version 1.6.2.

Over the course of these renovations, we have also overhauled the way Geant4’s informational messages and error codes are transmitted and displayed. It is now possible to redirect the Geant4 output to any level of the Allpix Squared logger and therefore to properly see and log e.g. exceptions from problematic events generated. This can be done by setting the relevant configuration keys in the GeometryBuilderGeant4 module:

[GeometryBuilderGeant4]
# Target logging level for Geant4 messages from the G4cerr (error) stream:
log_level_g4cerr = WARNING
# Target logging level for Geant4 messages from the G4cout stream:
log_level_g4cout = TRACE

Charge Carrier Recombination & Lifetime

In many scenarios the finite lifetime of electrons and holes generated by the incident radiation in the silicon lattice is relevant, e.g. because of low electric field regions and high doping concentrations. In order to simulate the recombination of these charge carriers properly, all available propagation modules, GenericPropagation, ProjectionPropagation and TransientPropagation have been fitted with the functionality of charge carrier recombination. The lifetime of any charge carrier is derived from the position-dependent doping concentration of the sensor and calculated from one of the different lifetime models selectable via the configuration

Three models have been implemented and validated with a simulation of monolithic sensor implemented in a commercial CMOS imaging technology, namely the Shockley-Read-Hall model, the Auger model, which applies to minority charge carriers only, and a combination of the two. At every step of the charge carrier propagation, the exponentially decreasing survival probability of a charge carrier is evaluated by comparing the duration of the time step to the lifetime provided by the above mentioned models.

The required effective doping concentration in the sensor is provided through a new module called DopingProfileReader, allowing the user to define a position dependent concentration throughout the sensor. The doping profile can be provided in three ways:

  • A constant doping concentration can be defined for the entire sensor volume.
  • Several regions of different doping concentrations can be defined via a segmentation along the sensor depth.
  • A three-dimensional scalar map can be parsed from files in the APF or INIT formats, the same formats already used by the electric field and the weighting potential maps.

The models are implemented as separate classes to avoid duplication of code and to allow for a more coherent use of these models throughout the codebase. The individual models can be enabled via a parameter in the corresponding configuration section.

Selecting the Charge Carrier Mobility Model

In previous versions, the charge carrier mobility used by Allpix Squared for the signal formation was fixed to the implemented Jacoboni-Canali model. For the new release, additional models as well as the option to configure the mobility model from the configuration file have been added.

Allpix Squared now provides the following mobility models:

A detailed description of the different models, their validity range and the parameters chosen is provided in the user manual. The desired mobility model can be selected from within the charge carrier propagation module configuration, while the previously used Jacoboni-Canali model remains the default option. More models can be added upon request.

Similar to the lifetime models, also the mobility models are provided centrally and can be selected via the corresponding module configurations.

Other Notable Changes to the Framework Core

New C++ Standard Requirement: C++17

The minimum supported C++ standard has been increased from C++14 to C++17. This allows us to use more modern features such the file system access functionality introduced with C++17 or the possibility to use structured bindings. This means, the minimum compiler requirements have been adapted accordingly, and now at least GCC 8.0, LLVM 7.0 or AppleClang 12.0 are required.

Boost Random Number Generator

We have struggled for a long time with cross-platform reproducibility of simulations due to the lax requirements on random number distributions imposed by the C++ Standard. As one piece of the puzzle for full reproducibility of simulations, the C++ STL random number distributions were therefore replaced by the implementation provided by the Boost.Random library. This implementation is platform independent and also allows users to run testing locally on their machines.

This implicates a dependency of Allpix Squared on Boost.Random with a version of at least 1.64, which is available for all supported platforms.

Unused Configuration Keys

With an increasing number of available configuration keywords and modules in a typical simulation pipeline, sometimes typos go undetected, spoiling the expected simulation result. To counteract this, the user is now made aware of unused configuration keys via WARNING log messages at the end of the simulation run, hinting at unintended configuration scenarios or potential typos in the main configuration file:

 (STATUS) Finished run of 1 events
 (STATUS) Finalization completed
(WARNING) Unused configuration keys in section ElectricFieldReader:mydetector:
          unused_key
 (STATUS) Executed 1 instantiations in 0 seconds, spending 0% of time in slowest instantiation ElectricFieldReader:mydetector
   (INFO)  Module ElectricFieldReader:mydetector took 0.000701816 seconds

Introducing enum reflection for module configuration parameters

Many parameters involve selection of models from a list of implementations. In Allpix Squared 1.x, this selection was usually made via string comparison and if-else clauses, selecting the correct value. With models often being listed in enums, we have introduced Enum reflection to allow direct parsing of configuration values to enums.

This changes a code such as:

auto model = config_.get<std::string>("model");
std::transform(model.begin(), model.end(), model.begin(), ::tolower);
if(model == "a") {
    model_ = MyModel::A;
} else if(model == "b") {
    model_ = MyModel::B;
} else {
    throw InvalidValueError(config_, "model", "Invalid model, only 'a' and 'b' are supported.");
}

to a much shorter and more concise snippet:

model_ = config_.get<MyModel>("model");

In case an invalid model (i.e. one that is not contained in the enum) is selected, an exception is thrown, stating all possible options and therefore replicating the behavior of the manually created exception from the example above:

  (FATAL) [C:MyModule:mydetector] Error in the configuration:
                                  Could not convert value 'c' from key 'model' in section 'MyModule' to type MyModule::MyModel: invalid value, possible values are: a, b
                                  The configuration needs to be updated. Cannot continue.

Analytic Electric Fields

More complex, analytic electric fields can be defined by the module ElectricFieldReader. In addition to the three-dimensional map and the analytic linear and constant electric field profiles, the user can now choose a parabolic parametrisation, as well as provide a custom analytic parametrisation. The latter features either one- or three-dimensional electric field vectors provided in ROOT’s TF1 syntax:

[ElectricFieldReader]
model = "custom"
# Parabolic in x and y, linear in z:
field_function = "[0]*x*y","[0]*x*y","[0]*z + [1]"
field_parameters = 12500V/mm/mm/mm, 12500V/mm/mm/mm, 6000V/cm/cm, 5000V/cm

Custom Impulse Response Functions in CSADigitizer

It is now possible to configure arbitrary, user-defined impulse response functions in the CSADigitizer module. The function is provided as string and internally parsed as ROOT::TFormula, parameters of the function can be provided via a separate configuration parameter. More details can be found in the user manual or the module README file.

Front-End saturation in DefaultDigitizer

A very simple front-end saturation simulation has been added to the DefaultDigitizer.

With this feature enabled, pixel charge values above a configurable saturation limit are replaced by a value drawn from a Gaussian distribution with a mean at the saturation limit and a configurable width.

New Example: EUDET-type beam telescope & DUT

An example has been added resembling the setup of a typical test beam experiment. It comprises a EUDET-type telescope using a similar configuration as in the corresponding standalone example, and two additional DUTs, in this case representing detector modules of the RD53a campaign. Furthermore, an FEI4 reference plane is added as the last detector plane, similar to a timing reference plane in a test beam experiment.

A particle beam of 5 GeV electrons is simulated to traverse the setup. The data is finally exported in the LCIO data format.

This example showcases the handling of different detector types by the use of different configurations of the same modules, restricted to certain detector types and names.

Changes to the Continuous Integration

The continuous integration has seen a number of improvements and extensions with the most notable being listed here.

Testing

The module test configurations have been moved into the respective module directories, which should provide a better overview over which module an individual test is probing. With close to 100 module tests, this approach also scales better.

Many module tests have also been simplified to only require a minimal set of other modules absolutely necessary for their execution. This includes for example the replacement of many DepositionGeant4 module instances with the much simpler DepositionPointCharge which does not rely on external libraries for charge generation.

In addition, tests will now fail if any WARNING, ERROR or FATAL messages appear in the test log, making the testing much more robust against undesired changes in the module behavior.

Linting & Formatting for CMake

Formatting rules have been introduced for CMake files, similar to the formatting already applied to the C++ source code. Corresponding tests are performed as part of the Continuous Integration.

Materials

Several materials were added to the dictionary, including PPO foam and Polystyrene. Furthermore, it is now possible to request any material known to the Geant4 NIST material manager.

Additional Features & Changes

  • Core:
    • Catch SIGABRT signal used by Geant4 in case of exceptions
  • CI:
    • Add CI for github repository via github actions
    • Update to LCG version LCG_99
    • Update to Geant4 10.7
    • Clang & MacOS binaries are deployed to CVMFS
  • Materials:
    • Fix in definition of cellulose
    • Added option to define the object color for the visualisation
  • Module CSADigitizer:
    • Fix global time calculation
    • Fix handling of uninitialised pulses
  • Module DepositionGeant4:
    • Add possibility for uniform beam profile
    • Fix beam divergence in case of beam direction not pointing along the z-axis
  • Module DepositionPointCharge:
    • Make number of charge carriers created per unit length configurable
  • Module DepositionReader:
    • Fix generation of MCParticle objects and their parent relation
    • Add module tests based on the python tool for the generation of deposits
    • Allow for non-sequential reading of events
  • Module ElectricFieldReader:
    • Fix missing of z scale for field representations
    • Warn at unphysically high electric fields
  • Module PulseTransfer
    • Add maps of induced charge per event
  • Module ProjectionPropagation:
    • Charge deposited in the sensor surface no longer results in an exception
  • Module TransientPropagation:
    • Fix double counting effect of induced current in case of propagations around the border of a pixel cell
  • Tools:
    • MeshConverter: Cache parsed fields for more efficient conversion
    • MeshPlotter: Fix slicing for two-dimensional fields

Multithreading Demonstration

Screencasts of the same simulation with identical initial seed, run with a single worker and three workers

Single Worker

Three Workers

Development Visualization

We cannot go without - a fast-forward through the full development cycle of Allpix Squared, from its first commit up to the release of 2.0: