Writing Thread-Safe Code

In Allpix Squared events are processed fully parallel on separate threads which requires some consideration when writing module code. This section briefly lists the most important aspects to take into account.

Member Variables

While the initialize() and finalize() of the module are guaranteed to be called sequentially, the run() method will be called simultaneously from different threads and for different events. Therefore, no module data members must be altered from within the run() function, otherwise these changes will affect other events being processed in parallel on other threads. Configuration parameters cached as member variables should therefore be set only in the initialize() function.

For initialization and finalization of thread-local data members, i.e. structures which have to be configured for each of the worker threads the module is executed on, the initializeThread() and finalizeThread() methods are available. They are called once on each worker thread after the initialize() and before the finalize() methods, respectively.

Histograms

Allpix Squared uses ROOT histograms for collecting and storing statistics and other additional information about the simulation process. ROOT provides the template class ROOT::TThreadedObject which allows to use histograms in multithreaded environments but slightly alters the interface of the histogram objects. Furthermore, there have been significant changes to the class between minor release version of ROOT and it doesn’t scale well with a large number of predefined threads. Therefore, Allpix Squared provides its own re-implementation of this class, allpix::ThreadedHistogram which also restores the original interface of the histogram classes, i.e. it is possible to instantiate, fill and store histograms the same way as in a single-threaded environment.

This class can be used as follows:

// Declaration of a new histogram of type "TH1D"
Histogram<TH1D> my_histogram;

// Creation of the histogram using the CreateHistogram helper method:
my_histogram = CreateHistogram<TH1D>("name", "title", 100, 0., 100.);

// Filling, setting bin contents and writing the histogram works as before:
my_histogram->Fill(12.);
my_histogram->SetBinContent(15, 23.);
my_histogram->Write();

Declaring a Module Thread-Safe

If a module is thread-safe, i.e. its run() function can be called from different threads in parallel without locking, it can be declared as thread-safe to the framework. In this case the ModuleManager will allow multithreading of calls to this module.

This declaration is done by placing the following call in the constructor of the module:

MyParallelModule::MyParallelModule(Configuration& config, Messenger* messenger, std::shared_ptr<Detector> detector)
    : Module(std::move(config), std::move(detector)) {
    // This module is thread-safe and can be called from different threads simultaneously:
    allow_multithreading();
}

By adding this statement, the module certifies to work correctly if its run() method is executed multiple times in parallel, for different events. This means in particular that the module will safely handle access to shared (for example static) variables as described in Section 10.4 and that it will properly assign and bind ROOT histograms to their respective directories in the output ROOT file before the event processing starts and the run() method is called the first time. Access to constant operations in the GeometryManager, Detector and DetectorModel is always valid between various threads. In addition, sending and receiving messages is thread-safe.

Since multithreading might be disabled by other modules in the chain or by the user via the configuration file or command line, it might be required to check at runtime of the module if it is currently running in a multithreaded environment. This can be achieved with the following method:

MyParallelModule::run(Event* event) {
    if(multithreadingEnabled()) {
        // This module is currently running in a multithreaded environment
    } else {
        // This module is running in a fully sequential environment
    }
}