| Geant4 User's Guide For Application Developers Examples

Parallel Geant4 (ParGeant4)

## What is ParGeant4 ?

ParGeant4 is a parallel version of Geant4 that implements event-level parallelism to simulate separate events on remote processors. Typical simulations demonstrate a nearly linear speedup in running time as the number of remote processors increases. The needed enhancements of Geant4 are included in the examples/extended/parallel directory of the Geant4 distribution.

## Why is ParGeant4 useful?

When doing a large Geant4 simulation, one often wishes to run on many processors to reduce the overall time. Traditionally, this has been done by splitting the events into multiple groups, and running Geant4 independently on each processor for its own group of events. This requires restarting a run if a processor goes down. It also requires saving the histogram files from each run, and merging the files prior to using the analysis tool. The human effort in this is considerable.

ParGeant4 provides a much simpler mechanism. After setting up ParGeant4 one links and runs the sequential Geant4 application exactly as before, but additionally linking with some parallel libraries. Upon execution, ParGeant4 on the console sends out events to slave processes, collects all hits, and calls any analysis tool -- exactly as one would do in the sequential case.

There is no need to split events into separate groups, track whether one of the processors crashed, merge histogram files, etc. If a slave processor crashes, ParGeant automatically re-sends the events of that slave processor to a new slave processor for re-execution.

## What is the performance of ParGeant4?

As a rule of thumb, speedup will be nearly linear when each event simulation lasts for at least several milliseconds. ParGeant4 has been tested extensively on parallelizations of examples/novice/N02 and of examples/advanced/underground_physics . On N02, we see a speedup of 27 for 50 nodes and a speedup of 33 for 100 nodes. When using the `--aggregated-tasks=50` option (see below) the speedup improves to 35 for 50 nodes and 60 for 100 nodes.

In tests of underground_physics, events are longer and we see nearly linear speedup (94 times speedup with 100 nodes).

## Getting started

Detailed information is under extended/parallel/ParN02/docs/000README. There are four steps:

1. Install TOP-C.
2. Compile ParN02 by running `gmake`.
3. Make sure the "procgroup" file is correct and copy it to directory of the executable binary file (for example, `$G4BIN/Linux-g++`).
4. Run the parallel binary program.

## What is involved in setting up ParGeant4?

To set up ParGeant4, one needs TOP-C and Marshalgen (free, open source software). If one is parallelizing a new Geant4 application, one must then add/modify approximately 20 lines of annotations (C++ comments to indicate shallow vs. deep copying of pointers, etc.) in the .h files for each hit type being defined by the application. For details of the annotations, refer to the manual of the Marshalgen package.
Finally, in the main routine of the application, one replaces the call to the `G4RunManager` constructor by a call to the `ParRunManager` constructor. (`ParRunManger` is a derived class of `G4RunManager`.)

After this, one invokes the already provided GNUMakefile (a slightly modified version of the Geant4 example GNUMakefile) to create the parallel application. Finally, one writes a "procgroup" file, which declares the names of the remote hosts to use in the parallel computation. Optionally, one may also specify filenames (e.g. slave1.out, slave2.out, ...) to store the printout from each slave process. One then calls the ParGeant4 binary exactly as one would call the Geant4 binary, and the results appear as normal, only faster.

## Are there examples of using ParGeant4?

Yes. ParGeant4 includes parallelizations of other examples from the Geant4 distribution. Specifically, ParGeant4 includes example parallelizations of novice/N02, novice/N04, and advanced/underground_physics .

## What are some of the features of ParGeant4?

ParGeant4 includes all of the features of TOP-C. In particular, after building a binary, "parMySimulation", one might call:

./parMySimulation --TOPC-help
  Display command options, and then exit.

./parMySimulation --TOPC-trace=0
  By default, ParGeant4 traces each time a new event is sent to a task. This turns it off.

./parMySimulation --TOPC-verbose=0
  By default, ParGeant4 provides statistics indicating what process was run, when it was run, what machine, the running times and elapsed times of master and the average slave, and other information. This turns off the statistics.

./parMySimulation --TOPC-aggregated-tasks=10
  By default, ParGeant4 sends one event to one remote process before turning to the next process. This option sends 10 events to a single remote process in one message. This is useful when events are relatively short, and the network latency of sending a message is starting to dominate the running time.

./parMySimulation --TOPC-slave-timeout=3600
  By default, if a remote process has not communicated with the master after 1800 seconds (a half hour), the slave process will kill itself. This prevents runaway processes that may be in an infinite loop for some event, or may have lost their socket to communicate with the master process. In this example, we allow 7200 seconds (two hours) because we expect simulation of some events to last up to (but not more than) 7200 seconds.

By default, ParGeant4 uses its own subset implementation of MPI (MPINU). ParGeant4 adds approximately 50 KB to the "footprint" of the binary executable. By default, ParGeant4 uses "ssh" to set up remote processes. Those who wish to use their own MPI (perhaps if a batch cluster requires a specific MPI) may do so. (See "Configuring a Different `MPI')" in the TOP-C manual.)

About the authors