$Id: NEW_APP_INSTALL,v 1.2 2002/03/09 08:07:55 cooperma Exp $ ------------------------------------------------------------------------------ These are instructions for parallelizing a new application. Before doing so, try out the examples, ParN02 and ParN04, which have parallelized examples/novice/N02 and examples/novice/N04. These instructions are written relative to ParN02, but also see ParN04 for a more extended example of marshaling hits. TO INSTALL: 1. Install Geant4, CLHEP, and TOP-C To ensure compatibility, use same compiler for all three. (Currently, Geant4 recommends GNU g++ version 2.91.66 or 2.95.2 or higher. Try `g++ --version' to check.) TOP-C: http://www.ccs.neu.edu/home/gene/topc.html Geant4: http://cern.ch/geant4/source/source.html CLHEP: http://wwwinfo.cern.ch/asd/lhc++/DISTRIBUTION/hep.html 2. Source in the standard Geant4 and CLHEP environment variables. Environment variables are described in the Geant4 Installation Guide from: http://cern.ch/geant4/G4UsersDocuments/Overview/html 3. Use same compiler for TOP-C as recommended for Geant4. (currently GNU g++ version 2.91.66 or 2.95.2 or higher are recommended) (Try g++ --version to check) Make sure that topc-config is in your path. (Try `which topc-config') Test that TOP-C is working. (cd to TOP-C install dir and `make test') Also test that a standard Geant4 application (such as novice/N02) works. 4. For clarity, we assume we are parallelizing novice/N02 from the Geant4 distribution. The instructions should work for most Geant4 applications. First, copy the following files: cp ParExample.icc ParGNUmakefile procgroup $G4INSTALL/example/novice/N02/ cp Par*.cc $G4INSTALL/example/novice/N02/src/ cp Par*.hh $G4INSTALL/example/novice/N02/include/ 5. Change to the N02 directory: cd $G4INSTALL/example/novice/N02/ 6. Modify GNUmakefile by adding a line at the beginning: include ParGNUmakefile Modify exampleN02.cc by adding a line just before the definition of main(): [IMPORTANT: This must be the last include file in exampleN02.cc] #include "ParTopC.icc" 7. Optionally modify include/ParRandomState.hh If this is your first installation, I recommend using the provided default. This arranges to generate a random seed on the master for each event, and copy it to the slave processing that event. The seed is a pseudorandom one explicitly depending on the event number. This provides both reasonable efficiency and repeatability that is independent of the number of slave process and of the order in which slaves complete their tasks. 8. Modify src/ParExN02MarshaledHits.cc according to the comments in that file. You will find examples of the required modification both in ParN02 and in ParN04/src/ParExN04MarshaledHits.cc. ParN04 illustrates a larger variety of marshaled hits, but read those in ParN02 first, in order to understand the principles. The modifications are required to implement marshaling of each type of hit (e.g. for N04: ExN04TrackerHit, ExN04MuonHit, ExN04CalorimeterHit) You will need to modify the functions: MarshaledHCofThisEvent::MarshalHitsCollection MarshaledHCofThisEvent::UnmarshalSlaveHitsCollection in order to add additional cases in the `if ... else if ...' statement. You should also remove the existing case of ExN02TrackerHit if you are not parallelizing examples/novice/N02. There are already default template functions for each type of hit, T: TMarshaledHitsCollection::UnmarshalHit TMarshaledHitsCollection::MarshalHit where T is a G4VHit class such as ExN02TrackerHit. The default copies the entire class into a single buffer. If your hit class includes pointers, and if you use those pointer in your analysis (AnalyzeEvent), then you will need to override the default with an explicit specialization that marshals the pointers. You may also wish to override the default in order to write more efficient marshaling code. 9. The file `procgroup' determines on which machines the remote processes run. The makefile automatically copies the local procgroup to the G4BINDIR just before each run. For testing, you can use the provided file, as is, and it will set up two slave processes on the localhost. For production runs, you will want to change it. See the file procgroup for comments on ways to change it. Note that the default procgroup redirects standard output on the slave (within an event) into a file slave00X.out . Without this, the output of the slaves and the master would be displayed intermixed on the user console. 10. Before you compile, make sure your Geant4 environment variables point to where you want. This is important because your application binary must be in a location visible to all the computers running your parallel application. But your Geant4 source and libraries may have to stay on a different partition, where you have more disk space. For example, in my case, I was using source from $G4INSTALL, but I was building my application binary and libraries elsewhere. I had: setenv $4SYSTEM Linux-g++ I wanted to build my application libraries in: ~/geant4-cern/lib/Linux-g++ So, I did: setenv G4WORKDIR ~/geant4-cern I also made sure that ~/geant4-cern/bin/Linux-g++ existed, since the default for the application binary was $G4WORKDIR/bin/$G4SYSTEM . Because I copied the precompiled Geant4 libraries into ~/geant4-cern/lib-geant4/Linux-g++ I also needed to tell Geant4 about this non-standard location via: setenv G4LIB ~/geant4-cern/lib-geant4 For more information, see: http://cern.ch/geant4/G4UsersDocuments/UsersGuides/ (cont.) InstallationGuide/html/UnixMachines/unixMachines.html As always, if you are using shared libaries (.so) under Linux, you may have to update your LD_LIBRARY_PATH . Since you will be running a parallel application, put this in your shell initialization file (in my case, in ~.tcshrc), so it will also be availabe for the remote processes that are generated. I use the original Geant4 libraries (lib-gent4), the application libraries (lib), and the CLHEP library. So, I do: setenv LD_LIBRARY_PATH \ ${LD_LIBRARY_PATH}:$HOME/geant4-cern/geant4.4.0/lib-geant4/Linux-g++ setenv LD_LIBRARY_PATH \ ${LD_LIBRARY_PATH}:$HOME/geant4-cern/geant4.4.0/lib/Linux-g++ setenv LD_LIBRARY_PATH \ ${LD_LIBRARY_PATH}:/afs/cern.ch/sw/lhcxx/specific/redhat61/CLHEP/1.7.0.0/lib 11. Modify ParGNUmakefile according to your goals. It may be enough to only change: MACROFILE (current default: ParN02.in) If you have a datafile (e.g.: examples/novice/N04/pythia_main.data) then also set DATAFILE. The GNUmakefile copies the MACROFILE and DATAFILE to the G4BINDIR. If you require other files (e.g.: pythia_event.data for ExN04), then copy them to the G4BINDIR manually or modify ParGNUmakefile to do so. Then choose your desired makefile target and go. Try one of: make compile make run make run-debug [Runs application inside gdb.] make parclean ============================================================================ Further possible customizations: If you wish to customize your code further, note that ParGNUmakefile contains TOP-C parameters. Try other parameters such as: --TOPC-verbose=0 --TOPC-trace=0 --TOPC-slave-timeout=XXX [ units are seconds, default 1800 seconds ] --TOPC-num-slaves=XXX --TOPC-help If you wish to make some of these the default, add a line like: TOPC_OPT_trace = 0; between `main( ... )' and `TOPC_init( ... )' in the main function. Read the TOP-C manual, available from the home page http://www.ccs.neu.edu/home/gene/topc.html for further customizations and uses of TOP-C. Read ParMarshaledObj.hh for the full set of primitive marshaling functions that you might want to use in writing MarshalHit() and UnmarshalHit().