Tbroadcast v2
- Title: MultiThreaded broadcast utility
- Author: Vincent Garonne
- Version: v2.0.4
- Date: 2006/09/09
Description
This is a standalone program in python and NOT a new feature of CMT. This program (named tbroadcast - for "threaded broadcast") basically operates very much like 'cmt broadcast`.
The use network forms a Directed Acyclic Graph [DAG] (or if there are cycles, they are detected and properly handled).
tbroadcast first gets all "leaves" of the use graph, and launches the requested command onto them in parallel in individual [POSIX]threads.
Then, once all children of a non-leaf package are completed, the requested command is launched onto this non-leaf package [still in an individual thread].
This is recursively repeated until all packages have been reached.
At any time the number of active threads is controlled so as to never exceed a limit [set to 20 threads by default but can be modified using the -threads=n option]. A local scheduler maintains a queue of threads when the limit is reached.
The individual thread activity consists in running the requested command - as usual - in the context of the reached package. The difference with the sequential cmt broadcast of course is that the output of the run is accumulated and displayed at once when every action is completed [to avoid intermixing of outputs from different threads].
When one activity returns an error, the error status is propagated to its parent which will prematurely complete, but not before all other simultaneously running threads are actually completed.
It's only when the threads of ALL children are completed (with success or error) that a node either starts (if all children successfully completed) or immediately terminates (if one of its children failed).
Basic usage
> tbroadcast -help Usage : > tbroadcast [global options] [<command>] # command : # <command>: command to execute # global options : # -f=<file> : Input file # -help : Print help # -local : Reach packages only within the current project # -global : Reach packages in all CMTPATH/CMTPROJECTPATH items # -ignore_cycles : Suppress automatically the cycles # -make=<file> : Generate a recursive Make, [see: http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html] # -nb=<num_worker> : Change the total number of threads[default is 20] # -no_keep_going : Exit after the first exit code > 1 found and return it in the shell # -output=<location> : Output directory to store output files with the form <package>_output.log # -error=<location> : Output directory to store error output with the form <package>_error.log # -perf=<file> : Store for each package the time for executing the command in the <file> file # -print : Print dependencies for each package # -version : version of tbroadcast # -silent : Disable print # -test : Simulate execution
Download
| Versions | URL |
| tbroadcast-v2.0.4 (Current version ) | tbroadcast-v2.0.4.tar.gz |
You can download it from the svn repository:
> svn co https://svn.lal.in2p3.fr:/projects/CMT/tbroadcast/<version> tbroadcast
How to setup
CMT should be installed and setup first :
[> wget http://www.cmtsite.org/tbroadcast/tbroadcast-<version>.tar.gz] [> tar -xvzf tbroadcast-<version>.tar.gz] > cd tbroadcast/<version>/cmt > cmt config > source setup.[c]sh
How to use tbroadcast at CERN
> source /afs/cern.ch/sw/contrib/CMT/tbroadcast/<version>/cmt/setup.[c]sh OR > source $CMTROOT/../tbroadcast/<version>/cmt/setup.[c]sh
Notes
Due to the parallel nature of threads, you have to keep some things in mind to do not use simultaneous threads for tasks were they compete for a single, scarce resource (e.g. a harddisk or stdout). This will probably be slower than taking a serialized approach, i.d. classic cmt broadcast.
