MultiThreaded broadcast utility

MultiThreaded broadcast utility

This document presents a multithreaded version of the cmt broadcast facility.

This is a standalone program and NOT a new feature of CMT. If it appears to be safe and reliable, then of course it may become part of the plain CMT.

This program (named tbroadcast - for "threaded broadcast") basically operates very much like "cmt broadcast", ie it accepts most of the same options and produces the same output.

Report on the CMT mailing list your experience, bugs, comments, ... and measurements ! Of course the expected improvement will largely depend on the type of command and on the number of processors...


Principle

The use network forms a Directed Acyclic Graph [DAG] (or if there are cycles, they are detected and properly handled).

tbroadcast first gets all "leaves" of the use graph, and launches the requested command onto them in parallel in individual [POSIX]threads.

Then, once all children of a non-leaf package are completed, the requested command is launched onto this non-leaf package [still in an individual thread].

This is recursively repeated until all packages have been reached.

At any time the number of active threads is controlled so as to never exceed a limit [set to 20 threads by default but can be modified using the -threads=n option]. A local scheduler maintains a queue of threads when the limit is reached.

The individual thread activity consists in running the requested command - as usual - in the context of the reached package. The difference with the sequential cmt broadcast of course is that the output of the run is accumulated and displayed at once when every action is completed [to avoid intermixing of outputs from different threads].

When one activity returns an error, the error status is propagated to its parent which will prematurely complete, but not before all other simultaneously running threads are actually completed.

It's only when the threads of ALL children are completed (with success or error) that a node either starts (if all children successfully completed) or immediately terminates (if one of its children failed).


How to use it:


Examples

This example shows how to use tbroadcast in the context of the Atlas project
> source setup.sh -tag=7.5.0
> cd .....dist/7.5.0/AtlasRelease/*/cmt
> alias tbroadcast=$CMTROOT/../tbroadcast/v1/Linux/tbroadcast
> tbroadcast -select=/Database/
> tbroadcast -global
> tbroadcast -begin=/StoreGate/

          

Christian Arnault
Last modified: Fri Jan 30 12:50:46 CET 2004