MultiThreaded broadcast utility

This document presents a multithreaded version of the cmt broadcast facility.

This is a standalone program and NOT a new feature of CMT. If it appears to be safe and reliable, then of course it may become part of the plain CMT.

This program (named tbroadcast - for "threaded broadcast") basically operates very much like "cmt broadcast", ie it accepts most of the same options and produces the same output.

Report on the CMT mailing list your experience, bugs, comments, ... and measurements ! Of course the expected improvement will largely depend on the type of command and on the number of processors...

Principle

The use network forms a Directed Acyclic Graph [DAG] (or if there are cycles, they are detected and properly handled).

tbroadcast first gets all "leaves" of the use graph, and launches the requested command onto them in parallel in individual [POSIX]threads.

Then, once all children of a non-leaf package are completed, the requested command is launched onto this non-leaf package [still in an individual thread].

This is recursively repeated until all packages have been reached.

At any time the number of active threads is controlled so as to never exceed a limit [set to 20 threads by default but can be modified using the -threads=n option]. A local scheduler maintains a queue of threads when the limit is reached.

The individual thread activity consists in running the requested command - as usual - in the context of the reached package. The difference with the sequential cmt broadcast of course is that the output of the run is accumulated and displayed at once when every action is completed [to avoid intermixing of outputs from different threads].

When one activity returns an error, the error status is propagated to its parent which will prematurely complete, but not before all other simultaneously running threads are actually completed.

It's only when the threads of ALL children are completed (with success or error) that a node either starts (if all children successfully completed) or immediately terminates (if one of its children failed).

How to use it:

• The source kit is available for Linux platforms only from CVS:

  > export CVSROOT=:pserver:anonymous@cvsserver.lal.in2p3.fr:/projects/cvs
  > cvs login
  [no password]
  > cd /somewhere
  > cvs co -d tbroadcast CMT/tbroadcast
  > cd tbroadcast/cmt
  > cmt config
  > gmake
              

• The application is also available for Linux platforms at CERN:

  ${CMTROOT}/../tbroadcast/v1/Linux/tbroadcast
              

• The options are:

  > tbroadcast [option...] [shell-command]
  
      where option can be:
  
        -select="<selection-list>"
        -exclude="<exclusion-list>"
        -begin=<begin-pattern>
        -global                   Reach packages in all CMTPATH items
        -threads=<n>              Change the maximum thread number [default is 20]
  
        <selection-list> is a space-separated list of sub-strings that actual paths of
                         all reached packages must match
  
        <exclusion-list> is a space-separated list of sub-strings that actual paths of
                         all reached packages must NOT match
  
        <begin-pattern>  is a sub-string matching a package path for beginning the broadcast loop
  
        -select=, -exclude=, -begin= can be combined
  
        -select=, -exclude= can be accumulated
  
  	    

  If the shell-command is omitted, the broadcast loop simply traverses the graph, reaching all packages and doing nothing. This is useful to test the traversal.

Examples

This example shows how to use tbroadcast in the context of the Atlas project

> source setup.sh -tag=7.5.0
> cd .....dist/7.5.0/AtlasRelease/*/cmt
> alias tbroadcast=$CMTROOT/../tbroadcast/v1/Linux/tbroadcast
> tbroadcast -select=/Database/
> tbroadcast -global
> tbroadcast -begin=/StoreGate/