[2] | 1 | <!-- |
---|
| 2 | //----------------------------------------------------------- |
---|
| 3 | // Copyright Christian Arnault LAL-Orsay CNRS |
---|
| 4 | // arnault@lal.in2p3.fr |
---|
| 5 | // See the complete license in cmt_license.txt "http://www.cecill.info". |
---|
| 6 | //----------------------------------------------------------- |
---|
| 7 | --> |
---|
| 8 | |
---|
| 9 | <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> |
---|
| 10 | <html> |
---|
| 11 | <head> |
---|
[268] | 12 | <!-- phpmyvisites --> |
---|
| 13 | <a href="http://www.phpmyvisites.net/" title="phpMyVisites | Open source web analytics" |
---|
| 14 | onclick="window.open(this.href);return(false);"> |
---|
| 15 | <script type="text/javascript"> |
---|
| 16 | <!-- |
---|
| 17 | var a_vars = Array(); |
---|
| 18 | var pagename=''; |
---|
| 19 | |
---|
| 20 | var phpmyvisitesSite = 1; |
---|
| 21 | var phpmyvisitesURL = "http://www.cmtsite.org/phpmv2/phpmyvisites.php"; |
---|
| 22 | //--> |
---|
| 23 | </script> |
---|
| 24 | <script language=javascript src="http://www.cmtsite.org/phpmv2/phpmyvisites.js" |
---|
| 25 | type="text/javascript"></script> |
---|
| 26 | <noscript> |
---|
| 27 | <p>phpMyVisites | Open source web analytics |
---|
| 28 | <img src="http://www.cmtsite.org/phpmv2/phpmyvisites.php" alt="phpMyVisites" style="border:0" /> |
---|
| 29 | </p> |
---|
| 30 | </noscript> |
---|
| 31 | </a> |
---|
| 32 | <!-- /phpmyvisites --> |
---|
| 33 | |
---|
| 34 | |
---|
[2] | 35 | <title>MultiThreaded broadcast utility</title> |
---|
| 36 | </head> |
---|
| 37 | |
---|
| 38 | <font face="Arial, Helvetica" color="#000000"> |
---|
| 39 | |
---|
| 40 | <body bgcolor="#ffffff" link="#550088" alink="#007777" alink="#007777"> |
---|
| 41 | <h1>MultiThreaded broadcast utility</h1> |
---|
| 42 | |
---|
| 43 | <p>This document presents a multithreaded version of the cmt |
---|
| 44 | broadcast facility.</p> |
---|
| 45 | |
---|
| 46 | <p>This is a standalone program and NOT a new feature of CMT. If |
---|
| 47 | it appears to be safe and reliable, then of course it may become |
---|
| 48 | part of the plain CMT.</p> |
---|
| 49 | |
---|
| 50 | <p>This program (named tbroadcast - for "threaded broadcast") |
---|
| 51 | basically operates very much like "cmt broadcast", ie it accepts |
---|
| 52 | most of the same options and produces the same output.</p> |
---|
| 53 | |
---|
| 54 | <p>Report on the CMT mailing list your experience, bugs, comments, |
---|
| 55 | ... and measurements ! Of course the expected improvement will |
---|
| 56 | largely depend on the type of command and on the number of |
---|
| 57 | processors...</p> |
---|
| 58 | |
---|
| 59 | <hr /> |
---|
| 60 | <h2>Principle</h2> |
---|
| 61 | |
---|
| 62 | <blockquote> |
---|
| 63 | |
---|
| 64 | The use network forms a Directed Acyclic Graph [DAG] (or if |
---|
| 65 | there are cycles, they are detected and properly handled). |
---|
| 66 | |
---|
| 67 | <p><b>tbroadcast</b> first gets all "leaves" of the use graph, and |
---|
| 68 | launches the requested command onto them in parallel in |
---|
| 69 | individual [POSIX]threads.</p> |
---|
| 70 | |
---|
| 71 | <p>Then, once all children of a non-leaf package are completed, |
---|
| 72 | the requested command is launched onto this non-leaf package |
---|
| 73 | [still in an individual thread].</p> |
---|
| 74 | |
---|
| 75 | <p>This is recursively repeated until all packages have been |
---|
| 76 | reached.</p> |
---|
| 77 | |
---|
| 78 | <p>At any time the number of active threads is controlled so as |
---|
| 79 | to never exceed a limit [set to 20 threads by default but can be |
---|
| 80 | modified using the <b>-threads=n</b> option]. A local scheduler |
---|
| 81 | maintains a queue of threads when the limit is reached.</p> |
---|
| 82 | |
---|
| 83 | <p>The individual thread activity consists in running the |
---|
| 84 | requested command - as usual - in the context of the reached |
---|
| 85 | package. The difference with the sequential <b>cmt broadcast</b> |
---|
| 86 | of course is that the output of the run is accumulated and |
---|
| 87 | displayed at once when every action is completed [to avoid |
---|
| 88 | intermixing of outputs from different threads].</p> |
---|
| 89 | |
---|
| 90 | <p>When one activity returns an error, the error status is |
---|
| 91 | propagated to its parent which will prematurely complete, but |
---|
| 92 | not before all other simultaneously running threads are actually |
---|
| 93 | completed.</p> |
---|
| 94 | |
---|
| 95 | <p>It's only when the threads of ALL children are completed |
---|
| 96 | (with success or error) that a node either starts (if all |
---|
| 97 | children successfully completed) or immediately terminates (if |
---|
| 98 | one of its children failed).</p> |
---|
| 99 | |
---|
| 100 | </blockquote> |
---|
| 101 | |
---|
| 102 | <hr /> |
---|
| 103 | |
---|
| 104 | <h2>How to use it:</h2> |
---|
| 105 | |
---|
| 106 | <blockquote> |
---|
| 107 | |
---|
| 108 | <ul> |
---|
| 109 | |
---|
| 110 | <li> The source kit is available for Linux platforms only from CVS: |
---|
| 111 | |
---|
| 112 | <pre> |
---|
| 113 | > export CVSROOT=:pserver:anonymous@cvsserver.lal.in2p3.fr:/projects/cvs |
---|
| 114 | > cvs login |
---|
| 115 | [no password] |
---|
| 116 | > cd /somewhere |
---|
| 117 | > cvs co -d tbroadcast CMT/tbroadcast |
---|
| 118 | > cd tbroadcast/cmt |
---|
| 119 | > cmt config |
---|
| 120 | > gmake |
---|
| 121 | </pre> |
---|
| 122 | |
---|
| 123 | </li> |
---|
| 124 | |
---|
| 125 | <li> The application is also available for Linux platforms |
---|
| 126 | at CERN: |
---|
| 127 | |
---|
| 128 | <pre> |
---|
| 129 | ${CMTROOT}/../tbroadcast/v1/Linux/tbroadcast |
---|
| 130 | </pre> |
---|
| 131 | |
---|
| 132 | </li> |
---|
| 133 | |
---|
| 134 | <li> The options are: |
---|
| 135 | |
---|
| 136 | <pre> |
---|
| 137 | > tbroadcast [option...] [shell-command] |
---|
| 138 | |
---|
| 139 | where option can be: |
---|
| 140 | |
---|
| 141 | -select="<selection-list>" |
---|
| 142 | -exclude="<exclusion-list>" |
---|
| 143 | -begin=<begin-pattern> |
---|
| 144 | -global Reach packages in all CMTPATH items |
---|
| 145 | -threads=<n> Change the maximum thread number [default is 20] |
---|
| 146 | |
---|
| 147 | <selection-list> is a space-separated list of sub-strings that actual paths of |
---|
| 148 | all reached packages must match |
---|
| 149 | |
---|
| 150 | <exclusion-list> is a space-separated list of sub-strings that actual paths of |
---|
| 151 | all reached packages must NOT match |
---|
| 152 | |
---|
| 153 | <begin-pattern> is a sub-string matching a package path for beginning the broadcast loop |
---|
| 154 | |
---|
| 155 | -select=, -exclude=, -begin= can be combined |
---|
| 156 | |
---|
| 157 | -select=, -exclude= can be accumulated |
---|
| 158 | |
---|
| 159 | </pre> |
---|
| 160 | |
---|
| 161 | <p>If the shell-command is omitted, the broadcast loop |
---|
| 162 | simply traverses the graph, reaching all packages and |
---|
| 163 | doing nothing. This is useful to test the traversal.</p> |
---|
| 164 | |
---|
| 165 | </blockquote> |
---|
| 166 | |
---|
| 167 | <hr /> |
---|
| 168 | <h2>Examples</h2> |
---|
| 169 | |
---|
| 170 | <blockquote> |
---|
| 171 | |
---|
| 172 | <i>This example shows how to use tbroadcast in the context of the Atlas project</i> |
---|
| 173 | |
---|
| 174 | <pre>> source setup.sh -tag=7.5.0 |
---|
| 175 | > cd .....dist/7.5.0/AtlasRelease/*/cmt |
---|
| 176 | > alias tbroadcast=$CMTROOT/../tbroadcast/v1/Linux/tbroadcast |
---|
| 177 | > tbroadcast -select=/Database/ |
---|
| 178 | > tbroadcast -global |
---|
| 179 | > tbroadcast -begin=/StoreGate/ |
---|
| 180 | |
---|
| 181 | </pre> |
---|
| 182 | |
---|
| 183 | </blockquote> |
---|
| 184 | |
---|
| 185 | <hr> |
---|
| 186 | <address><a href="mailto:arnault@lal.in2p3.fr">Christian Arnault</a></address> |
---|
| 187 | <!-- Created: Fri Jan 30 12:20:51 CET 2004 --> |
---|
| 188 | <!-- hhmts start --> |
---|
| 189 | Last modified: Fri Jan 30 12:50:46 CET 2004 |
---|
| 190 | <!-- hhmts end --> |
---|
| 191 | </body> |
---|
| 192 | </font> |
---|
| 193 | </html> |
---|