1 | <!-- |
---|
2 | //----------------------------------------------------------- |
---|
3 | // Copyright Christian Arnault LAL-Orsay CNRS |
---|
4 | // arnault@lal.in2p3.fr |
---|
5 | // See the complete license in cmt_license.txt "http://www.cecill.info". |
---|
6 | //----------------------------------------------------------- |
---|
7 | --> |
---|
8 | |
---|
9 | <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> |
---|
10 | <html> |
---|
11 | <head> |
---|
12 | <title>MultiThreaded broadcast utility</title> |
---|
13 | </head> |
---|
14 | |
---|
15 | <font face="Arial, Helvetica" color="#000000"> |
---|
16 | |
---|
17 | <body bgcolor="#ffffff" link="#550088" alink="#007777" alink="#007777"> |
---|
18 | <h1>MultiThreaded broadcast utility</h1> |
---|
19 | |
---|
20 | <p>This document presents a multithreaded version of the cmt |
---|
21 | broadcast facility.</p> |
---|
22 | |
---|
23 | <p>This is a standalone program and NOT a new feature of CMT. If |
---|
24 | it appears to be safe and reliable, then of course it may become |
---|
25 | part of the plain CMT.</p> |
---|
26 | |
---|
27 | <p>This program (named tbroadcast - for "threaded broadcast") |
---|
28 | basically operates very much like "cmt broadcast", ie it accepts |
---|
29 | most of the same options and produces the same output.</p> |
---|
30 | |
---|
31 | <p>Report on the CMT mailing list your experience, bugs, comments, |
---|
32 | ... and measurements ! Of course the expected improvement will |
---|
33 | largely depend on the type of command and on the number of |
---|
34 | processors...</p> |
---|
35 | |
---|
36 | <hr /> |
---|
37 | <h2>Principle</h2> |
---|
38 | |
---|
39 | <blockquote> |
---|
40 | |
---|
41 | The use network forms a Directed Acyclic Graph [DAG] (or if |
---|
42 | there are cycles, they are detected and properly handled). |
---|
43 | |
---|
44 | <p><b>tbroadcast</b> first gets all "leaves" of the use graph, and |
---|
45 | launches the requested command onto them in parallel in |
---|
46 | individual [POSIX]threads.</p> |
---|
47 | |
---|
48 | <p>Then, once all children of a non-leaf package are completed, |
---|
49 | the requested command is launched onto this non-leaf package |
---|
50 | [still in an individual thread].</p> |
---|
51 | |
---|
52 | <p>This is recursively repeated until all packages have been |
---|
53 | reached.</p> |
---|
54 | |
---|
55 | <p>At any time the number of active threads is controlled so as |
---|
56 | to never exceed a limit [set to 20 threads by default but can be |
---|
57 | modified using the <b>-threads=n</b> option]. A local scheduler |
---|
58 | maintains a queue of threads when the limit is reached.</p> |
---|
59 | |
---|
60 | <p>The individual thread activity consists in running the |
---|
61 | requested command - as usual - in the context of the reached |
---|
62 | package. The difference with the sequential <b>cmt broadcast</b> |
---|
63 | of course is that the output of the run is accumulated and |
---|
64 | displayed at once when every action is completed [to avoid |
---|
65 | intermixing of outputs from different threads].</p> |
---|
66 | |
---|
67 | <p>When one activity returns an error, the error status is |
---|
68 | propagated to its parent which will prematurely complete, but |
---|
69 | not before all other simultaneously running threads are actually |
---|
70 | completed.</p> |
---|
71 | |
---|
72 | <p>It's only when the threads of ALL children are completed |
---|
73 | (with success or error) that a node either starts (if all |
---|
74 | children successfully completed) or immediately terminates (if |
---|
75 | one of its children failed).</p> |
---|
76 | |
---|
77 | </blockquote> |
---|
78 | |
---|
79 | <hr /> |
---|
80 | |
---|
81 | <h2>How to use it:</h2> |
---|
82 | |
---|
83 | <blockquote> |
---|
84 | |
---|
85 | <ul> |
---|
86 | |
---|
87 | <li> The source kit is available for Linux platforms only from CVS: |
---|
88 | |
---|
89 | <pre> |
---|
90 | > export CVSROOT=:pserver:anonymous@cvsserver.lal.in2p3.fr:/projects/cvs |
---|
91 | > cvs login |
---|
92 | [no password] |
---|
93 | > cd /somewhere |
---|
94 | > cvs co -d tbroadcast CMT/tbroadcast |
---|
95 | > cd tbroadcast/cmt |
---|
96 | > cmt config |
---|
97 | > gmake |
---|
98 | </pre> |
---|
99 | |
---|
100 | </li> |
---|
101 | |
---|
102 | <li> The application is also available for Linux platforms |
---|
103 | at CERN: |
---|
104 | |
---|
105 | <pre> |
---|
106 | ${CMTROOT}/../tbroadcast/v1/Linux/tbroadcast |
---|
107 | </pre> |
---|
108 | |
---|
109 | </li> |
---|
110 | |
---|
111 | <li> The options are: |
---|
112 | |
---|
113 | <pre> |
---|
114 | > tbroadcast [option...] [shell-command] |
---|
115 | |
---|
116 | where option can be: |
---|
117 | |
---|
118 | -select="<selection-list>" |
---|
119 | -exclude="<exclusion-list>" |
---|
120 | -begin=<begin-pattern> |
---|
121 | -global Reach packages in all CMTPATH items |
---|
122 | -threads=<n> Change the maximum thread number [default is 20] |
---|
123 | |
---|
124 | <selection-list> is a space-separated list of sub-strings that actual paths of |
---|
125 | all reached packages must match |
---|
126 | |
---|
127 | <exclusion-list> is a space-separated list of sub-strings that actual paths of |
---|
128 | all reached packages must NOT match |
---|
129 | |
---|
130 | <begin-pattern> is a sub-string matching a package path for beginning the broadcast loop |
---|
131 | |
---|
132 | -select=, -exclude=, -begin= can be combined |
---|
133 | |
---|
134 | -select=, -exclude= can be accumulated |
---|
135 | |
---|
136 | </pre> |
---|
137 | |
---|
138 | <p>If the shell-command is omitted, the broadcast loop |
---|
139 | simply traverses the graph, reaching all packages and |
---|
140 | doing nothing. This is useful to test the traversal.</p> |
---|
141 | |
---|
142 | </blockquote> |
---|
143 | |
---|
144 | <hr /> |
---|
145 | <h2>Examples</h2> |
---|
146 | |
---|
147 | <blockquote> |
---|
148 | |
---|
149 | <i>This example shows how to use tbroadcast in the context of the Atlas project</i> |
---|
150 | |
---|
151 | <pre>> source setup.sh -tag=7.5.0 |
---|
152 | > cd .....dist/7.5.0/AtlasRelease/*/cmt |
---|
153 | > alias tbroadcast=$CMTROOT/../tbroadcast/v1/Linux/tbroadcast |
---|
154 | > tbroadcast -select=/Database/ |
---|
155 | > tbroadcast -global |
---|
156 | > tbroadcast -begin=/StoreGate/ |
---|
157 | |
---|
158 | </pre> |
---|
159 | |
---|
160 | </blockquote> |
---|
161 | |
---|
162 | <hr> |
---|
163 | <address><a href="mailto:arnault@lal.in2p3.fr">Christian Arnault</a></address> |
---|
164 | <!-- Created: Fri Jan 30 12:20:51 CET 2004 --> |
---|
165 | <!-- hhmts start --> |
---|
166 | Last modified: Fri Jan 30 12:50:46 CET 2004 |
---|
167 | <!-- hhmts end --> |
---|
168 | </body> |
---|
169 | </font> |
---|
170 | </html> |
---|