source: presentations/09-06-2006_SIT_CMT/Index.htm

Last change on this file was 199, checked in by garnier, 19 years ago

inversion de slidy.js et slidy-toc.js car le second a besoin du premier et generait donc une erreur javascript
File size: 10.8 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
2<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" dir="ltr" lang="en-US"><head profile="http://purl.org/net/uriprofile/">
3
4
5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
6 <meta http-equiv="Content-Language" content="ja">
7 <meta http-equiv="Content-Style-Type" content="text/css">
8 <meta http-equiv="Content-Script-Type" content="text/javascript"><title>JI'06:Intro</title>
9
10 <meta name="author" content="Vincent Garonne">
11 <meta name="description" content="description€‚">
12 <meta name="keywords" content="XHTML, CSS, HTML, (X)HTML">
13 <link rev="made" href="mailto:garonne@lal.in2p3.fr">
14 <link rel="stylesheet" href="..Tools/v2/Slidy/slidy-toc.css" type="text/css" media="screen, projection, print">
15 <link rel="stylesheet" href="../Tools/v2/css/screen.css" type="text/css" media="screen, projection, tv, print">
16
17 <script src="../Tools/v1/Slidy/slidy.js" type="text/javascript"></script>
18 <script src="../Tools/v1/Slidy/slidy-toc.js" type="text/javascript"></script>
19 </head>
20
21 <body style="font-size: 18pt;">
22
23
24<!-- Introduction [Start] -->
25
26<div style="border-style: none; display: block; visibility: visible; position: absolute;" class="slide" id="cover">
27
28 <br><br><br>
29 <p id="title">CMT:<br> tbroadcast utility <br>(version 2)</p>
30
31 <!--<p>Christian Arnault,
32 &lt;<a href="mailto:arnault@lal.in2p3.fr">arnault@lal.in2p3.fr</a>&gt;</p>-->
33 <p><a href="http://www.lal.in2p3.fr/~garonne">Vincent Garonne</a>,
34 &lt;<a href="mailto:garonne@lal.in2p3.fr">garonne@lal.in2p3.fr</a>&gt;</p>
35 <br><br>
36 <p id="date">6th September 2006</p>
37 <p id="event">Atlas/SIT, CERN</p>
38</div>
39
40<!--<div class="slide toc">
41 <h1>Table of contents</h1>
42</div> -->
43
44
45<!-- Introduction [Start] -->
46<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide" id="logo">
47<h1 id="introduction">tbroadcast v2</h1>
48 <ul>
49 <li> <em>tbroadcast</em> implements parallelism across packages</li>
50 <li> <em>tbroadcast</em> v1 was in c++ (<a href="http://www.cmtsite.org/tbroadcast.html">www.cmtsite.org/tbroadcast.html)</a></li>
51 <li> <em>tbroadcast</em> v2 has been completely rewritten in python
52 <ul>
53 <li>easier to program, modify and maintain</li>
54 <li>better management of objects/model patterns, e.g. thread pools, semaphore, ...</li>
55 </ul>
56 </li>
57 <li> <em>Details:</em>
58 <ul>
59 <li> defined on top of CMT</li>
60 <li> parses the <em>&quot;cmt show uses&quot;</em> command to get the dependency graph
61 and other package informations</li>
62 </ul>
63 </li>
64 <li>Installed at cern as a standalone package
65 <ul>
66 <li>To use it:<pre>&gt; source /afs/cern.ch/sw/contrib/CMT/tbroadcast/v2/cmt/source.[c]sh</pre></li>
67
68 </ul>
69 </li>
70 <li>Will be SOON distributed as CMT external stuffs
71 <ul>
72 <li>could be invoked as cmt native function, i.d. CMT action concept
73 <pre>&gt; cmt tbroadcast &lt;options&gt;</pre></li>
74
75 <li>For the moment available by svn
76 <pre>&gt; svn co http://svn.lal.in2p3.fr:/projects/CMT/tbroadcast/</pre></li>
77 </ul>
78 </li>
79</div>
80
81<!-- Introduction [1] -->
82<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide" id="logo">
83
84 <h1>Command and options</h1>
85
86 <dl>
87 <dt><pre>&gt; tbroadcast <b>&lt;options&gt;</b> <b>&lt;command&gt;</b></pre></dt>
88 <dd>&lt;command&gt;: <pre>Command to propagate</pre></dd>
89 <dd>&lt;options&gt;:
90<pre>-nb=&lt;N&gt;: N - Total number of threads (20 by default)
91-local: run only in the current project (by default, we do a global broadcast)
92-output=&lt;directory&gt;: store the output in real time for every package in directory
93-error=&lt;directory&gt;: store the error output separately if specified
94-ignore_cycles : ignore cycles in packages.
95Cyclic dependencies still exist in few packages. One can check with "cmt -private show
96cycles". It will only work for cycles which have cmt dependency not for build
97dependency.
98-test: Simulate execution
99-print: Print for each package its dependencies
100...
101-help: Print help
102</pre>
103 </dd>
104 </dl>
105<!---->
106</div>
107
108
109<!-- Diapo -->
110 <div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide">
111
112 <h1>Simple example</h1>
113 <table class="noborder">
114 <tbody>
115 <tr>
116 <td class="noborder">
117 <ul>
118 <li> Let A, B, C, D, E be packages</li>
119 <br>
120 <li> Let consider the following dependencies:
121 <ul>
122 <li>A uses {B, C}</li>
123 <li>C uses {D, E}</li>
124 </ul>
125 </li>
126 </ul>
127 <br><br>
128 </td>
129 <td class="noborder">
130 <ul>
131 <li style="position:absolute; left: 700px; top:150px;">Corresponding execution graph:
132 <img src="Images/graph.png" style="margin-top:1em" alt="Dependency graph"/></li>
133 <!-- -->
134 </ul>
135 </td>
136 </tr>
137 <tr>
138 <td class="noborder">
139 <ul>
140 <li>Corresponding Gantt chart:</li>
141 </ul>
142 </td>
143 </tr>
144 <tr>
145 <td class="noborder">
146 <ul class="incremental">
147 <ul style="margin-left:1em">1) Sequential mode:</ul>
148 </ul>
149 <div class="incremental" style="position: relative;">
150 <img src="Images/Gantt.png" height="272" width="575"/>
151 <img src="Images/Gantt1.png" style="position: absolute; left: 0pt; top: 0pt;" height="272" width="575">
152 </div>
153 <!-- -->
154 </td>
155 <td class="noborder">
156 <ul class="incremental">
157 <ul>2) Multi-threaded mode:</ul>
158 <ul><img src="Images/Gantt2.png" height="272" width="575"/>
159 </ul>
160 </ul>
161 <!-- -->
162 </td>
163 </tr>
164 </tbody>
165 </table>
166</div>
167
168
169<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide">
170
171 <h1>Evaluation &amp; Performances</h1>
172
173 <ul>
174 <li>Build of AtlasCore 2.2.0 with a total of 136 packages and N=60 threads </li>
175 <li>On lxplus with AFS (/afs/cern.ch/atlas/maxidisk/d57/testarea/AtlasCore/2.2.0)
176 <ul>
177 <li>Node characteristics: Intel(R) Xeon(TM) CPU 2.80GHz, Memory ~2 Giga<li>
178 </ul>
179 </li>
180 <br>
181 <li> Results: </li>
182 <table summary="€‚">
183 <thead>
184 <tr>
185 <th width="300"><p style="text-align: right;">Commands</p><p style="text-align: left;">Mode</p></th>
186 <th>cmt config</th>
187 <th>cmt make</th>
188 </tr>
189 </thead>
190 <tbody>
191 <tr>
192 <th width="300" style="text-align: left;">Sequential</th>
193 <td style="text-align: center;" >5:17.81 (min.)</td>
194 <td style="text-align: center;" >1:49:44.7 (h.)</td>
195 </tr>
196 <tr>
197 <th width="300" style="text-align: left;">Multi-threaded</th>
198 <td style="text-align: center;" >1:49:43.65 (min.)</td>
199 <td style="text-align: center;" > 55:47.76 (h.)</td>
200 </tr>
201 </tbody>
202 </table>
203 <br>
204 <li> Note: AFS provides non determistic access time...</li>
205 <br>
206 <li> Compilation validation
207 <ul>
208 <li> check the total number of object, archive and library files between sequential and parallel compilation
209 <ul>
210 <li>Successful after an intensive multi-threaded debugging session</li>
211 <li>NICOS should be used in the future</li>
212 <li>More tests are undergoing with other projects</li>
213 </ul>
214 </li>
215 </ul>
216 </li>
217 </ul>
218</div>
219
220
221<!-- Diapo -->
222<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide" id="logo">
223<h1>Gantt chart: AtlasCore compilation (1)</h1>
224<!--<ul><li>card(thread pool)=60</li></ul>-->
225 <center><img src="Images/AtlasCoreGantt.jpg" alt="Gantt" height="600" width="950"></center>
226 <!--height="798" width="1112" -->
227</div>
228
229<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide" id="logo">
230<h1>Gantt chart: AtlasCore compilation (2)</h1>
231 <center><img src="Images/AtlasCoreGantt2.jpg" alt="Gantt" height="500" width="1056"></center>
232 <!--height="272" width="575" -->
233</div>
234
235
236
237<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide">
238 <h1>Conclusion</h1>
239 <ul>
240 <li> tbroadcast gives clearly better compilation time (~55 min. for AtlasCore)</li>
241 <li> tbroadcast utilization will be another quality test for Atlas software as each shared resource between packages
242 must be cleany specified</li>
243 <li>Could be used with distcc/gmake -j&lt;N&gt; (currently under testing by Emil)
244 <ul>
245 <li> a possible factor ~2 for nightlies :) ?</li>
246 </ul>
247
248 </li>
249 <br>
250 <li> Future plans:
251 <ul>
252 <li> more tests, tests and tests</li>
253 <li> Adapt tbroadcast at the project level for providing sequential and multi-threaded mode</li>
254 <li> Is a distributed Master/slaves model with a worker node pool relevant at the project level ?</li>
255 </ul>
256 </li>
257</li>
258 </ul>
259
260</div>
261
262
263<!-- Thank You [Start] -->
264<div style="border-style: none; display: none; visibility: hidden; position: absolute;" class="slide" id="thanks">
265 <p>
266 Questions ?
267 </p>
268
269</div>
270
271
272
273<!-- [End] -->
274
275</body>
276</html>
Note: See TracBrowser for help on using the repository browser.