source: BAORadio/AmasNancay/trunk/analysis-scriptsAndprg-description.txt @ 575

Last change on this file since 575 was 575, checked in by campagne, 13 years ago

introduce SCRIPTPATH environment variable (jec)
File size: 8.8 KB
Line 
1Title: Status of the programs and scripts used for Amas@Nancay analysis at CCIN2P3
2Revision: 11/10/11 (JEC)
3Author: J.E Campagne 22/7/11
4----------------------------
5
6Intro:
7All these tools are located for the while in J.E Campagne private directory, so they are not
8yet available to public usage and this note is essentially for information and software tracking.
9
10-----------------------------------------------------------
11Analysis program
12------------------------------------------------------------
13o analysis.cc
14        - perfom
15                . gain spectra determination + monitoring
16                . ON-OFF spectra without calibration
17                . calibration coefficient determination + monitoring
18                . ON-OFF spectra determination (no RT calibration included)
19        - Todo
20                . include RT calibration, or a RT/BAO calibration using a calibrator source
21        - comments
22                . the STANDARD usage since end september 11 is GAIN + ON-OFF wo calibration
23               
24-----------------------------------------------------------
25Collection of  Bourne SHELL script for analysis
26------------------------------------------------------------
27
28A) Utilitary scripts to follow the DAQ & Analysis progress
29++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
30o mkarchive.sh
31---------------
32        - perform the archive of a processing in Irods
33          . create a <ppTag>Arch<apTag> directory
34            . move the cabli_*.txt files
35            . move the *.ppf / *.fits files results of the analysis
36            . for the gain...fits only a copy is done to keep a version
37            . move the Off and On directorys
38          . create new fresh On and Off directory
39
40
41o statusdaq.sh
42--------------
43        - perform
44                . scan all the /baoradio/data/AmamNancay to look at the directories and and make
45                        a summary of the Data Acquisition
46
47o statusheader.sh
48-----------------
49        - perform
50                . scan Irods directories to look at header fits files extraction progress
51               
52o statusgain.sh
53---------------
54        - perform
55                . scan Irods directories to look at FINAL gain_* files production progress
56
57o statuscalib.sh
58----------------
59        - perform
60                . scan Irods directories to look at INTERMEDIATE calibcycle files production progress
61        - Todo
62                . scan also for FINAL calib_* files progress
63o statusonoff.sh
64----------------
65        - perform
66                . scan Irods On and Off directories to look at datacycle<cycle> contents
67
68o statusCycles.sh
69-----------------
70        - perform
71                . scan Irods directories to look at sca<>.sum files to count the number of cycles per run
72
73B) Utilitary scripts to send to BQS system high level task
74++++++++++++++++++++++++++++++++++++++++++++++++++++++++++             
75o submit2bqs-hdrfitsextractor.sh
76--------------------------------
77        - SHOULD BE DONE BEFORE gain/calib/analysis
78        - send the hdrfitsextractor.sh script to BQS system     
79        - Todo list
80                - TO MIGRATE TO NEW BATCH
81
82o submit2bqs-scaextractor.sh
83--------------------------------
84        - SHOULD BE DONE BEFORE gain/calib/analysis
85        - send the scaextractor.sh script to BQS system
86                although this script is called by getscafiles lower level script
87        - Todo list
88                - TO MIGRATE TO NEW BATCH
89
90o submit2bqs-procspecmfib.sh
91-----------------------------
92        - SHOULD BE DONE for processing signals for FFT/paquet median filtering
93        - GAIN/CALIB/ON, OFF...
94        - prepare and send in batch the proc_specmfib.sh to BQS
95        - exemple
96        ./submit2bqs-procspecmfib.sh -src Abell85 -date 20110422 -type CALIBOFF -fcycle 8 -lcycle 14
97        - Todo list
98                - TO MIGRATE TO NEW BATCH
99
100o submit2bqs-anagainmaker.sh
101----------------------------
102        - done after proc_specmfib
103        - prepare and send the analyse.cc prg for gain computation
104        - exemple
105        - Todo list
106                - TO MIGRATE TO NEW BATCH
107
108o submit2bqs-anacalibmaker.sh
109----------------------------
110        - done after proc_specmfib
111        - prepare and send the analyse.cc prg for calibration computation
112        - exemples:
113        > $./submit2bqs-anacalibmaker.sh -src Abell85 -date 20110410
114
115       
116        - Todo list
117                - TO MIGRATE TO NEW BATCH
118       
119o submit2qs-onoffmaker.sh (OBSOLETE ???)
120--------------------------
121        - send the onoffmaker.sh script to BQS system
122        - Todo list
123                - TO MIGRATE TO NEW BATCH
124
125o submit2bqs-rawonoffmaker.sh (OBSOLETE ???)
126-----------------------------
127        - used after the anagainmaker
128        - prepare and send in batch the proc_rawonoff.sh to BQS system
129        - exemple submit2bqs-rawonoffmaker.sh -src Abell85 -date 20110417 -fcycle 8 -lcycle 23 -type ON
130        - Todo list
131                - TO MIGRATE TO NEW BATCH
132
133o submit2qs-gainmaker.sh (OBSOLETE)
134-------------------------
135        - send the gainmaker.sh script to BQS system
136        - Todo list
137                - TO MIGRATE TO NEW BATCH
138
139o submit2qs-calibmaker.sh (OBSOLETE)
140--------------------------
141        - send the calibmaker.sh script to BQS system
142        - Todo list
143                - TO MIGRATE TO NEW BATCH
144
145               
146C) High level scripts
147++++++++++++++++++++++         
148o hdrfitsextractor.sh
149---------------------
150        - SHOULD BE DONE BEFORE ANYTHING
151        - perform
152                . extraction of the header part of the signalXYZ.fits files and
153                . put the results into Irods   
154        - use
155                . statusdaq.sh 
156
157o anagainmaker.sh
158-----------------
159        - perform
160          . the analysis of the above fits files to determine the
161                        gains of each channel: produce gain_* files (monitoring + gain files)
162          . put the results into Irods
163        - use
164                - getscafiles.sh
165                - analysis.cc   
166
167o anacalibmaker.sh
168------------------
169        - perfom
170          . prepare the analysis of the above fits files
171          . put calib_* files (monitoring +  calibration files) into Irods
172        - use
173                - getscafiles.sh
174                - analysis.cc   
175
176
177o gainmaker.sh (OBOLETE)
178---------------
179        - perform
180                1) the extraction of signalXYZ.fits files
181                        concerned by the OFF-calibration time window
182                        of the cycle located at the middle of the run
183                2) . the analysis of the above fits files to determine the
184                        gains of each channel: produce gain_* files (monitoring + gain files)
185                        . put the results into Irods
186        - use
187                - statusdaq.sh, proc_script.sh, getscafiles.sh
188                - analysis.cc   
189       
190o calibmaker.sh (OBSOLETE)
191---------------
192        - perfom
193                1) several x-checks to see at which stage of the analysis process we are
194                2) trig the proc_script.sh script if necessary to create the spectra fits files via specmfib.cc prg
195                3) . prepare the analysis of the above fits files
196                   . put calib_* files (monitoring +  calibration files) into Irods
197        - use
198                - statusdaq.sh, proc_script.sh, getscafiles.sh
199                - analysis.cc   
200
201o onoffmaker.sh (preliminary)
202----------------------------
203        - perform
204                1) several x-checks to see at which stage of the analysis process we are
205                2) trig the proc_script.sh script if necessary to create the spectra fits files via specmfib.cc prg
206                3) prepare the analysis of the above fits files
207                4) put dataRaw_*        (raw spectra),
208                       dataBAOCalib_*   (after BAO calibration),
209                       dataBAORTCalib_* (after BAO & RT calibration),
210                       diffOnOff_*      (ON-OFF spectra)
211
212D) Lower level scripts
213++++++++++++++++++++++
214
215o proc_specmfib.sh
216------------------
217        - perform all the preparation work to send in batch the "specmfib"
218          program which perform the FFT and median/mean spectra
219          for the GAIN/CALIB/ON-OFF... task
220        - use
221          - getscafiles.sh, select.sh, statusdaq.sh, getsignalfiles.sh
222        - comments
223          - this is the pre-processing of the data to be used before
224            anaXYZ script which are post-processing
225
226
227o proc_rawonoff.sh (OBSOLETE)
228------------------
229        - this is the script to use to perform the "datacycle" contents
230        - perform all the preparation work to send in batch the "specmfib" program
231                        which perform the FFT and median/mean spectra for the ON-OFF task
232        - use
233          - getscafiles.sh, select.sh, statusdaq.sh, getsignalfiles.sh
234        - comments
235          - contrary to proc_script (below) specmfib is activated in the same thread
236
237o proc_script.sh (OBSOLETE)
238-----------------
239        - VERY BUSY script
240        - perform
241                1) all the preparation work to send in batch the "specmfib" program
242                        which perform the FFT and median/mean spectra for GAIN, CALIBRATION, ON-OFF task
243                2) the script that is sent in batch put the results into Irods 
244        - use
245                - getscafiles.sh, select.sh, getsignalfiles.sh
246        - Todo list
247                - TO MIGRATE TO NEW BATCH
248               
249
250o scaextractor.sh
251-----------------
252        - perform
253                1) extraction of the scaXYZ file
254                2) if the scaXYZ.sum and the scaXYZ.sum.trans files do not exist
255                        . create and put them into Irods
256                        . scaXYZ.sum = summary of the SCA file
257                        . scaXYZ.sum.trans = transformation of the date format of the scaXYZ.sum file   
258        - use
259                - sca.sh, transdate.sh
260               
261o sca.sh
262--------
263        - perform
264                . produce the scaXYZ.sum (= summary of the SCA file) from a scaXYZ file
265               
266o transdate.sh
267--------------
268        - perform
269                . produce the scaXYZ.sum.trans (= transformation of the date format) from the scaXYZ.sum file   
270
271o select.sh
272-----------
273        -perform
274                . the selection of the signalXYZ.fits files according to the type of task requested
275                        as ON, OFF, CALIBON, CALIBOFF, GAIN
276               
277o getscafiles.sh
278----------------
279        - perform
280                . download from Irods the scaXYZ, scaXYZ.sum and scaXYZ.sum.trans files
281        - use
282                scaextractor.sh
283               
284o getsignals.sh
285---------------
286        - perform
287                . download from Irods the signalXYZ.fits files
288               
289               
290               
291
Note: See TracBrowser for help on using the repository browser.