source: BAORadio/AmasNancay/trunk/analysis-scriptsAndprg-description.txt @ 613

Title: Status of the programs and scripts used for Amas@Nancay analysis at CCIN2P3
Revision: 28/11/11 (JEC+AST)
Author: J.E Campagne 22/7/11
----------------------------

Intro:
All these tools are located for the while in J.E Campagne private directory, so they are not
yet available to public usage and this note is essentially for information and software tracking.

-----------------------------------------------------------
Analysis program
------------------------------------------------------------
o analysis.cc
        - perfom
                . gain spectra determination + monitoring
                . ON-OFF spectra without calibration 
                . calibration coefficient determination + monitoring
                . ON-OFF spectra determination (no RT calibration included)
        - Todo
                . include RT calibration, or a RT/BAO calibration using a calibrator source
        - comments
                . the STANDARD usage since end september 11 is GAIN + ON-OFF wo calibration



                
-----------------------------------------------------------
Collection of  Bourne SHELL script for analysis
------------------------------------------------------------

A) Utilitary scripts to follow the DAQ & Analysis progress
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
o mkarchive.sh
---------------
        - perform the archive of a processing in Irods
          . create a <ppTag>Arch<apTag> directory 
            . move the cabli_*.txt files
            . move the *.ppf / *.fits files results of the analysis
            . for the gain...fits only a copy is done to keep a version
            . move the Off and On directorys
          . create new fresh On and Off directory


o statusdaq.sh
--------------
        - perform
                . scan all the /baoradio/data/AmamNancay to look at the directories and and make
                        a summary of the Data Acquisition

o statusheader.sh
-----------------
        - perform
                . scan Irods directories to look at header fits files extraction progress 
                
o statusgain.sh
---------------
        - perform
                . scan Irods directories to look at FINAL gain_* files production progress 

o statuscalib.sh
----------------
        - perform
                . scan Irods directories to look at INTERMEDIATE calibcycle files production progress
        - Todo
                . scan also for FINAL calib_* files progress
o statusonoff.sh
----------------
        - perform
                . scan Irods On and Off directories to look at datacycle<cycle> contents

o statusCycles.sh 
-----------------
        - perform
                . scan Irods directories to look at sca<>.sum files to count the number of cycles per run

B) Utilitary scripts to send to GE system high level task (note old BQS batch is obsolete now)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++              
o submit2ge-hdrfitsextractor.sh
--------------------------------
        - SHOULD BE DONE BEFORE gain/calib/analysis
        - send the hdrfitsextractor.sh script to GE system      


o submit2ge-procspecmfib.sh
-----------------------------
        - SHOULD BE DONE for processing signals for FFT/paquet median filtering
        - GAIN/CALIB/ON, OFF... 
        - prepare and send in batch the proc_specmfib.sh to GE
        - exemple
        ./submit2ge-procspecmfib.sh -src Abell85 -date 20110422 -type CALIBOFF -fcycle 8 -lcycle 14

o submit2ge-anagainmaker.sh
----------------------------
        - done after proc_specmfib
        - prepare and send the analyse.cc prg for gain computation
        - exemple

o submit2ge-anacalibmaker.sh
----------------------------
        - done after proc_specmfib
        - prepare and send the analyse.cc prg for calibration computation
        - exemples: 
        > $./submit2ge-anacalibmaker.sh -src Abell85 -date 20110410

o submit2ge-scaextractor.sh
--------------------------------
        - SHOULD BE DONE BEFORE gain/calib/analysis
        - send the scaextractor.sh script to GE system
                although this script is called by getscafiles lower level script
        
                
C) High level scripts 
++++++++++++++++++++++          
o hdrfitsextractor.sh
---------------------
        - SHOULD BE DONE BEFORE ANYTHING
        - perform
                . extraction of the header part of the signalXYZ.fits files and
                . put the results into Irods    
        - use 
                . statusdaq.sh  

o anagainmaker.sh
-----------------
        - perform 
          . the analysis to determine the 
                        gains of each channel: produce gain_* files (monitoring + gain files)
          . put the results into Irods
        - use 
                - getscafiles.sh
                - analysis.cc   

o anacalibmaker.sh
------------------
        - perfom
          . prepare the analysis
          . put calib_* files (monitoring +  calibration files) into Irods
        - use 
                - getscafiles.sh
                - analysis.cc   



o anarawonoffmaker.sh
---------------------
        - perform
                . prepare the analysis
                . put dataRaw_*        (raw spectra), 
                      diffOnOffRaw_*   (ON-OFF spectra although not used by final analysis) 

D) Lower level scripts
++++++++++++++++++++++

o proc_specmfib.sh
------------------
        - perform all the preparation work to send in batch the "specmfib" 
          program which perform the FFT and median/mean spectra 
          for the GAIN/CALIB/ON-OFF... task
        - use
          - getscafiles.sh, select.sh, statusdaq.sh, getsignalfiles.sh
        - comments
          - this is the pre-processing of the data to be used before
            anaXYZ script which are post-processing


o scaextractor.sh
-----------------
        - perform
                1) extraction of the scaXYZ file
                2) if the scaXYZ.sum and the scaXYZ.sum.trans files do not exist
                        . create and put them into Irods
                        . scaXYZ.sum = summary of the SCA file
                        . scaXYZ.sum.trans = transformation of the date format of the scaXYZ.sum file   
        - use
                - sca.sh, transdate.sh
                
o sca.sh
--------
        - perform
                . produce the scaXYZ.sum (= summary of the SCA file) from a scaXYZ file
                
o transdate.sh
--------------
        - perform
                . produce the scaXYZ.sum.trans (= transformation of the date format) from the scaXYZ.sum file   

o select.sh
-----------
        -perform 
                . the selection of the signalXYZ.fits files according to the type of task requested
                        as ON, OFF, CALIBON, CALIBOFF, GAIN
                
o getscafiles.sh
----------------
        - perform
                . download from Irods the scaXYZ, scaXYZ.sum and scaXYZ.sum.trans files 
        - use
                scaextractor.sh
                
o getsignals.sh
---------------
        - perform
                . download from Irods the signalXYZ.fits files