Title: Status of the programs and scripts used for Amas@Nancay analysis at CCIN2P3 Revision: 11/10/11 (JEC) Author: J.E Campagne 22/7/11 ---------------------------- Intro: All these tools are located for the while in J.E Campagne private directory, so they are not yet available to public usage and this note is essentially for information and software tracking. ----------------------------------------------------------- Analysis program ------------------------------------------------------------ o analysis.cc - perfom . gain spectra determination + monitoring . ON-OFF spectra without calibration . calibration coefficient determination + monitoring . ON-OFF spectra determination (no RT calibration included) - Todo . include RT calibration, or a RT/BAO calibration using a calibrator source - comments . the STANDARD usage since end september 11 is GAIN + ON-OFF wo calibration ----------------------------------------------------------- Collection of Bourne SHELL script for analysis ------------------------------------------------------------ A) Utilitary scripts to follow the DAQ & Analysis progress ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ o mkarchive.sh --------------- - perform the archive of a processing in Irods . create a Arch directory . move the cabli_*.txt files . move the *.ppf / *.fits files results of the analysis . for the gain...fits only a copy is done to keep a version . move the Off and On directorys . create new fresh On and Off directory o statusdaq.sh -------------- - perform . scan all the /baoradio/data/AmamNancay to look at the directories and and make a summary of the Data Acquisition o statusheader.sh ----------------- - perform . scan Irods directories to look at header fits files extraction progress o statusgain.sh --------------- - perform . scan Irods directories to look at FINAL gain_* files production progress o statuscalib.sh ---------------- - perform . scan Irods directories to look at INTERMEDIATE calibcycle files production progress - Todo . scan also for FINAL calib_* files progress o statusonoff.sh ---------------- - perform . scan Irods On and Off directories to look at datacycle contents o statusCycles.sh ----------------- - perform . scan Irods directories to look at sca<>.sum files to count the number of cycles per run B) Utilitary scripts to send to BQS system high level task ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ o submit2bqs-hdrfitsextractor.sh -------------------------------- - SHOULD BE DONE BEFORE gain/calib/analysis - send the hdrfitsextractor.sh script to BQS system - Todo list - TO MIGRATE TO NEW BATCH o submit2bqs-scaextractor.sh -------------------------------- - SHOULD BE DONE BEFORE gain/calib/analysis - send the scaextractor.sh script to BQS system although this script is called by getscafiles lower level script - Todo list - TO MIGRATE TO NEW BATCH o submit2bqs-procspecmfib.sh ----------------------------- - SHOULD BE DONE for processing signals for FFT/paquet median filtering - GAIN/CALIB/ON, OFF... - prepare and send in batch the proc_specmfib.sh to BQS - exemple ./submit2bqs-procspecmfib.sh -src Abell85 -date 20110422 -type CALIBOFF -fcycle 8 -lcycle 14 - Todo list - TO MIGRATE TO NEW BATCH o submit2bqs-anagainmaker.sh ---------------------------- - done after proc_specmfib - prepare and send the analyse.cc prg for gain computation - exemple - Todo list - TO MIGRATE TO NEW BATCH o submit2bqs-anacalibmaker.sh ---------------------------- - done after proc_specmfib - prepare and send the analyse.cc prg for calibration computation - exemples: > $./submit2bqs-anacalibmaker.sh -src Abell85 -date 20110410 - Todo list - TO MIGRATE TO NEW BATCH o submit2qs-onoffmaker.sh (OBSOLETE ???) -------------------------- - send the onoffmaker.sh script to BQS system - Todo list - TO MIGRATE TO NEW BATCH o submit2bqs-rawonoffmaker.sh (OBSOLETE ???) ----------------------------- - used after the anagainmaker - prepare and send in batch the proc_rawonoff.sh to BQS system - exemple submit2bqs-rawonoffmaker.sh -src Abell85 -date 20110417 -fcycle 8 -lcycle 23 -type ON - Todo list - TO MIGRATE TO NEW BATCH o submit2qs-gainmaker.sh (OBSOLETE) ------------------------- - send the gainmaker.sh script to BQS system - Todo list - TO MIGRATE TO NEW BATCH o submit2qs-calibmaker.sh (OBSOLETE) -------------------------- - send the calibmaker.sh script to BQS system - Todo list - TO MIGRATE TO NEW BATCH C) High level scripts ++++++++++++++++++++++ o hdrfitsextractor.sh --------------------- - SHOULD BE DONE BEFORE ANYTHING - perform . extraction of the header part of the signalXYZ.fits files and . put the results into Irods - use . statusdaq.sh o anagainmaker.sh ----------------- - perform . the analysis of the above fits files to determine the gains of each channel: produce gain_* files (monitoring + gain files) . put the results into Irods - use - getscafiles.sh - analysis.cc o anacalibmaker.sh ------------------ - perfom . prepare the analysis of the above fits files . put calib_* files (monitoring + calibration files) into Irods - use - getscafiles.sh - analysis.cc o gainmaker.sh (OBOLETE) --------------- - perform 1) the extraction of signalXYZ.fits files concerned by the OFF-calibration time window of the cycle located at the middle of the run 2) . the analysis of the above fits files to determine the gains of each channel: produce gain_* files (monitoring + gain files) . put the results into Irods - use - statusdaq.sh, proc_script.sh, getscafiles.sh - analysis.cc o calibmaker.sh (OBSOLETE) --------------- - perfom 1) several x-checks to see at which stage of the analysis process we are 2) trig the proc_script.sh script if necessary to create the spectra fits files via specmfib.cc prg 3) . prepare the analysis of the above fits files . put calib_* files (monitoring + calibration files) into Irods - use - statusdaq.sh, proc_script.sh, getscafiles.sh - analysis.cc o onoffmaker.sh (preliminary) ---------------------------- - perform 1) several x-checks to see at which stage of the analysis process we are 2) trig the proc_script.sh script if necessary to create the spectra fits files via specmfib.cc prg 3) prepare the analysis of the above fits files 4) put dataRaw_* (raw spectra), dataBAOCalib_* (after BAO calibration), dataBAORTCalib_* (after BAO & RT calibration), diffOnOff_* (ON-OFF spectra) D) Lower level scripts ++++++++++++++++++++++ o proc_specmfib.sh ------------------ - perform all the preparation work to send in batch the "specmfib" program which perform the FFT and median/mean spectra for the GAIN/CALIB/ON-OFF... task - use - getscafiles.sh, select.sh, statusdaq.sh, getsignalfiles.sh - comments - this is the pre-processing of the data to be used before anaXYZ script which are post-processing o proc_rawonoff.sh (OBSOLETE) ------------------ - this is the script to use to perform the "datacycle" contents - perform all the preparation work to send in batch the "specmfib" program which perform the FFT and median/mean spectra for the ON-OFF task - use - getscafiles.sh, select.sh, statusdaq.sh, getsignalfiles.sh - comments - contrary to proc_script (below) specmfib is activated in the same thread o proc_script.sh (OBSOLETE) ----------------- - VERY BUSY script - perform 1) all the preparation work to send in batch the "specmfib" program which perform the FFT and median/mean spectra for GAIN, CALIBRATION, ON-OFF task 2) the script that is sent in batch put the results into Irods - use - getscafiles.sh, select.sh, getsignalfiles.sh - Todo list - TO MIGRATE TO NEW BATCH o scaextractor.sh ----------------- - perform 1) extraction of the scaXYZ file 2) if the scaXYZ.sum and the scaXYZ.sum.trans files do not exist . create and put them into Irods . scaXYZ.sum = summary of the SCA file . scaXYZ.sum.trans = transformation of the date format of the scaXYZ.sum file - use - sca.sh, transdate.sh o sca.sh -------- - perform . produce the scaXYZ.sum (= summary of the SCA file) from a scaXYZ file o transdate.sh -------------- - perform . produce the scaXYZ.sum.trans (= transformation of the date format) from the scaXYZ.sum file o select.sh ----------- -perform . the selection of the signalXYZ.fits files according to the type of task requested as ON, OFF, CALIBON, CALIBOFF, GAIN o getscafiles.sh ---------------- - perform . download from Irods the scaXYZ, scaXYZ.sum and scaXYZ.sum.trans files - use scaextractor.sh o getsignals.sh --------------- - perform . download from Irods the signalXYZ.fits files