source: Sophya/trunk/SophyaLib/Manual/sophya.tex@ 1347

\documentclass[twoside,11pt]{article}
%  Package standard : Utilisation de caracteres accentues, mode francais et graphique
\usepackage{url}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage{graphicx}
% package a mettre pour faire du pdf
\usepackage{palatino}

%  Extension de symboles mathematiques
\usepackage{amssymb}

%  Definition pour Docs Sophya
\usepackage{defsophya}



\begin{document}

\begin{titlepage}
%  The title page - top of the page with the title of the paper
\titrehp{Sophya  \\ An overview }
%  Authors list
\auteurs{
R. Ansari            &  ansari@lal.in2p3.fr       \\
E. Aubourg           &  aubourg@hep.saclay.cea.fr \\
G. Le Meur           &  lemeur@lal.in2p3.fr       \\
C. Magneville        &  cmv@hep.saclay.cea.fr     \\
S. Henrot-Versille   &  versille@in2p3.fr     
}
% \auteursall
%  The title page - bottom of the page with the paper number
\titrebp{1}
\end{titlepage}

\tableofcontents

\newpage

\section{Introduction}

   {\bf SOPHYA} ({\bf SO}ftware for {\bf PHY}sics {\bf A}nalysis) 
is a collection of C++ classes designed for numerical and
physics analysis software development. Our goal is to provide
easy to use, yet powerful classes which can be used by scientists.
We have decided to use as much as possible available 
numerical analysis libraries, encapsulating them whenever
possible.
  
  The SOPHYA design and implementation has been carried out
with the specific goal of providing the general framework for
the Planck-HFI data processing software. However, most of the 
packages presented here have a more general scope than the CMB analysis
and Planck mission problem.
  The source directory tree 
\footnote{ CVS server: cvsserver.lal.in2p3.fr:/projects/Eros/CVSPlanck} 
is organised into a number of modules. 

\begin{itemize}
\item[] {\bf Mgr/}  Scripts for code management, 
makefile generation and software installation
\item[] {\bf SysTools/} General architecture support classes such 
as {\tt PPersist, NDataBlock<T>}, and few utility classes 
({\tt DataCard, DVList} \ldots). 
\item[] {\bf TArray/} template numerical arrays, vectors and matrices \\
({\tt PixelMap<T> SphericalMap<T>} \ldots) 
\item[] {\bf NTools/} Some standard numerical analysis tools 
(linear, and non linear parameter fitting, FFT, \ldots)
\item[] {\bf HiStats/}  Histogram-ming and data set handling classes \\
({\tt Histo Histo2D NTuple XNTuple} \ldots)
\end{itemize}

The modules listed below are more tightly related to the 
CMB (Cosmic Microwave Background) data analysis problem:
\begin{itemize}
\item[] {\bf SkyMap/} Local and full sky maps, and few geometry 
handling utility classes. \\
({\tt PixelMap<T>, LocalMap<T>, SphericalMap<T>, \ldots})
\item[] {\bf SkyT/} 
classes for spectral emission and detector frequency response modelling \\
({\tt SpectralResponse, RadSpectra, BlackBody} \ldots)
\item[] {\bf Samba/} Spherical harmonic analysis.
\end{itemize}

The following modules contain the interface classes with 
external libraries:
\begin{itemize}
\item[] {\bf FitsIOServer/} Classes for handling file input-output 
in FITS format using the cfitsio library.
\item[] {\bf LinAlg/} Interface with Lapack linear algebra package
\item[] {\bf IFFTW/} Interface with FFTW package (libfftw.a)
\end{itemize}

Other modules:
\begin{itemize}
\item[] {\bf Tests/} Simple test programs
\item[] {\bf PrgUtil/} Various utility programs (runcxx, scanppf, scanfits, \ldots)
\item[] {\bf PMixer/} skymixer and related programs
\item[] {\bf ProgPI/}  interactive analysis tool - It should be noted that 
this module uses the SOPHYA class library and is based on {\bf PI}
which is a C++ library defining a complete GUI program 
architecture. An additional module (PIext) define the interactive
analysis program framework and the interfaces with the objects
in SOPHYA.  The {\bf PI/} \footnote{the PI package documentation 
is available from {\bf http://www.lal.in2p3.fr/recherche/eros/PeidaDoc/} } 
and {\bf PIext/} modules are not currently part 
of the SOPHYA CVS structure.
\end{itemize}   

\newpage 

\section{Using Sophya}
Two environment variables {\bf DPCBASEREP} and {\bf EROSCXX} are used 
to define the path where the Sophya libraries and executable are installed.
{\bf DPCBASEREP} defines the base directory path and {\bf EROSCXX} the 
name of the C++ compiler. The complete path is built using {\bf DPCBASEREP},
the operating system name (as obtained by the {\tt uname} command), and
the compiler name. In the example below, we show the complete path 
for a {\tt Linux} system, using the GNU g++ compiler:

\begin{itemize}
\item \$DPCBASEREP/Include : Include (.h) files
\item \$DPCBASEREP/Linux-g++/Libs : Path for the archive libraries (.a)
\item \$DPCBASEREP/Linux-g++/ShLibs : Shared library path (.so)
\item \$DPCBASEREP/Linux-g++/Exec : Executable file path
\end{itemize}

In order to use the shared libraries, the {\bf LD\_LIBRARY\_PATH} variable
should contain the Sophya shared library path 
({\tt \$DPCBASEREP/Linux-g++/ShLibs } when using g++ compiler on Linux)

For modules using external libraries, the {\bf EXTLIBDIR} 
environment variable should contain the path to these libraries
and corresponding include files.
C-FitsIO anf FFTW include files should be accessible through: \\
{\tt \$EXTLIBDIR/Include/FitsIO } \\
{\tt \$EXTLIBDIR/Include/FFTW } \\
The corresponding libraries are expected to be found in: \\
{\tt \$EXTLIBDIR/Linux-g++/Libs} \\

The file {\tt \$DPCBASEREP/Include/MakefileUser.h} defines the compilation 
flags and the list of Sophya libraries. It should be included in the 
user's makefile. The default compilation rules assumes that the object (.o) 
and executable files would be put in the following diretories: \\
{\tt \$HOME/`uname`-\$EROSCXX/Objs} \\ 
{\tt \$HOME/`uname`-\$EROSCXX/Exec}.
In the case of a {\tt Linux} system and using {\tt g++} as the C++ compiler, 
these two directories would be translated to \\
{\tt \$HOME/Linux-g++/Objs} and {\tt \$HOME/Linux-g++/Exec}. 
The GNU make program should be used.
\par
The file {\tt \$DPCBASEREP/Include/makefile\_auto} defines the rules to compile
a given source program, and link it against the Sophya libraries to produce
an executable. The example below shows the steps to compile a program named
{\tt trivial.cc }.
\begin{verbatim}
csh> cp \$DPCBASEREP/Include/makefile_auto makefile
csh> make trivial
\end{verbatim}
This command should compile the {\tt trivial.cc} file, 
and link it against the sophya libraries. The object and executable 
file names are: \\ 
{\tt \$HOME/`uname`-\$EROSCXX/Objs/trivial.o} \\
{\tt \$HOME/`uname`-\$EROSCXX/Exec/trivial}.
\par 
The file {\tt \$DPCBASEREP/Include/makefile\_example} provides another 
example makefile.

Basic usage of Sophya classes are described in in the following sections.
Complete Sophya documentation can be found at our web site: \\
{\bf http://hfi-l2.in2p3.fr}. 


\section{Module SysTools}

{\bf SysTools} contains utility classes such as {\tt DataCards} or 
{\tt DVlist}, an hierarchy of exception classes for Sophya, a template
class {\tcls{NDataBlock}} for handling reference counting on numerical
arrays, as well as classes providing the services for implementing simple
serialization. 
\vspace*{5mm}

\subsection{SOPHYA persistence}
\begin{figure}[hbt]
\dclsa{PPersist}
\dclsbb{PIOPersist}{PInPersist}
\dclsb{POutPersist}
\caption{partial class diagram for classes handling persistence in Sophya}
\end{figure}
A simple persistence mechanism is defined in SOPHYA. Its main 
features are:
\begin{itemize}
\item[] Portable file format, containing the description of the data structures
and object hierarchy. \\
{\bf PPF} {\bf P}ortable {\bf P}ersistence file {\bf F}ormat.
\item[] Handling of read/write for mutiply referenced objects.
\item[] All write operations are carried using sequential access only. This
holds also for read operations, unless positional tags are used. 
SOPHYA persistence services can thus be used to transfer objects
through network links.
\end{itemize}
The example below shows writing of objects through the use of overloaded 
operator $ << $ :
\begin{verbatim}
#include "fiondblock.h"
// ...
POutPersist pos("aa.ppf");
NDataBlock<r_4> rdb(40);
rdb = 567.89;
pos << rdb;
// We can also use the PutObject method
NDataBlock<int_4> idb(20);
idb = 123;
pos.PutObject(idb);
\end{verbatim}
The following sample programs show the reading of the created PPF file : 
\begin{verbatim}
PInPersist pis("aa.ppf");
NDataBlock<r_4> rdb;
pis >> rdb;
cout << rdb;
NDataBlock<int_4> idb;
cout << idb;
\end{verbatim}

\subsection{Using DataCards}
The {\bf DataCards} class can be used to read parameters from a file.
Each line in the file starting with \@ defines a set of values
associated with a keyword. In the example below, we read the 
parameters corresponding with the keyword {\tt SIZE} from the 
file {\tt ex.d}. We suppose that {\tt ex.d} contains the line: \\
{\tt @SIZE 400 250} \\
\begin{verbatim}
#include "datacards.h"
// ...
// Initializing DataCards object dc from file ex.d
DataCards dc( "ex.d" );
// Getting the first and second parameters for keyword size
// We define a default value 100
int size_x = dc.IParam("SIZE", 0, 100);
int size_y = dc.IParam("SIZE", 1, 100);
cout << " size_x= " << size_x << " size_y= " << size_y << endl;
\end{verbatim} 

\subsection{Dynamic linker}
The class {\bf PDynLinkMgr} can be used for managing shared libraries
at run time. The example below shows the run time linking of a function:\\
{\tt extern "C" { void myfunc(); } } \\
\begin{verbatim}
#include "pdlmgr.h"
// ...
string soname = "mylib.so";
string funcname = "myfunc";
PDynLinkMgr dyl(soname);
DlFunction f = dyl.GetFunction(funcname); 
if (f != NULL) { 
// Calling the function 
  f();    
}
\end{verbatim}

\section{Module TArray}
{\bf TArray} module contains template classes for handling standard 
operations on numerical arrays. Using the class {\tt \tcls{TArray} },
it is possible to create and manipulate up to 5-dimension numerical
arrays {\tt (int, float, double, complex, \ldots)}.   

\begin{figure}[hbt]
\dclsccc{AnyDataObj}{BaseArray}{\tcls{TArray}}
\ldots \\
\dclsccc{\tcls{TArray}}{\tcls{TMatrix}}{\tcls{TVector}}
\caption{partial class diagram for arrays, matrices and vectors}
\end{figure}

\subsection{Using arrays}

The example below shows basic usage of arrays: 
\begin{verbatim}
#include "array.h"
// ...
// Creation , filling of a Matrix
  TMatrix<r_4> ma(7,9);
  ma = RegularSequence(0.1, 0.05);   
  cout << "\n ma = " << ma << endl; 
\end{verbatim}

Example of a simple low-pass filter on a one dimensional array (Vector) 
\begin{verbatim}
// Input Vector containing a noisy periodic signal
  Vector in(1024), out(1024);
  in = RandomSequence(RandomSequence::Gaussian, 0., 1.);
  for(int kk=0; kk<in.Size(); kk++)
    in(kk) += 2*sin(kk*0.05);
// Compute the output vector by a simple low pass filter
  Vector out(1024);
  int w = 2;
  for(int k=w; k<in.Size()-w; k++)
    out(k) = in(Range(k-w, k+w).Sum()/(2.*w+1.);
\end{verbatim}

\newpage 

\section{Module HiStats}
\begin{figure}[hbt]
\dclsccc{AnyDataObj}{Histo}{HProf}
\dclsbb{AnyDataObj}{Histo2D}
\dclsbb{AnyDataObj}{Ntuple}
\dclsb{XNtuple}
\caption{partial class diagram for histograms and ntuples}
\end{figure}

{\bf HiStats} contains classes for creating, filling, printing and
doing various operations on one or two dimensional histograms
{\tt Histo} and {\tt Histo2D} as well as profile histograms {\tt HProf}. \\
This module also contains {\tt NTuple} and {\tt XNTuple} which are
more or less the same that the binary FITS tables.

\subsection{1D Histograms}

For 1D histograms, various numerical methods are provided such as
computing means and sigmas, finding maxima, fitting, rebinning,
integrating \dots \\
The example below shows creating and filling a one dimensionnal histogram
of 100 bins from $-5.$ to $+5.$ to create a gaussian normal distribution
with errors~:
\begin{verbatim}
#include "histos.h"
// ...
Histo H(-0.5,0.5,100);
H.Errors();
for(int i=0;i<25000;i++) {
  double x = NorRand();
  H.Add(x);
}
H.Print(80);
\end{verbatim}

\subsection{2D Histograms}

Much of these operations are also valid for 2D histograms. 1D projection
or slices can be set~:
\begin{verbatim}
#include "histos2.h"
// ...
Histo H2(-1.,1.,100,0.,60.,50);
H2.SetProjX(); // create the 1D histo for X projection
H2.SetBandX(25.,35.); // create 1D histo  projection for 25.<y<35.
H2.SetBandX(35.,45.); // create 1D histo  projection for 35.<y<45.
H2.SetBandX(40.,55.); // create 1D histo  projection for 40.<y<55.
//... fill H2 with what ever you want
H2.Print();
Histo *hx = H2.HProjX();
  hx->Print(80);
Histo *hbx2 = HBandX(1); // Get the second X band (35.<y<45.)
  hbx2->Print(80);
\end{verbatim}

\subsection{Profile Histograms}

Profiles histograms contains the mean and the sigma of the distribution
of the values filled in each bin. The sigma can be changed to
the error on the mean. When filled, the profile histogram looks
like a 1D histogram and much of the operations that can be done on 1D histo
may be applied onto profile histograms.

\subsection{Ntuples}

NTuple are memory resident tables of 32 bits floating values (float).
They are arranged in columns. Each line is often called an event.
These objects are frequently used to analyze data.
Graphicals tools (spiapp) can plot a column against an other one
with respect to various selection cuts. \\
Here is an example of creation and filling~:
\begin{verbatim}
#include "ntuple.h"
#include "srandgen.h"
// ...
char* nament[4] = {"i","x","y","ey"};
r_4 xnt[4];
NTuple NT(4,nament);
for(i=0;i<5000;i++) {
  xnt[0] = i+1;
  xnt[1] = 5.*drandpm1();       // a random value between -5 and +5
  xnt[2] = 100.*exp(-0.5*xnt[1]*xnt[1]) + 1.;
  xnt[3] = sqrt(xnt[2]);
  xnt[2] += xnt[3] * NorRand(); // add a random gaussian error
  NT.Fill(xnt);
}
\end{verbatim}

XNtuple are sophisticated NTuple : they accept various types
of column values (float,double,int,...) and can be as big as
needed (they used buffers on hard disk).

\subsection{Writing, seeing \dots }

All these objects have been design to be written to or read from a persistant file.
The following example shows how to write the previously created objects
into such a file~:
\begin{verbatim}
//-- Writing
{
char *fileout = "myfile.ppf";
string tag;
POutPersist outppf(fileout);
tag = "H";   outppf.PutObject(H,tag);
tag = "H2";   outppf.PutObject(H2,tag);
tag = "NT";   outppf.PutObject(NT,tag);
}  // closing ``}'' destroy ``outppf'' and automatically close the file !
\end{verbatim}

Sophya graphical tools (spiapp) can automatically display and operate
all these objects.

\section{Module SkyMap}

\section{Module NTools}

This module provides elementary numerical tools for numerical integration,
fitting, sorting and ODE solving. FFTs are also provided (Mayer,FFTPack).

\subsection{Fitting}

Fitting is done with two classes {\tt GeneralFit} and {\tt GeneralFitData}
and is based on the Levenberg-Marquardt method.
GeneralFitData is a class which provide a description of the data
to be fitted. GeneralFit is the fitter class. Parametrized functions
can be given as classes which inherit {\tt GeneralFunction}
or as simple C functions. Classes of pre-defined functions are provided
(see files fct1dfit.h and fct2dfit.h). The user interface is very close
from that of the CERN {\tt Minuit} fitter.
Number of objects (Histo, HProf \dots ) are interfaced with GeneralFit
and can be easily fitted. \\
Here is a very simple example for fitting the previously created NTuple
with a gaussian~:
\begin{verbatim}
#include "fct1dfit.h"
// ...

// Read from ppf file
NTuple nt;
{
PInPersist pis("myfile.ppf");
string tag = "NT";  pis.GetObject(nt,tag);
}

// Fill GeneralData
GeneralData mGdata(nt.NEntry());
for(int i=0; i<nt.NEntry(); i++)
  mGdata.AddData1(xnt[1],xnt[2],xnt[3]); // Fill x, y and error on y
mGData.PrintStatus();

// Function for fitting : y = f(x) + noise
Gauss1DPol mFunction; // gaussian + constant

// Prepare for fit
GeneralFit mFit(&mFunction);  // create a fitter for the choosen function
mFit.SetData(&mGData);        // connect data to the fitter

// Set and initialize the parameters (that's non-linear fitting!)
//           (num par, name, guess start, step, [limits min and max])
mFit.SetParam(0,"high",90.,1..);
mFit.SetParam(1,"xcenter",0.05,0.01);
mFit.SetParam(2,"sigma",sig,0.05,0.01,10.);
                              // Give limits to avoid division by zero
mFit.SetParam(3,"constant",0.,1.);

// Fit and print result
int rcfit = mFit.Fit();
mFit.PrintFit();
if(rcfit>0) {)
  cout<<"Reduce_Chisquare = "<<mFit.GetChi2Red()
      <<" nstep="<<mFit.GetNStep()<<" rc="<<rcfit<<endl;
} else {
  cout<<"Fit_Error, rc = "<<rcfit<<"  nstep="<<mFit.GetNStep()<<endl;
  mFit.PrintFitErr(rcfit);
}

// Get the result for further use
TVector<r_8> ParResult = mFit.GetParm();
cout<<ParResult;
\end{verbatim}

Much more usefull possibilities and detailed informations might be found
in the HTML pages of the Sophya manual.


\section{Module Samba}

\section{Module SkyT}


\newpage
\section{Building and installing Sophya}
Presently, the Sophya library compilation has been tested with the following
compiler/platform pairs:

\begin{center}
\begin{tabular}{ll}
Compaq/DEC OSF1  &     cxx  (6.0 , 6.2)  \\
Linux            &     g++  (2.91 , 2.95)  \\
Linux            &     KCC  (3.4)   \\
Solaris          &     g++  (2.95)  \\
SGI IRIX64       &     CC           \\
\end{tabular}
\end{center}

Some of the modules in the Sophya package uses external libraries. The 
{\bf FitsIOServer} is the example of such a module, where the {\tt libcfitsio.a}
is used.
The build procedure expects to find the include files and the libraries in: \\
{\tt \$EXTLIBDIR/Include/FitsIO } \\
{\tt \$EXTLIBDIR/`uname`-\$EROSCXX/Libs} \\

The object files from a given Sophya module are grouped in an archive library 
with the module's name ({\tt libmodulename.a}). All Sophya modules 
 are grouped in a single shared library ({\tt libsophya.so}), while the 
modules with reference to external libraries are grouped in 
({\tt libextsophya.so}). The {\bf PI} and {\bf PIext} modules are 
grouped in ({\tt libPI.so}).

The environment variables {\bf DPCDEVREP}, {\bf EXTLIBDIR} and {\bf EROSCXX}
must be defined in order to install the Sophya package.
In the example below, we assume that we want to install Sophya from a 
released (tagged) version in the source directory {\tt \$SRC} in the
{\tt /usr/local/Sophya} diretory, using {\tt g++}. We assume that 
the external libraries directory tree has been set up in 
{\tt /usr/local/ExtLibs/}. \\[3mm]
\centerline{ 
          \rule{20mm}{0.5mm} 
        {\bf \large the use of GNU make is mandatory} 
        \rule{20mm}{0.5mm} }
 
\vspace*{3mm}
\begin{verbatim}
# We select our C++ compiler
csh> setenv EROSCXX g++
# Setup the build directory
csh> mkdir /usr/local/Sophya/
csh> setenv DPCDEVREP /usr/local/Sophya/
csh> setenv EXTLIBDIR /usr/local/ExtLibs/
# Use the top level makefile in Mgr/ 
csh> cd \$SRC
csh> cp Mgr/Makefile Makefile
# Step 1: Create the directory tree and copy the include files (.h)
csh> make depend
# Step 2: Compile the modules without external library reference
csh> make libs
# Step 3: Compile the modules WITH  external library reference (optional) 
csh> make extlibs
# Step 4: Build libsophya.so
csh> make slb
# Step 5: Build libextsophya.so (optional)
csh> make slbext
# Step 6: Compile the PI and PIext modules (optional)
csh> make PI
# Step 7: Build the corresponding shared library libPI.so (optional)
csh> make slbpi
\end{verbatim}

To compile all modules and build the shared libraries, it is possible 
to use:
\begin{verbatim}
# Step 2,3,6
csh> make all
# Step 4,5,7
csh> make slball
\end{verbatim}

At this step, all libraries sould have been made. Programs using
Sophya libraries can now be built:
\begin{verbatim}
# To compile test programs
csh> cd Tests
csh> make arrt ...
csh> cd ..
# To compile other programs, for example from the PMixer module
csh> cd PMixer
csh> make
csh> cd ..
# To build (s)piapp (libPI.so is needed)
csh> cd ProgPI
csh> make 
csh> cd ..
\end{verbatim}

\subsection{Mgr module}
This module contains scripts which can be used for generating the 
makefiles for each module.
\begin{itemize}
\item {\bf Makefile} Top level Makefile for builiding the libraries.
\item {\bf Makefile.h} contains the defintion of compilation flags for the 
different compilers and systems. This file is used for building the 
library and generating {\bf MakefileUser.h} (to be included in makefiles).
\item {\bf Makefile.slb} contains the rules for building shared libraries
for the different compilers and systems. (to be included in makefiles)
\item {\bf crerep\_sophya} c-shell script for creating the directory tree
under {\tt \$DPCBASEREP} and {\tt \$DPCDEVREP}
\item {\bf install\_sophya} c-shell script for installing the Sophya package.
Usually from {\tt \$DPCDEVREP} to {\tt \$DPCBASEREP}
\item {\bf mkmflien} c-shell script for making symbolic links or copying
include files to {\tt \$DPCDEVREP/Include} or {\tt \$DPCBASEREP/Include}
\item {\bf mkmf} c-shell script for generating module makefiles and the 
top level makefile (named GNUmakefile)
\item {\bf mkmflib} c-shell script for generating each library module 
makefile (named GNUmakefile)
\item {\bf mkmfprog} c-shell script for generating makefile for a module
containing the source for executable programs (named GNUmakefile).
\item {\bf mkmfPI} c-shell script for generating makefile for PI and PIext
modules (named GNUmakefile)
\item {\bf libdirs} List of Sophya modules without reference to external
libraries.
\item {\bf extlibdirs} List of Sophya modules with reference to external
libraries.

\end{itemize}
 
\newpage
\appendix
\section{Exception handling: An example}
For simple programs, it is a good practice to handle 
the exceptions at least at high level, in the {\tt main()} function. 
The example below shows the exception handling and the usage 
of Sophya persistence.
 
\input{ex1.inc}


\end{document}