= How to Install Quattor with SCDB =
[[TracNav]]

[[TOC(inline)]]

''Note : the installation process described below is as generic as
possible, but does also contains a set of commands that have been
tested only on Scientific Linux version 4. They may need to be
modified for other platforms.''

== OS Installation ==

Quattor server requires a machine installed with a default server
installation of any RH-based Linux distro. There is no specific
requirements for the OS configuration itself. The server can be
installed by any mean available at the site (CD-Rom, Kickstart,
imaging...). When Quattor server is readyn it will be possible to
manage the server itself with Quattor, except for OS upgrades.

== Web Server Installation ==

The Quattor server needs to run a Web server to serve profiles,
kickstart configuration files and execute the CGI script at end of
installation to change PXE boot to local disk. In addition, it is
recommended (but not necessary) to use this Web server for serving
RPMs.

Web server installation requires nothing specific, just the
configuration of a document root with enough space if you plan to
serve RPMs and the configuration of CGIs. This Web server can be
shared with other usages and you can use a specific virtual host
instead of a dedicated server.

Apache is the recommended Web server (installation instructions here
refer to Apache) and it can be installed from the OS
distribution. Note that for subversion http mode, Apache version 2 or
above is needed.

=== Apache Recommended Settings ===

SCDB has no strong requirement concerning Apache configuration. It
generally uses 3 distinct URLs for 3 different purposes :

 * Profiles : machine profiles are served by one specific URL shared
   by all machines. The files there are XML files produces by the PAN
   compiler when executing `ant deploy`.
 * Kickstart configuration files : this URL is used to store the
   Kickstart configuration file for each machines. These files are
   produced by `aii-shellfe --configure`.
 * RPM packages : RPMs are grouped in repositories, each repository
   has its own URL. This is a common setting to have one common parent
   URL for all repositories but this is not at all a requirement.

Recommended setting for these 3 areas are :

 * Restrict access to profile and Kickstart configuration to IP
   adresses (or subnets) matching Quattor clients, as these files may
   contain sensitive information like encrypted passwords or MySQL
   passwords (cleartext).
 * Configure all these areas to ignore any `index.html` file and
   auto-indexing. This is particularly important for RPM repositories
   URLs, as presence of an `index.html` will prevent SCDB tools to get
   the list of RPMs in the repository.

Configuration for these areas is normally done by creating a file
`/etc/httpd/conf.d/quattor.conf` with directives like the following
one for each area (replace `/path/to/area` by your actual directoy
name) :

{{{
<Directory /path/to/area>
    Options Indexes
    DirectoryIndex VeryUnlikelyDirectoryIndex.none
    AllowOverride None
</Directory>
}}}

It is also better to add the following directive in our
`/etc/httpd/conf.d/quattor.conf` to work around a problem in some RPM
versions:

{{{
<IfModule mod_setenvif.c>
  BrowserMatch "rpm/.*" nokeepalive force-response-1.0
</IfModule>
}}}

''Note: if you are installing a new Apache server, don't forget to
edit `DocumentRoot` in `/etc/httpd/conf/httpd.conf` to reflect your
local configuration.`

''Note: even though it is easily redone, it is better to backup
`quattor.conf` file.''

== Subversion Server ==

There is no need for a Subversion server dedicated to Quattor. SCDB is
just one repository from the Subversion point of view. If you already
run a Subversion server, you can skip the installation part and go
directly to the configuration part.

=== Subversion Installation and Configuration ===

There are many possible installation options for a Subversion
server. The best is to install it as Apache module, anyway. There is
no requirement for the Subversion server to run on a Linux machine,
even if it is the installation option documented here. You can even
choose to use a Subversion server outside of your site, if you think
the network connection is good enough.

If you need to install a Subversion server, the easiest is to install
Apache using YUM. Another option is to retrieve the RPMs for
Subversion from [http://subversion.tigris.org/project_packages.html
Subversion site]. Don't forget to install the Apache module which is
in a separate RPM.

A typical SVN installation with YUM is:
{{{
yum install subversion mod_dav_svn
}}}

After installing, you have to configure the Subversion server. Refer to [http://subversion.tigris.org Subversion web site] for details. Configuration the SVN server typically involves:
 * Creation of directory which will contain the Quattor repository (this example uses `/var/svn`):
{{{
mkdir -p /var/svn
}}}
 * Create Subversion repository that will be used for Quattor SCDB (don't forget to '''backup this directory'''):
{{{
svnadmin create /var/svn/quattor
# Repository must be owned by Apache account
chown -R apache:apache /var/svn/quattor
}}}

Apache SVN module configuration (`/etc/httpd/conf.d/subversion.conf`)
must be edited to configure URL used by SVN. A typical example, based
on previously created repository (adjust paths to reflect your
configuration) is:

{{{
<Location /svn>
   DAV svn
   SVNParentPath /var/svn

   AuthzSVNAccessFile security/svn-repositories-access

   AuthType        Basic
   AuthUserFile    security/passwd
   AuthGroupFile   security/group
   AuthName        "Grid Tutorial SVN server"

   # Limit write permission to list of valid users.
   <LimitExcept GET PROPFIND OPTIONS REPORT>
      # Require SSL connection for password protection.
      # SSLRequireSSL

      Require valid-user
   </LimitExcept>
</Location>
}}}

To configure SVN authentication for SCDB repository, you need to
create one or more accounts in `/etc/httpd/security/passwd`. You can
use `htpasswd` or `openssl passwd -apr1` to generate an encrypted
password.

You also need to define SVN ACLs in
`/etc/httpd/security/svn-repositories-access`. A typical file to start
is (it assumes the account you created is called `quattormgr`, if this
is a list it must be comma separated):

{{{
[groups]
quattor-mgrs = quattormgr

[/]
* = r
@quattor-mgrs = rw
}}}

''Note: even though it is easily redone, it is better to backup `subversion.conf` file and files in `/etc/httpd/security`.''

=== Repository configuration ===

For Quattor, you need to create a repository with the standard structure inside it (or inside a branch) :
 * `trunk` : where you make the changes to your running configuration
 * `tags` : used by SCDB administration tool to do deployment
 * `branches` : for alternative developments

For example:

{{{
mkdir toto
cd toto
mkdir scdb
mkdir scdb/trunk
mkdir scdb/tags
mkdir scdb/branches
svn import scdb https://svn.server.tld/svn/quattor --message 'Initial repository layout'
}}}

Then, you can do the initial checkout with (this will create a `scdb` sub-directory of you current directory):

{{{
svn checkout https://svn.server.tld/svn/quattor/trunk scdb
}}}

== DHCP and TFTPD installation ==

Install DHCPD and TFTPD server from OS distribution. You can do it with the following YUM command:
{{{
yum install dhcp tftp-server
}}}

If the DHCP server is to be used for Quattor usage only, a basic DHCP configuration (`/etc/dhcpd.conf`) may be:
{{{
# DHCP server configuration

authoritative;
allow bootp;
#allow duplicates;
ddns-update-style none;
#omapi-port 7921;       # Use a non standard port (standard = 7911)

# Edit to reflect your DNS domain name and name servers (a comma-separated list is allowed)
option domain-name "lal.in2p3.fr";
option domain-name-servers nfsserv.lal.in2p3.fr;
option netbios-node-type 2;

# Update to reflect your IP subnet
subnet 134.158.72.0 netmask 255.255.255.254 {

  # Parameters for the installation via PXE using pxelinux
  filename                           "quattor/pxelinux.0";
  #option dhcp-class-identifier       "PXEClient";
  option vendor-encapsulated-options 01:04:00:00:00:00:ff;

  # This is now a required line in DHCP configuration.  This
  # option gives the behavior of the previous versions.
  ddns-update-style ad-hoc;

  option routers 134.158.72.1;
}
}}}

If you want to share DHCP between Quattor and non Quattor usage, it's
probably better to move the last part (`subnet...`) into a separate
file, like `/etc/dhcpd/quattor.conf` and replace it in the main
configuration file by:

{{{
include "/etc/dhcpd/quattor.conf";
}}}

See `man dhcpd` and `man dhcpd.conf` for details about DHCP server
configuration, in particular to support multiple subnets and other
advanced features.

TFTP server is run by `xinetd`. In the default configuration, it is
disabled. Enable it by editing `/etc/xinetd.d/tftp`, modifying
`disable` parameter from `yes` to `no`.

Note that default location for TFTP root in AII configuration files is
{{{/osinstall/nbp}}}. It must be explicitly defined if you want to use
{{{/tftpboot}}} or another location.


== Quattor Server ==

In addition to the base system installation, you need to install the
following RPMs on a Quattor server where you want to use SCDB :

 * Java VM > 1.5.0
 * Subversion client (preferably > 1.4)
 * cdb-sync
 * ncm-template
 * aii-server (2.4 or higher)
 * ncm-lib-blockdevices (0.18.5 or 0.20)
 * aii-ks
 * aii-pxelinux

All but Java and SVN client can be download from
http://quattorsrv.lal.in2p3.fr/packages/quattor/sl. Always use the
last version, unless explicitly mentioned. You can also use APT or YUM
from http://quattorsw.web.cern.ch/quattorsw/software/quattor.

== SCDB Initialization ==

To start with SCDB, you first need to install a
[http://subversion.tigris.org Subversion] server, an open source
product. The http based repository access '''must''' be used for
quattor, the standalone access wont work (limitation of the build
script).

After you have a Subversion server installed, you need to :

 * Create a Subversion repository that will be used for SCDB, if it
   doesn't exist yet, and associate this repository with a URL (this
   can involve modifying Apache configuration). There is no need to
   use a dedicated repository. E.g. :
   `http://svn.example.org/Quattor`.
 * Create a branch in this repository where SCDB will be stored, if
   the repository is not dedicated to SCDB. E.g. :
   `http://svn.example.org/Quattor/CDB`.
 * In this branch, create 2 branches `trunk` and `tags` (`tags` is
   managed by SCDB tools, all the actions you'll do later will be done
   in `trunk`. You can also create other branches for your
   conveniences (like `branches` but they are not used by standard
   tools).
 * Choose the QWG templates version that suit your needs and import
   [source:SCDB/tags/pro SCDB base] and QWG templates in directory
   that will become you working area. See [wiki:Download/QWGTemplates
   QWG download] for detailed instructions. The easiest is to download
   and use [source:templates/trunk/tools/check-compile.sh
   check-compile.sh] (use option `-h` to get the list of available
   options). For example, assuming you want to create a `cdb`
   sub-directory of your current directory and download QWG templates
   gLite-3.0.2-10 :

{{{
check-compile.sh -d cdb /templates/tags/gLite-3.0.2-10
}}}
 * Change current directory to the working area, for example :
{{{
cd cdb
}}}
 * Checkout SCDB trunk (empty) in your SCDB working directory :
{{{
svn co http://svn.example.org/Quattor/CDB/trunk .
}}}
 * Configure the repository to ignore some files produced when
   compiling, using the following command :
{{{
cat > /tmp/ignore <<EOF
.settings
build
build.saved
deploy
quattor.build.properties
.project
EOF
svn propset svn:ignore -F /tmp/ignore .
svn ci
}}}
 * Add everything to your repository with command :
{{{
svn add *
}}}
 * Commit your vanilla SDCB with :
{{{
svn ci -m 'Create initial SCDB'
}}}

== Site Configuration ==

After copying the SCDB distribution, you need to create your first
site. You can do this by copying `sites/example` directory and
customizing a few templates.

=== RPM Repositories ===

To use Quattor, you need to deploy software repositories. Even if you
want to customize it later, you are probably better to start with a
configuration similar to what is provided in `repository` directory of
`sites/example` directory. You can retrieve an initial directory
content for each RPM repository by downloading the contents of the URL
specified in comments at the beginning of each repository templates.

=== Basic System Configuration ===

Basic system configuration (network parameters, DNS servers, ...) are
grouped in template `pro_site_cluster_info.tpl` in `site` directory of
your site. Look at comments to understand what you need to modify.

=== Middleware Configuration ===

Middleware configuration is located in template
`pro_lcg2_config_site.tpl` in `site` directory of your site. Look at
comments to understand what you need to modify.


== Cluster Configuration ==

After creating your site, you need to create your first cluster. You
can do this by copying `clusters/example` directory and customizing a
few templates.

=== Hardware description ===

You need to create a template describing the hardware configuration of
your machines. This is generally placed in `hardware`sub-directory of
site directory. Look at examples.

=== Adding Machine to pro_site_database.tpl ===

Before being able to configure the machine, you need to create an
entry for the machine name in both tables of
`pro_site_database.tpl`. First entry defines the address associated
with the machine name, second entry defines the hardware template
associated with the machne.

=== Creating Machine Profile ===

Copy an existing profile in examples corresponding to the machine type
you want to create.

== Quattor Server Final configuration ==

Before being able to deploy the created configuration, there is a last
configuration step needed to allow deployment of the configuration
after successful compilation. This involves :

 * Adding a hook script to the Subversion repository to trigger the
   deployment
 * Adding a script on the Quattor server that will be launched by the
   hook script, using ssh
 * Configuring SSH keys to allow execution of the previous script as
   root (preferably) from the Apache account
 * Add a CGI script on Quattor server used at end of installation of a
   machine to allow next boot from local disk.
 * Configuration of AII

=== Installation of hook script and server script ===

The hook script, [source:SCDB/tags/pro/src/hooks/post-commit post-commit], is provided as part of SCDB, in the `src/hooks` directory. It must be installed on your Subversion server, in the `hook` directory of the repository, and given executable permission for Apache user. This script requires a configuration file `/etc/quattor-deploy.conf`, see SCDB [wiki:Doc/SCDB/Server server-side customizations] for details. 

The other script, [source:SCDB/tags/pro/src/hooks/build-tag.pl build-tag.pl], also provided as part of SCDB, in the `src/hooks` directory, must be installed (root executable) in `/root/quattor/scripts` on the Quattor server. It also requires a configuration file, `/etc/build-tag.conf`. See SCDB [wiki:Doc/SCDB/Server server-side customizations] for details.

`build-tag.pl` requires file `quattor.build.properties` to be created in `/root/quattor`. A [source:SCDB/tags/pro/src/hooks/quattor.build.properties template] of this file is available in SCDB distribution, in `src/hooks` directory. It must be edited to reflect your local configuration.

''Note: you can download the last version of these scripts from QWG repository with the following command:''
{{{
wget --no-check-certificate "https://svn.lal.in2p3.fr/LCG/QWG/SCDB/trunk/src/hooks/post-commit" -O /var/svn/quattor/hooks/post-commit
chmod 755 /var/svn/quattor/hooks/post-commit
wget --no-check-certificate "https://svn.lal.in2p3.fr/LCG/QWG/SCDB/trunk/src/hooks/build-tag.pl" -O /root/quattor/scripts/build-tag.pl
chmod 755 /root/quattor/scripts/build-tag.pl
}}}

For more details about these scripts and their customization, see the page on SCDB [wiki:Doc/SCDB/Server server-side customizations].

=== Creation of SSH Keys ===

Currently, deployment of new version of the templates is done by the hook script triggered by `ant deploy` executing the server script `build-tag.pl` through ssh. There is no way to enter a password at this time, thus ssh must be configured in such a way that the Apache account on the Subversion server can do a ssh connection as root on the Quattor server, without password. The easiest is to use ssh keys to do that.

''Note: if you run SVN server on the Quattor server, an alternative to SSH is to use `sudo`. This currently requires to use a specific variant of the `post-commit` script, `post-commit.sudo`. After installing the script, instead of configuring SSH keys, you need to add the following configuration lines (customize them to reflect your local configuration) to `sudo` with `visudo` utility:''
{{{
Cmnd_Alias   QUATTORDEPLOY=/root/quattor/scripts/build-tag.pl *
Cmnd_Alias   AIIACKCGI=/usr/sbin/aii-shellfe

apache ALL = NOPASSWD: QUATTORDEPLOY
apache ALL = NOPASSWD: AIIACKCGI
}}}

=== Post-installation CGI Script ===

At the end of a machine installation, as part of the Kickstart
post-intallation script, a CGI script is executed on the Quattor
server to change PXE configuration in order for the machine to boot
from local disk next time. This allows to set PXE as the first boot
device in the BIOS and control re-installation via `aii-shellfe`
command.

This script, `aii-installack.cgi`, can be found in SCDB directory
`src/cgis`. It must be placed on the Web server running on the Quattor
server, in the directory for CGIs.

The apache server must be able to run that script as root. Best is to
have {{{sudo}}} installed and use {{{visudo}}} to add the following to
{{{/etc/sudoers}}}:
 
{{{apache sbgat419.in2p3.fr=(ALL) NOPASSWD: /usr/sbin/aii-shellfe}}}


=== Configuration of AII ===

This involves 2 separate steps :
 * Customization of `/etc/aii/aii*.conf` files
 * Customization of AII related variables in templates

To customize AII configuration files, located in `/etc` and named
`aii-*.conf`, refer to the comment in each files. Main parameters to
customize are the URL to use to download profiles (in
`aii-shellfe.conf`) and the directory where to place kickstart
configuration files produced by AII (in `aii-osinstall.conf`).

There are a few variables to customize in site templates to reflect
your Quattor and AII configuration, mainly :

 * `QUATTOR_PROFILE_URL` : URL to use to download machine profiles.
 * `AII_OSINSTALL_SRV` : Name of the Web server serving kickstart
 * configuration files and RPMs.
 * `AII_ACKSRV` : Name of the Web server to use for the
   post-installation CGI. Defaults to `AII_OSINSTALL_SRV`
 * `AII_ACKCGI` : post-installation CGI URL. Defaults to
   `/cgi-bin/aii-installack.cgi`.
 * `AII_OSINSTALL_TEMPLATE` : name of the Kickstart configuration
   template to use. Defaults to `i386_sl3_ks.conf`.

These variables are generally defined site-wide, in the template
`pro_site_global_variables.tpl` located in site directory. Look at
provided examples, in SCDB distribution.

=== Downloading the distribution's images ===

If you want to perform network-based installations, you need to
download the distribution's CDs. They contain the kernel and initrd to
be used during the installation, which will re-direct to Red Hat's
installer. This installer is also located on the CD (usually the first
CD) or DVD of your distribution.

The easiest way is to download the full DVD of your distro, f.i, SL:

{{{
wget http://.../distro-version.iso
}}}

Then, mount it somewhere Apache can read to. For instance,
`/var/www/html/your_platform`:

{{{
mount -o bind /path/to/dvd/image /var/www/html/sl520-x86_64
}}}

Add it to your fstab, if needed. Next, you'll need to copy the files
used for PXE somewhere the TFTP server can reach them. Their location
depends on the distribution, it's usually on a directory called
pxeboot:

{{{
mkdir /osinstall/nbp/<platform>
cp  /var/www/html/<platform>/.../pxeboot/* /osinstall/nbp/<platform>
}}}

== Compiling and Deploying ==

After the configuration is finished, you can try to compile your first
profile, deploy it and install the machine. This involves the
following steps :

 * In SCDB (working area copy) top level directory :
   * Update of RPM repository templates :
{{{
external/ant/bin/ant update.rep.templates
}}}
   * Profile compilation and deployment (deployment will not occur until compilation succeds) :
{{{
external/ant/bin/ant deploy
}}} 
 * On the Quattor server :
   * Creation of Kickstart configuration file for the machine :
{{{
aii-shellfe --configure your.machine.domain
}}}
   * Update of DHCP and PXE for the machine to be installed at next boot :
{{{
aii-shellfe --install your.machine.domain
}}}


== Troubleshooting Initial Installation ==

=== Deployment doesn't work ===

Look at SCDB [https://trac.lal.in2p3.fr/LCGQWG/wiki/Doc/SCDB/Server server-side customizations] page.