= How to Install Quattor with SCDB =
[[TracNav]]

[[TOC(inline)]]

''Note : the installation process described below is as generic as
possible, but does also contains a set of commands that have been
tested only on Scientific Linux version 4. They may need to be
modified for other platforms.''

''Note: you may want to refer to another step-by-step [http://sourceforge.net/apps/mediawiki/quattor/index.php?title=StepByStepQuattorInstall installation guide] containing more details for the initial steps. When in doubt, use this page as the authoritative documentation.''


== OS Installation ==

Quattor server requires a machine installed with a default server
installation of any RH-based Linux distro. There is no specific
requirements for the OS configuration itself. The server can be
installed by any mean available at the site (CD-Rom, Kickstart,
imaging...). When Quattor server is readyn it will be possible to
manage the server itself with Quattor, except for OS upgrades.

== Web Server Installation ==

The Quattor server needs to run a Web server to serve profiles,
kickstart configuration files and execute the CGI script at end of
installation to change PXE boot to local disk. In addition, it is
recommended (but not necessary) to use this Web server for serving
RPMs.

Web server installation requires nothing specific, just the
configuration of a document root with enough space if you plan to
serve RPMs and the configuration of CGIs. This Web server can be
shared with other usages and you can use a specific virtual host
instead of a dedicated server.

Apache is the recommended Web server (installation instructions here
refer to Apache) and it can be installed from the OS
distribution. Note that for subversion http mode, Apache version 2 or
above is needed.

=== Apache Recommended Settings ===

SCDB has no strong requirement concerning Apache configuration. It
generally uses 3 distinct URLs for 3 different purposes :

 * Profiles : machine profiles are served by one specific URL shared
   by all machines. The files there are XML files produces by the PAN
   compiler when executing `ant deploy`.
 * Kickstart configuration files : this URL is used to store the
   Kickstart configuration file for each machines. These files are
   produced by `aii-shellfe --configure`.
 * RPM packages : RPMs are grouped in repositories, each repository
   has its own URL. This is a common setting to have one common parent
   URL for all repositories but this is not at all a requirement.

Recommended setting for these 3 areas are :

 * Restrict access to profile and Kickstart configuration to IP
   adresses (or subnets) matching Quattor clients, as these files may
   contain sensitive information like encrypted passwords or MySQL
   passwords (cleartext).
 * Configure all these areas to ignore any `index.html` file and
   auto-indexing. This is particularly important for RPM repositories
   URLs, as presence of an `index.html` will prevent SCDB tools to get
   the list of RPMs in the repository.

Configuration for these areas is normally done by creating a file
`/etc/httpd/conf.d/quattor.conf` with directives like the following
one for each area (replace `/path/to/area` by your actual directoy
name) :

{{{
<Directory /path/to/area>
    Options Indexes
    DirectoryIndex VeryUnlikelyDirectoryIndex.none
    AllowOverride None
</Directory>
}}}

It is also better to add the following directive in our
`/etc/httpd/conf.d/quattor.conf` to work around a problem in some RPM
versions:

{{{
<IfModule mod_setenvif.c>
  BrowserMatch "rpm/.*" nokeepalive force-response-1.0
</IfModule>
}}}

''Note: if you are installing a new Apache server, don't forget to
edit `DocumentRoot` in `/etc/httpd/conf/httpd.conf` to reflect your
local configuration.`

''Note: even though it is easily redone, it is better to backup
`quattor.conf` file.''

== Subversion Server ==

There is no need for a Subversion server dedicated to Quattor. SCDB is
just one repository from the Subversion point of view. If you already
run a Subversion server, you can skip the installation part and go
directly to the configuration part.

=== Subversion Installation and Configuration ===

There are many possible installation options for a Subversion
server. The best is to install it as Apache module, anyway. There is
no requirement for the Subversion server to run on a Linux machine,
even if it is the installation option documented here. You can even
choose to use a Subversion server outside of your site, if you think
the network connection is good enough.

If you need to install a Subversion server, the easiest is to install
Apache using YUM. Another option is to retrieve the RPMs for
Subversion from [http://subversion.tigris.org/project_packages.html
Subversion site]. Don't forget to install the Apache module which is
in a separate RPM.

A typical SVN installation with YUM is:
{{{
yum install subversion mod_dav_svn
}}}

After installing, you have to configure the Subversion server. Refer to [http://subversion.tigris.org Subversion web site] for details. Configuration the SVN server typically involves:
 * Creation of directory which will contain the Quattor repository (this example uses `/var/svn`):
{{{
mkdir -p /var/svn
}}}
 * Create Subversion repository that will be used for Quattor SCDB (don't forget to '''backup this directory'''):
{{{
svnadmin create /var/svn/quattor
# Repository must be owned by Apache account
chown -R apache:apache /var/svn/quattor
}}}

Apache SVN module configuration (`/etc/httpd/conf.d/subversion.conf`)
must be edited to configure URL used by SVN. A typical example, based
on previously created repository (adjust paths to reflect your
configuration) is:

{{{
<Location /svn>
   DAV svn
   SVNParentPath /var/svn

   AuthzSVNAccessFile security/svn-repositories-access

   AuthType        Basic
   AuthUserFile    security/passwd
   AuthGroupFile   security/group
   AuthName        "Grid Tutorial SVN server"

   # Limit write permission to list of valid users.
   <LimitExcept GET PROPFIND OPTIONS REPORT>
      # Require SSL connection for password protection.
      # SSLRequireSSL

      Require valid-user
   </LimitExcept>
</Location>
}}}

To configure SVN authentication for SCDB repository, you need to
create one or more accounts in `/etc/httpd/security/passwd`. You can
use `htpasswd` or `openssl passwd -apr1` to generate an encrypted
password.

You also need to define SVN ACLs in
`/etc/httpd/security/svn-repositories-access`. A typical file to start
is (it assumes the account you created is called `quattormgr`, if this
is a list it must be comma separated):

{{{
[groups]
quattor-mgrs = quattormgr

[/]
* = r
@quattor-mgrs = rw
}}}

''Note: even though it is easily redone, it is better to backup `subversion.conf` file and files in `/etc/httpd/security`.''

=== Repository configuration ===

For Quattor, you need to create a repository with the standard structure inside it (or inside a branch) :
 * `trunk` : where you make the changes to your running configuration
 * `tags` : used by SCDB administration tool to do deployment
 * `branches` : for alternative developments

For example:

{{{
mkdir toto
cd toto
mkdir scdb
mkdir scdb/trunk
mkdir scdb/tags
mkdir scdb/branches
svn import scdb https://svn.server.tld/svn/quattor --message 'Initial repository layout'
}}}

Then, you can do the initial checkout with (this will create a `scdb` sub-directory of you current directory):

{{{
svn checkout https://svn.server.tld/svn/quattor/trunk scdb
}}}

== DHCP and TFTPD installation ==

Install DHCPD and TFTPD server from OS distribution. You can do it with the following YUM command:
{{{
yum install dhcp tftp-server
}}}

If the DHCP server is to be used for Quattor usage only, a basic DHCP configuration (`/etc/dhcpd.conf`) may be:
{{{
# DHCP server configuration

authoritative;
allow bootp;
#allow duplicates;
ddns-update-style none;
#omapi-port 7921;       # Use a non standard port (standard = 7911)

# Edit to reflect your DNS domain name and name servers (a comma-separated list is allowed)
option domain-name "lal.in2p3.fr";
option domain-name-servers nfsserv.lal.in2p3.fr;
option netbios-node-type 2;

# Update to reflect your IP subnet
subnet 134.158.72.0 netmask 255.255.255.254 {

  # Parameters for the installation via PXE using pxelinux
  filename                           "quattor/pxelinux.0";
  #option dhcp-class-identifier       "PXEClient";
  option vendor-encapsulated-options 01:04:00:00:00:00:ff;

  # This is now a required line in DHCP configuration.  This
  # option gives the behavior of the previous versions.
  ddns-update-style ad-hoc;

  option routers 134.158.72.1;
}
}}}

If you want to share DHCP between Quattor and non Quattor usage, it's
probably better to move the last part (`subnet...`) into a separate
file, like `/etc/dhcpd/quattor.conf` and replace it in the main
configuration file by:

{{{
include "/etc/dhcpd/quattor.conf";
}}}

See `man dhcpd` and `man dhcpd.conf` for details about DHCP server
configuration, in particular to support multiple subnets and other
advanced features.

TFTP server is run by `xinetd`. In the default configuration, it is
disabled. Enable it by editing `/etc/xinetd.d/tftp`, modifying
`disable` parameter from `yes` to `no`.

Note that default location for TFTP root in AII configuration files is
{{{/osinstall/nbp}}}. It must be explicitly defined if you want to use
{{{/tftpboot}}} or another location.


== Quattor Server ==

In addition to the base system installation, you need to install the
following RPMs on a Quattor server where you want to use SCDB :

 * Java VM > 1.5.0
 * Subversion client (preferably > 1.4)
 * cdb-sync
 * ncm-template
 * aii-server (2.4 or higher)
 * ncm-lib-blockdevices (0.18.5 or 0.20)
 * aii-ks
 * aii-pxelinux
 * ncm-ncd
 * ncm-ccm

All but Java and SVN client can be download from
http://quattorsrv.lal.in2p3.fr/packages/quattor/sl. Always use the
last version, unless explicitly mentioned. You can also use APT or YUM
from http://quattorsw.web.cern.ch/quattorsw/software/quattor.

== SCDB Initialization ==

To start with SCDB, you first need to install a
[http://subversion.tigris.org Subversion] server, an open source
product. The http based repository access '''must''' be used for
quattor, the standalone access won't work (limitation of the build
script).

After you have a Subversion server installed, you need to :

 * Create a Subversion repository that will be used for SCDB (if it
   doesn't already exist) and associate this repository with a URL (this
   can involve modifying Apache configuration). There is no need to
   use a dedicated repository. For example:
   `http://svn.example.org/Quattor`.
 * If the repository is not dedicated to SCDB, create a branch in this repository where SCDB will be stored,
    For example:
   `http://svn.example.org/Quattor/CDB`.
 * In this branch, create 2 branches `trunk` and `tags` (`tags` is
   managed by SCDB tools, all the actions you'll do later will be done
   in `trunk`. You can also create other branches for your
   conveniences (like `branches` but they are not used by standard
   tools).
 * Populate your repository with the version of SCDB and QWG templates
   you want to use.  The easiest is to download
   and use [source:templates/trunk/tools/check-compile.sh?format=txt check-compile.sh]
   (use option `-h` to get the list of available
   options).  This will download SCDB, QWG, and compile the example to 
   ensure that everything works.  To populate the `cdb` subdirectory for example,
   with SCDB and last version from gLite 3.1 branch do:
{{{
check-compile.sh -d cdb glite-3.1
}}}
   The equivalent can be done manually.  See the following link for SCDB and QWG: 
   [source:SCDB/tags/pro SCDB base] and [wiki:Download/QWGTemplates QWG download]. 
 * Change current directory to the working area, for example :
{{{
cd cdb
}}}
 * Checkout SCDB trunk (empty) in your SCDB working directory :
{{{
svn co http://svn.example.org/Quattor/CDB/trunk .
}}}
 * Configure the repository to ignore some files produced when
   compiling, using the following command :
{{{
cat > /tmp/ignore <<EOF
.settings
build
build.saved
deploy
.project
EOF
svn propset svn:ignore -F /tmp/ignore .
svn ci
}}}
 * Add everything to your repository with command :
{{{
svn add *
}}}
 * Commit your vanilla SDCB with :
{{{
svn ci -m 'Create initial SCDB'
}}}

== Site Configuration ==

After copying the SCDB distribution, you need to create your first
site. You can do this by copying `sites/example` directory and
customizing a few templates.

=== RPM Repositories ===

To use Quattor, you need to deploy software repositories. Even if you
want to customize it later, you are probably better to start with a
configuration similar to what is provided in `repository` directory of
`sites/example` directory. 

You can retrieve an initial directory content for each RPM repository by downloading the contents of the URL
specified in comments at the beginning of each repository templates. A convenient way to do it is:
{{{
src/utils/misc/rpmUpdates /your/dir/ectory URL_found_in_template_header > /dev/null
}}}

''Note: this command produces a rather verbose output, just ignore it.''

You have to do it for the following repository templates:
 * OS: to start, you need only to download the OS version you plan to use. ''Note: there are several repositories per OS versions, be sure to download all of them.''
 * gLite: to start download all the repositories related to the version you are planning to use, ignore others.
 * All the repositories not related to an OS or gLite version

You are probably better to remove any template you think you will not use (you can always revert your change in case of mistakes).

''Note on Apache configuration for RPM repositories: it is recommended to disable the use of index files on these directories iwth the following options in Apache configuration:''
{{{
<Directory "/www/htdocs/packages">
    Options Indexes
    DirectoryIndex VeryUnlikelyDirectoryIndex.none
    AllowOverride None
</Directory>
}}}

=== Basic System Configuration ===

Basic system configuration (network parameters, DNS servers, ...) are
grouped in template `pro_site_config.tpl` and `pro_site_global_variables.tpl`  in `site` directory of
your site. Look at comments to understand what you need to modify.

=== Middleware Configuration ===

Middleware configuration is located in template
`config.tpl` in `site/glite` directory of your site. Look at
comments to understand what you need to modify.

''Note: do not be afraid of putting wrong values in your gLite parameters, this can easily be refined later. As a general rule, keep the example values when you don't understand how to change them.''


== Cluster Configuration ==

After creating your site, you need to create your first cluster. You
can do this by copying `clusters/example-3.1` directory, removing all profiles from `clusters/example-3.1/profiles` and customizing a
few templates.

=== Hardware description ===

You need to create a template describing the hardware configuration of
your machines. This is generally placed in `hardware` sub-directory of
site directory. Look at examples, copy one with a configuration close to yours as a starting point. The name of the hardware template is at your convenience and will be associated with a host name later.

''Note: recommendation is to have name of templates describing HW boxes derived from box location rather than its current host name as the same HW may be affected to another host later.''

=== Adding Machine to pro_site_database.tpl ===

Before being able to configure the machine, you need to create an
entry for the machine name in both variables of
`pro_site_database.tpl`. First variable defines the address associated
with the machine name, second variable defines the hardware template
associated with the machine.

In both variable, the key is the escaped host name. The value is an IP address in the first case, the name of the template (relative to your site name in `sites` directory) you created in the previous step for the second variable.

=== Creating Machine Profile ===

Copy an existing profile in examples corresponding to the machine type
you want to create.

''Note: as a first node, it is recommended to configure a site BDII as this is the most simple node. When successful with site BDII, the next ones are generally CE and one WN.''

== Quattor Server Final configuration ==

Before being able to deploy the created configuration, there is a last
configuration step needed to allow deployment of the configuration
after successful compilation. This involves :

 * Adding a hook script to the Subversion repository to trigger the
   deployment
 * Adding a script on the Quattor server that will be launched by the
   hook script, using ssh
 * Configuring SSH keys to allow execution of the previous script as
   root (preferably) from the Apache account
 * Add a CGI script on Quattor server used at end of installation of a
   machine to allow next boot from local disk.
 * Configuration of AII

=== Installation of hook script and server scripts ===

The hook script, [source:SCDB/tags/pro/src/hooks/post-commit post-commit], is provided as part of SCDB, in the `src/hooks` directory. It must be installed on your Subversion server, in the `hook` directory of the repository, and given executable permission for Apache user. This script requires a configuration file `/etc/quattor-deploy.conf`, see SCDB [wiki:Doc/SCDB/Server server-side customizations] for details. 

The other script, [source:SCDB/tags/pro/src/hooks/build-tag.pl build-tag.pl], also provided as part of SCDB, in the `src/hooks` directory, must be installed (root executable) in `/root/quattor/scripts` on the Quattor server. It also requires a configuration file, `/etc/build-tag.conf`. See SCDB [wiki:Doc/SCDB/Server server-side customizations] for details.

`build-tag.pl` requires file `quattor.build.properties` to be created in the parent of the directory specified in `svn_cache` paramater of [source:SCDB/tags/pro/src/hooks/build-tag.pl build-tag.pl] configuration file or its default value (`/root/quattor` for example provided here). A [source:SCDB/tags/pro/src/hooks/quattor.build.properties template] of this file is available in SCDB distribution, in `src/hooks` directory. It must be edited to reflect your local configuration.

''Note: you can download the last version of these scripts from QWG repository with the following command:''
{{{
wget --no-check-certificate "https://svn.lal.in2p3.fr/LCG/QWG/SCDB/trunk/src/hooks/post-commit" -O /var/svn/quattor/hooks/post-commit
chmod 755 /var/svn/quattor/hooks/post-commit
wget --no-check-certificate "https://svn.lal.in2p3.fr/LCG/QWG/SCDB/trunk/src/hooks/build-tag.pl" -O /root/quattor/scripts/build-tag.pl
chmod 755 /root/quattor/scripts/build-tag.pl
}}}

For more details about these scripts and their customization, see the page on SCDB [wiki:Doc/SCDB/Server server-side customizations].

=== Creation of SSH Keys ===

Currently, deployment of new version of the templates is done by the hook script triggered by `ant deploy` executing the server script `build-tag.pl` through ssh. There is no way to enter a password at this time, thus ssh must be configured in such a way that the Apache account on the Subversion server can do a ssh connection as root on the Quattor server, without password. The easiest is to use ssh keys to do that.

''Note: if you run SVN server on the Quattor server, an alternative to SSH is to use `sudo`. This currently requires to use a specific variant of the `post-commit` script, [source:SCDB/tags/pro/src/hooks/post-commit.sudo post-commit.sudo]. After installing the script, instead of configuring SSH keys, you need to add the following configuration lines (customize them to reflect your local configuration) to `sudo` with `visudo` utility:''
{{{
Cmnd_Alias   QUATTORDEPLOY=/root/quattor/scripts/build-tag.pl *
apache ALL = NOPASSWD: QUATTORDEPLOY
Defaults:apache !requiretty
}}}
''The last line is required on later versions of RHEL-like operating systems.  If it is not given, the sudo will silently fail.''

=== Post-installation CGI Script ===

At the end of a machine installation, as part of the Kickstart
post-intallation script, a CGI script is executed on the Quattor
server to change PXE configuration in order for the machine to boot
from local disk next time. This allows to set PXE as the first boot
device in the BIOS and control re-installation via `aii-shellfe`
command.

This script, `aii-installack.cgi`, can be found in SCDB directory
`src/cgis`. It must be placed on the Web server running on the Quattor
server, in the directory for CGIs.

The apache server must be able to run that script as root. Best is to
have {{{sudo}}} installed and use {{{visudo}}} to add the following to
{{{/etc/sudoers}}}:
 
{{{
Cmnd_Alias   AIIACKCGI=/usr/sbin/aii-shellfe
apache ALL = NOPASSWD: AIIACKCGI
}}}


=== Configuration of AII ===

This involves 2 separate steps :
 * Customization of `/etc/aii/aii*.conf` files
 * Customization of AII-related variables in templates

There are 2 AII configuration files that need to be customized to reflect your site configuration:
 * `/etc/aii/aii-shellfe.conf`: a typical file is as follows, with paths edited to reflect your configuration.
{{{
# URL corresponding to compiled profiles generated by ant deploy
cdburl=http://quattor.web.lal.in2p3.fr/profiles
# use_fqdn must be set to true
use_fqdn=true
# Directory where to write Kickstart configuration files produced by aii-shellfe.
# Must match the directory served by URL defined in template variable QUATTOR_PROFILE_URL.
osinstalldir = /www/htdocs/ks
# Directory where pxelinux.cfg is installed. Default is the recommended locaiton.
nbpdir = /tftpboot/quattor/pxelinux.cfg
}}}
 * `/etc/aii/aii-dhcp.conf`: check that `dhcpconf` and `restartcmd` command match your configuration and edit as necessary. The file referred by `dhcpconf` must be writable from the AII server. `restartcmd` may launch a command on a remote machine through appropriate means.

To customize AII configuration files, located in `/etc` and named
`aii-*.conf`, refer to the comment in each files. Main parameters to
customize are in
`aii-shellfe.conf`:
 * `cdburl`: URL to use to download profiles
 * `osinstalldir`: directory where to place kickstart configuration files produced by AII

''Note: if `/etc/aii` doesn't exist or is empty on your server, copy templates of these files located in `/usr/share/doc/aii-x.y.z/eg` (with `x.y.z` matching AII version installed).''

There are a few variables to customize in site templates to reflect
your Quattor and AII configuration, mainly :

 * `QUATTOR_PROFILE_URL` : URL to use to download machine profiles.
 * `AII_OSINSTALL_SRV` : Name of the Web server serving kickstart
 * configuration files and RPMs.
 * `AII_ACKSRV` : Name of the Web server to use for the
   post-installation CGI. Defaults to `AII_OSINSTALL_SRV`
 * `AII_ACKCGI` : post-installation CGI URL. Defaults to
   `/cgi-bin/aii-installack.cgi`.

These variables are generally defined site-wide, in the template
`pro_site_global_variables.tpl` located in site directory. Look at
provided examples, in SCDB distribution.

=== Downloading the distribution's images ===

If you want to perform network-based installations, you need to
download the distribution's CDs. They contain the kernel and initrd to
be used during the installation, which will re-direct to Red Hat's
installer. This installer is also located on the CD (usually the first
CD) or DVD of your distribution.

The easiest way is to download the full DVD of your distro, f.i, SL:

{{{
wget http://.../distro-version.iso
}}}

Then, mount it somewhere Apache can read to. For instance,
`/var/www/html/your_platform`:

{{{
mount -o bind /path/to/dvd/image /var/www/html/sl520-x86_64
}}}

Add it to your fstab, if needed. Next, you'll need to copy the files
used for PXE somewhere the TFTP server can reach them. Their location
depends on the distribution, it's usually on a directory called
pxeboot:

{{{
mkdir /osinstall/nbp/<platform>
cp  /var/www/html/<platform>/.../pxeboot/* /osinstall/nbp/<platform>
}}}

== Compiling and Deploying ==

After the configuration is finished, you can try to compile your first
profile, deploy it and install the machine. This involves the
following steps :

 * In SCDB (working area copy) top level directory :
   * Update of RPM repository templates :
{{{
external/ant/bin/ant update.rep.templates
}}}
   * Profile compilation and deployment (deployment will not occur until compilation succeds) :
{{{
external/ant/bin/ant deploy
}}} 
 * On the Quattor server :
   * Creation of Kickstart configuration file for the machine :
{{{
aii-shellfe --configure your.machine.domain
}}}
   * Update of DHCP and PXE for the machine to be installed at next boot :
{{{
aii-shellfe --install your.machine.domain
}}}

''Note: you can ignore the warning returned by `aii-shellfe` about `base_url is not defined in configuration`.''

== Troubleshooting Initial Installation ==

=== Deployment doesn't work ===

Look at SCDB [https://trac.lal.in2p3.fr/LCGQWG/wiki/Doc/SCDB/Server server-side customizations] page.