Release Notes for SCDB

Table of Contents

1. Releases
2. Known Problems
   1. 3.0.0 requires svnkit >= 1.2
   2. 2.3.0/2.3.1: update.vo.config broken
3. New and Changed Features
   1. 3.0.0: group.name option added to ant
   2. 3.0.0: new deployment scripts
   3. 3.0.0: svnkit updated to 1.3 (SVN 1.6)
   4. 2.3.2: update.vo.config fixed
   5. 2.3.1: ability to generate a list of all configured package …
   6. 2.3.1: ant default configuration can be customized with …
   7. 2.3.1: fix for profiles-info.xml not properly updated
   8. 2.3.1: cluster groups support changes
   9. 2.3.0: support for hierarchical arrangement of clusters
   10. 2.3.0: panc upgraded to 8.2.8
   11. 2.3.0: src/utils renamed utils/
   12. 2.2.0: removal of obsolete tools ==)
   13. 2.2.0: sudo supported as replacement for ssh
   14. 2.2.0: all external tools updated to last version
   15. 2.2.0: new ant target to generate VO parameters
   16. 2.2.0: panc upgrade to v8,
   17. 2.1.5 : Script to generate OS errata added
   18. 2.1.5 : OS template generation tools fixed for RHEL and CentOS
   19. 2.1.5 : build-tag.pl detects is another instance is running
   20. 2.1.5 : PAN compiler upgraded to 7.2.9
   21. 2.1.4 : New script misc/buildOSTemplates
   22. 2.1.4 : New script misc/updateComponent
   23. 2.1.4 : New script misc/createPackagesTemplate
   24. 2.1.3 : Several message changes
   25. 2.1.3 : Compiling only 1 cluster
   26. 2.1.2 : Support for repository/ namespace
   27. 2.1.2 : New tag format requires a hook script upgrade
   28. 2.1.0 : Subversion 1.4 requires a new httpd version (SL3)
   29. 2.1.0 : SVNKit upgrade
   30. 2.0.0 : default compiler changed to panc v7
   31. 2.0.0 : configuration file for build-tag.pl
4. Known Problems
   1. 2.1.2 : update.rep.templates possible issue with empty repositories
   2. 2.0.0 : issue with accentuated characters
5. Change Log

SCDB is a set of related tools to administer Quattor templates. This page describes new features introduced, known problems and displays last entries from Subversion log.

Releases

The production version of SCDB Tools is can be downloaded from ​https://svn.lal.in2p3.fr/LCG/QWG/SCDB/tags/pro.

Date	Version	Description
27/03/07	2.0.0	panc v7 as default compiler
20/04/07	2.1.0	SVNKit upgrade (SVN 1.4 compatible)
04/05/07	2.1.1	panc v7.2.0
21/05/07	2.1.2	New tag format, support for repository/ namespace
19/06/07	2.1.3	panc 7.2.3, support for compilation of one cluster, bug fixes
07/11/07	2.1.4	panc 7.2.4, fixed OS templates generation tools
12/02/08	2.1.5	panc 7.2.9, fixed OS templates generation tools
11/11/08	2.2.0	panc 8.2.3, checkdeps.py, sudo-based hook script
12/07/09	2.3.0	panc 8.2.8, quatview, hiearchical arrangement of clusters
23/08/09	2.3.1	Fixes for problems in 2.3.0, rewrite of cluster group support, improvements to ant configuration
14/9/09	2.3.2	Fixed update.vo.config (completely broken in previous 2.3.x and buggy in 2.2)

Known Problems

3.0.0 requires svnkit >= 1.2

SCDB release 3.0.0 is the first release to support svnkit 1.2 and later. Due to API changes in svnkit, it is not possible to use this SCDB version with previous versions of svnkit. If trying to do this, you'll get error messages like class or method not found.

2.3.0/2.3.1: update.vo.config broken

Due to a mistake when SCDB 2.3.0 and 2.3.1 were packaged, the scdb-ant-utils task VOConfigTask implementing ant target update.vo.config is broken. There is no workaround, a fix will be released as soon as it is available.

This problem has been fixed in SCDB release 2.2.3. The only changed file is external/scdb-ant-utils/lib/quattor.jar.

New and Changed Features

3.0.0: group.name option added to ant

A new option, group.name, has been added to ant to select a particular group of clusters, when this feature is enabled:

ant -Dgroup.name=example

This option can be combined with cluster.name to select a particular cluster.

3.0.0: new deployment scripts

SCDB release 3.0.0 provides a new set of the deployment scripts, post-commit.py and build-tag.py. These scripts are a replacement for previous ones and are required to use a SVN server 1.5 or later. The options have changed. Both scripts use the same configuration file, /etc/quattor-deploy.conf.

3.0.0: svnkit updated to 1.3 (SVN 1.6)

SCDB release 3.0.0 provides support for SVN 1.6. This includes a new version of svnkit, 1.3, the version matching SVN 1.6 and an updated version of SCDB Ant Utils. Due to several changes in svnkit API, SCDB 3.0.0 requires at least svnkit 1.2 (compatible with SVN 1.5) and is not compatible with svnkit 1.1.x (compatible with SVN 1.4). If you need to stay with svnkit 1.1.x, use SCDB 2.3.2 instead.

2.3.2: update.vo.config fixed

SCDB ant target update.vo.config was buggy in SCDB 2.2 series and completely broken in SCDB 2.3 series. This has been fixed in SCDB 2.3.2. As a consequence of the fixes and some internal changes made to ensure that the account name associated with a given FQAN will never change, this may lead to some account name changes for some VOs. The creation of the new accounts will be properly handled by QWG templates but the NCM component used to manage accounts, ncm-accounts, is not able to removed the obsolete acccounts. It may be necessary (it is highly recommended on a CE) to remove these accounts, easily identifed as their name contains fqan. A template configuring a script to do it on every node where VO accounts are created is provided as an example. To use it, define variable GLITE_BASE_CONFIG_SITE to the name of this template.

2.3.1: ability to generate a list of all configured package repositories added

It is now possible, when updating templates describing package repositories (ant update.rep.templates), to generate an additional template, repository/allrepositories.tpl, that defines the variable ALL_REPOSITORIES. This variable is a nlist where the key is the repository name and the value the repository information as expected by /software/repositories. See SCDB users guide for details.

2.3.1: ant default configuration can be customized with quattor.build.properties

It is now possible to use file quattor.build.properties in SCDB instance root directory (the same directory as the one where quattor.build.xml resides) to customize default configuration values used by SCDB Ant tools. For details, look at SCDB users guide.

2.3.1: fix for profiles-info.xml not properly updated

Due to changes introduced in SCDB 2.3.0 for initial support of cluster groups, profiles-info.xml, a file used by other Quattor components to know the list of existing node profiles on a deployment server, was updated too early and new profiles added where not relected in the list during the first deployment (#212). This problem has been fixed.

See cluster groups documentation to know how to enable a specific deployment area per cluster group. Note that this feature is not fully integrated with standard deployment method, based on ant deploy.

2.3.1: cluster groups support changes

Initial support of hierarchical arrangement of cluster, introduced in SCDB 2.3.0 has been rewritten for a better integration and renamed cluster groups. It is now fully documented in SCDB users guide.

2.3.0: support for hierarchical arrangement of clusters

Using ant variable cfg.siteprefix (that must be defined in quattor.build.properties), it is possible to group clusters by sites in clusters directory. sites here doesn't refer to the usual sites directory but is a group name for clusters.

2.3.0: panc upgraded to 8.2.8

PAN compiler has been upgraded to version 8.2.8. In addition to fixes, this adds several new features:

• Support for selective debug using ant variables pan.debug.include and pan.debug.exclude. Both variables can be defined with -D option in ant. Their value are a regexp matching a template namespace.
• Ability to ignore some templates in dependency calculations. This is enabled by default for repository templates as normally a profile doesn't need to be rebuilt if there is not another change related to the repository template change (eg. a package replaced).

2.3.0: src/utils renamed utils/

All scripts and tools previously in src/utils have been moved to utils. A symlink src/utils has been created for backward compatibility.

2.2.0: removal of obsolete tools ==)

Some obsolete tools have been removed from SCDB: lld2pkgs, xmldb2hw, xml2pkgs.

2.2.0: sudo supported as replacement for ssh

In SCDB 2.2.0, it is possible to use sudo instead of ssh to launch the deployment script (build-tag.pl) from the SVN hook script. This is currently implemented as an alternative version of the SVN post-commit script, post-commit.sudo.

2.2.0: all external tools updated to last version

All SCDB external tools (in external) have been updated to last version available. This includes svnkit, ant and Saxon.

Note: svnkit version provided is still 1.1.x, compatible with SVN client 1.4.x).

2.2.0: new ant target to generate VO parameters

SCDB 2.2.0 provides a new ant target, update.vo.config. This target allows to generate all the VO related templates from VO Identity Cards on ​CIC portal for all registered VOs.

2.2.0: panc upgrade to v8,

SCDB 2.2.0 includes a major upgrade of panc compiler (8.2.3). This version of panc is backward compatible with panc v7 but in v8, a few syntax constructs valid in v7 are now deprecated and must be fixed to ensure future compatibility with next major panc version, v9. Check panc released notes for more details about these changes.

2.1.5 : Script to generate OS errata added

A script to generate OS errata, src/utils/misc/rpmErrata.tpl has been added to SCDB tools.

2.1.5 : OS template generation tools fixed for RHEL and CentOS

OS template generation tools have been fixed to properly support non SL distribution. Support for Xen kernel has also been added. Options have been added to src/utils/misc/buildOSTemplates to support RHEL5-like distribuation layout.

2.1.5 : build-tag.pl detects is another instance is running

In previous version, build-tag.pl didn't detect another instance was already running and this often resulted in corruption of SVN cache leading to the need to check out SCDB again. This is fixed in version 2.1.5.

2.1.5 : PAN compiler upgraded to 7.2.9

PAN compiler has been upgraded to 7.2.9. This version mainly fixes a bug in self variable processing and is required by QWG Templates gLite-3.1.0-3 and gLite-3.0.2-14.

2.1.4 : New script misc/buildOSTemplates

A new script to ease generation of templates for an OS distribution is available, src/utils/misc/buildOSTemplates. Look at tool documentation? for more information.

2.1.4 : New script misc/updateComponent

A new script to help updating NCM components from CVS is available, source:SCDB/trunk/src/utils/misc/updateComponent src/utils/misc/updateComponent. Look at tool documentation? for more information.

2.1.4 : New script misc/createPackagesTemplate

Feature provided by src/utils/CAs/create_template.pl have been extended. The new script is misc/createPackagesTemplate. Look at CAs template? generation for more information.

src/utils/CAs/create_template.pl is now a wrapper to the new script.

2.1.3 : Several message changes

To enhance usefulness of messages displayed by SCDB Ant Tools, several messages have been changed. It can be an issue if you have some local tools doing pattern matching on these messages. This affects only messages produced by SCDB Ant Tools, not by PAN compiler.

Main changes are :

• Line indicating a modified file during deploy : now both local and remote changes are checked at the same time and the report includes if the changes are local and/or remote.
• If deploy fails, the last message is deploy aborted instead of tag aborted
• The cluster name being compiled used to be the cluster path and the useful part was at the end of the line. Now the message has the cluster name first and the path to the cluster template after.

2.1.3 : Compiling only 1 cluster

With SCDB Tools 2.1.3, it is possible to compile only one cluster by adding the following option to ant command :

-Dcluster.name=name_of_cluster

name_of_cluster is the name of the directory under cfg/clusters.

Note : this option is forbidden during deploy.

2.1.2 : Support for repository/ namespace

With SCDB Tools 2.1.2, the default with update.rep.templates is to generate a template name that uses agreed repository namespace 'repository/). The namespace is prefixed to the template name as specified in the name= parameter in header comments. This is a change compared to behaviour in previous versions, where prefix repository_ was added in front of the template name.

To ease the transition to namespace, new version of SCDB tools preserve the name of an existing template, as defined in structure template line. Thus to produce a non namespaced repository template, just add the structure template line with the appropriate name or modify the line created by update.rep.templates. This will be preserved during future changes.

2.1.2 : New tag format requires a hook script upgrade

If you want to use the new hierarchichal tag format (e.g. yyyy/MM/dd-HH.mm.ss) introduced with SCDB 2.1.2, you need to update Subversion repository hook script and Quattor server script. Refer to SCDB installation instructions to know how to update them. In particular, take care to merge your site customizations into the hook script, as this script doesn't yet support an external configuration file.

2.1.0 : Subversion 1.4 requires a new httpd version (SL3)

If you choose to upgrade your Subversion client to 1.4 or later to be consistent with new SVNKit version, on SL3.x machines, you need to install a new version of httpd 2.0.46. You can found the new RPM at URL ​http://quattorsrv.lal.in2p3.fr/packages/os/sl305-i386/updates/httpd-2.0.46-61.1.ent.i386.rpm.

On SL4.x machines, the standard httpd version works properly with new Subversion client.

2.1.0 : SVNKit upgrade

SVNKit (former JavaSVN) has been upgraded to 1.1.2. This release is compatible with Subversion 1.4 client.

Before upgrading, you need to be aware that SVN 1.4 introduces a change in working area metadata format. As a consequence you cannot access/manage a working area with both a SVN 1.4 client and a previous version. There is no restriction in client/server versions combinations.

This may have a consequence on SCDB, depending on the client you use to do the SVN operations. This is because ant deploy task uses SVNKit and thus require a 1.4 working area metadata format. To solve any potention problems you need to upgrade the SVN client on the machine you use to administer SCDB (where you run ant) to version 1.4 or later.

As an alternative, you can use jsvn command provided with SVNKit instead of svn. The syntax is the same, refert to svn help for the command documentation. To use jsvn provided with SCDB tools, enter the following command :

external/svnkit/jsvn

2.0.0 : default compiler changed to panc v7

panc v7 (Java version of panc) is now the default compiler. Old compiler can still be used if necessary appending .c to ant task names, except for deploy.

Be sure to read instructions about running PAN compiler if you experiment any strange behaviour or slowness with the new compiler.

2.0.0 : configuration file for build-tag.pl

build-tag.pl is the script used to deploy configuration. It runs on the Quattor server. Version provided with 2.0.0 supports local customization through file /etc/build-tag.conf.

Use of new version supposes that you manually replace your existing script on the server with the new one provided in this release

Known Problems

In addition to known problems documented here, look at SCDB FAQ.

2.1.2 : update.rep.templates possible issue with empty repositories

SCDB tools v2 including 2.1.2 has a possible issue with empty repositories (#63). This happens if you removed from the associated template everything except comment header and structure template line. In this case, the template is not updated and the propeprty name is missing. This leads to an error in SPMA functions.

The easiest workaround is to also suppress the structure template line and update template name if the template is not namespaced.

This should be fixed in 2.1.3.

2.0.0 : issue with accentuated characters

The new PAN compiler (v7) encodes non ASCII characters in UTF-8. Unfortunatly, NCM components (particularly ncm-filecopy, and CCM are not prepared to handle properly UTF-8 encoded characters. The workaround is to remove these non ASCII characters from the templates (they are generally comments).

This problem also affects CCM before release 1.5.13. If after upgrade to SCDB 2.0.0, CCM complains about illegal characters, you need either to revert to previous version of SCDB and upgrade QWG templates first to 3.0.2-11 or later (see QWG Templates Release Notes), or to temporaly comment out all non ASCII characters in your templates (generally they are part of comments).

This will be fixed in a future release (#17).

Change Log

Changelog not available