wiki:ReleaseNotes/SCDB

Release Notes for SCDB

Table of Contents

  1. Releases
  2. Known Problems
    1. 3.2.0: xmldb format deprecated in panc v9.2
    2. 3.1.0: new VOConfigTask backward compatibility
    3. 3.0.0 requires svnkit >= 1.2
    4. 2.3.0/2.3.1: update.vo.config broken
  3. New and Changed Features
    1. 3.2.0: VOConfigTask update
    2. 3.2.0: panc upgraded to v9.2.0
    3. 3.1.2: panc upgraded to 8.4 serie
    4. 3.1.1: VOConfigTaks improvements
    5. 3.1.0: full rewrite of VOConfigTask
    6. 3.0.0: update_yum_repos script added
    7. 3.0.0: group.name ant option renamed cluster.group
    8. 3.0.0: group.name option added to ant
    9. 3.0.0: new deployment scripts
    10. 3.0.0: svnkit updated to 1.3 (SVN 1.6)
    11. 2.3.3: update.vo.config fixed
    12. 2.3.2: update.vo.config fixed
    13. 2.3.1: ability to generate a list of all configured package …
    14. 2.3.1: ant default configuration can be customized with …
    15. 2.3.1: fix for profiles-info.xml not properly updated
    16. 2.3.1: cluster groups support changes
    17. 2.3.0: support for hierarchical arrangement of clusters
    18. 2.3.0: panc upgraded to 8.2.8
    19. 2.3.0: src/utils renamed utils/
    20. 2.2.0: removal of obsolete tools ==)
    21. 2.2.0: sudo supported as replacement for ssh
    22. 2.2.0: all external tools updated to last version
    23. 2.2.0: new ant target to generate VO parameters
    24. 2.2.0: panc upgrade to v8,
    25. 2.1.5 : Script to generate OS errata added
    26. 2.1.5 : OS template generation tools fixed for RHEL and CentOS
    27. 2.1.5 : build-tag.pl detects is another instance is running
    28. 2.1.5 : PAN compiler upgraded to 7.2.9
    29. 2.1.4 : New script misc/buildOSTemplates
    30. 2.1.4 : New script misc/updateComponent
    31. 2.1.4 : New script misc/createPackagesTemplate
    32. 2.1.3 : Several message changes
    33. 2.1.3 : Compiling only 1 cluster
    34. 2.1.2 : Support for repository/ namespace
    35. 2.1.2 : New tag format requires a hook script upgrade
    36. 2.1.0 : Subversion 1.4 requires a new httpd version (SL3)
    37. 2.1.0 : SVNKit upgrade
    38. 2.0.0 : default compiler changed to panc v7
    39. 2.0.0 : configuration file for build-tag.pl
  4. Known Problems
    1. 2.1.2 : update.rep.templates possible issue with empty repositories
    2. 2.0.0 : issue with accentuated characters
  5. Change Log

SCDB is a set of related tools to administer Quattor templates. This page describes new features introduced, known problems and displays last entries from Subversion log.

Releases

The production version of SCDB Tools is can be downloaded from https://svn.lal.in2p3.fr/LCG/QWG/SCDB/tags/pro.

Date Version Description
27/03/07 2.0.0 panc v7 as default compiler
20/04/07 2.1.0 SVNKit upgrade (SVN 1.4 compatible)
04/05/07 2.1.1 panc v7.2.0
21/05/07 2.1.2 New tag format, support for repository/ namespace
19/06/07 2.1.3 panc 7.2.3, support for compilation of one cluster, bug fixes
07/11/07 2.1.4 panc 7.2.4, fixed OS templates generation tools
12/02/08 2.1.5 panc 7.2.9, fixed OS templates generation tools
11/11/08 2.2.0 panc 8.2.3, checkdeps.py, sudo-based hook script
12/07/09 2.3.0 panc 8.2.8, quatview, hiearchical arrangement of clusters
23/08/09 2.3.1 Fixes for problems in 2.3.0, rewrite of cluster group support, improvements to ant configuration
14/9/09 2.3.2 Fixed update.vo.config (completely broken in previous 2.3.x and buggy in 2.2)
15/01/10 2.3.3 (really!) fixed update.vo.config (bugs in VO group processing)
22/01/10 3.0.0 SCDB 2.3.3 + SVN 1.5/16 support (SVN 1.4 no longer supported)
23/12/10 3.1.0 New VOConfigTask (full rewrite, same features)
14/01/11 3.1.1 VOConfigTask minor changes and fixes
02/03/11 3.1.2 panc 8.4.7
21/04/11 3.1.3 Removal of obsolete/deprecated components (including src/utils symlink)

Known Problems

3.2.0: xmldb format deprecated in panc v9.2

In panc v9.2, xmldb profile format has been deprecated. The new recommended format is pan. SCDB 3.2.0 has been updated to use this format by default.

If the format to use is defined in a site-specific quattor.build.properties, you should remove this definition or update it. Generally there is such a quattor.build.properties on the Quattor server.

3.1.0: new VOConfigTask backward compatibility

As described in new features, SCDB 3.1.0 provides a new version of VOConfigTask that implements a new algorithm for account name generation. This new algorithm is better at producing unique account names and is used by default. In an existing configuration, it is important to disable this new algorithm and to use the legacy one as changing account names requires a lot of planning and should not be attempted on the fly.

To use the legacy algorithm, it is necessary to add in quattor.build.properties (located in the same directory as quattor.build.xml) the following line:

voConfigLegacySuffixAlgorithm=true

Note that this is no longer required with SCDB 3.1.1.

3.0.0 requires svnkit >= 1.2

SCDB release 3.0.0 is the first release to support svnkit 1.2 and later. Due to API changes in svnkit, it is not possible to use this SCDB version with previous versions of svnkit. If trying to do this, you'll get error messages like class or method not found.

By default, SCDB 3.0.0 is delivered with svnkit 1.3.x which is compatible with SVN 1.6. If you want to run it with SVN 1.5, you need to replace the svnkit version by 1.2.x, available on svnkit site.

2.3.0/2.3.1: update.vo.config broken

Due to a mistake when SCDB 2.3.0 and 2.3.1 were packaged, the scdb-ant-utils task VOConfigTask implementing ant target update.vo.config is broken. There is no workaround, a fix will be released as soon as it is available.

This problem has been fixed in SCDB release 2.2.3. The only changed file is external/scdb-ant-utils/lib/quattor.jar.

New and Changed Features

3.2.0: VOConfigTask update

In SCDB 3.2.0, VOConfigTask, which allows to update VO templates for the authoritative operation portal, has been updated to support the new format of VO ID card and the new feed from the operation portal.

3.2.0: panc upgraded to v9.2.0

SCDB 3.2.0 features panc v9.2.0. See panc release notes for the detailed list of changes. In this version, the profile format used so far by SCDB (xmldb) has been marked depracated. Use pan format instead. see known problems above.

3.1.2: panc upgraded to 8.4 serie

Release 3.1.2 of SCDB provides version 8.4.7 of the Pan compiler. Additionally this brings support for .pan as a valid template extension, in addition to .tpl and support for namespaced object templates.

3.1.1: VOConfigTaks improvements

Release 3.1.1 of SCDB contains several fixes and enhancements to VOConfigTask used to generate VO-related templates from VO ID cards. In particular:

  • VO card inconsistency where the only VOMS server available is not declared to support voms-admin is handled and the VOMS server is consider as supporting voms-admin (required by mkgridmap).
  • The account suffix generated by the new algorithm introduced in 3.1.0 is now added to VO template as attribute suffix2. This allows both legacy and new suffix to be present in the template and simplifies the migration. The option related to the use of the legacy algorithm is no longer needed and supported.

3.1.0: full rewrite of VOConfigTask

VOConfigTask has been rewritten from scratch to solve maintainability issues and to provide support for more VO parameters from the VO ID cards. This also provides improved consistency checks of the parameters in the VO ID card. The main visible changes are:

  • A VOMS server certificate is not added to the configuration if it is not valid or expired. As a result, a lot of previously declared oldcert are now removed.
  • oldcert is not added to the VO configuration if there is not an existing certificate valid and different from the one in the VO card.
  • A new algorithm has been implemented to produce unique account names for each different FQAN. A flaw in the previous algorithm tended to generate the same account name for several FQANs (it was based on FQAN length) and had to retry several times before finding a unique one, making the generated account name dependent on FQAN order. The new algorithm is used by default: to produce backward compatible templates, see known problems above.

3.0.0: update_yum_repos script added

A new script has been added into SCDB miscellaneous scripts, utils/misc/update_yum_repos. This scripts allows to convert RPM repositories into YUM repositories and to maintain them (update with last content avaible). Use option --help for available options.

This script is intended to be typically run as a cron job.

3.0.0: group.name ant option renamed cluster.group

In early SCDB 3.0, ant option group.name was introduced to compile only a selected cluster group. For the sake of consistency for cluster.name that allows to select a specific cluster, this option has been renamed cluster.group. group.name is still accepted but may be removed in a future release.

3.0.0: group.name option added to ant

A new option, group.name, has been added to ant to select a particular group of clusters, when this feature is enabled:

ant -Dgroup.name=example

This option can be combined with cluster.name to select a particular cluster.

3.0.0: new deployment scripts

SCDB release 3.0.0 provides a new set of the deployment scripts, post-commit.py and build-tag.py. These scripts are a replacement for previous ones and are required to use a SVN server 1.5 or later. The options have changed. Both scripts use the same configuration file, /etc/quattor-deploy.conf.

3.0.0: svnkit updated to 1.3 (SVN 1.6)

SCDB release 3.0.0 provides support for SVN 1.6. This includes a new version of svnkit, 1.3, the version matching SVN 1.6 and an updated version of SCDB Ant Utils. Due to several changes in svnkit API, SCDB 3.0.0 requires at least svnkit 1.2 (compatible with SVN 1.5) and is not compatible with svnkit 1.1.x (compatible with SVN 1.4). If you need to stay with svnkit 1.1.x, use SCDB 2.3.2 instead.

2.3.3: update.vo.config fixed

Version of VOConfigTask, used by update.vo.config, delivered with SCDB 2.3.2 still had issues with the handling of group and role combinations. SCDB 2.3.3 fixes all the remaining know issues. Note that the generated account suffix for each FQAN is changed for many FQANs. Refer to QWG templates release notes for more information on upgrading from previously generated templates.

2.3.2: update.vo.config fixed

SCDB ant target update.vo.config was buggy in SCDB 2.2 series and completely broken in SCDB 2.3 series. This has been fixed in SCDB 2.3.2. As a consequence of the fixes and some internal changes made to ensure that the account name associated with a given FQAN will never change, this may lead to some account name changes for some VOs. The creation of the new accounts will be properly handled by QWG templates but the NCM component used to manage accounts, ncm-accounts, is not able to removed the obsolete acccounts. It may be necessary (it is highly recommended on a CE) to remove these accounts, easily identifed as their name contains fqan. A template configuring a script to do it on every node where VO accounts are created is provided as an example. To use it, define variable GLITE_BASE_CONFIG_SITE to the name of this template.

2.3.1: ability to generate a list of all configured package repositories added

It is now possible, when updating templates describing package repositories (ant update.rep.templates), to generate an additional template, repository/allrepositories.tpl, that defines the variable ALL_REPOSITORIES. This variable is a nlist where the key is the repository name and the value the repository information as expected by /software/repositories. See SCDB users guide for details.

2.3.1: ant default configuration can be customized with quattor.build.properties

It is now possible to use file quattor.build.properties in SCDB instance root directory (the same directory as the one where quattor.build.xml resides) to customize default configuration values used by SCDB Ant tools. For details, look at SCDB users guide.

2.3.1: fix for profiles-info.xml not properly updated

Due to changes introduced in SCDB 2.3.0 for initial support of cluster groups, profiles-info.xml, a file used by other Quattor components to know the list of existing node profiles on a deployment server, was updated too early and new profiles added where not relected in the list during the first deployment (#212). This problem has been fixed.

See cluster groups documentation to know how to enable a specific deployment area per cluster group. Note that this feature is not fully integrated with standard deployment method, based on ant deploy.

2.3.1: cluster groups support changes

Initial support of hierarchical arrangement of cluster, introduced in SCDB 2.3.0 has been rewritten for a better integration and renamed cluster groups. It is now fully documented in SCDB users guide.

2.3.0: support for hierarchical arrangement of clusters

Using ant variable cfg.siteprefix (that must be defined in quattor.build.properties), it is possible to group clusters by sites in clusters directory. sites here doesn't refer to the usual sites directory but is a group name for clusters.

2.3.0: panc upgraded to 8.2.8

PAN compiler has been upgraded to version 8.2.8. In addition to fixes, this adds several new features:

  • Support for selective debug using ant variables pan.debug.include and pan.debug.exclude. Both variables can be defined with -D option in ant. Their value are a regexp matching a template namespace.
  • Ability to ignore some templates in dependency calculations. This is enabled by default for repository templates as normally a profile doesn't need to be rebuilt if there is not another change related to the repository template change (eg. a package replaced).

2.3.0: src/utils renamed utils/

All scripts and tools previously in src/utils have been moved to utils. A symlink src/utils has been created for backward compatibility.

2.2.0: removal of obsolete tools ==)

Some obsolete tools have been removed from SCDB: lld2pkgs, xmldb2hw, xml2pkgs.

2.2.0: sudo supported as replacement for ssh

In SCDB 2.2.0, it is possible to use sudo instead of ssh to launch the deployment script (build-tag.pl) from the SVN hook script. This is currently implemented as an alternative version of the SVN post-commit script, post-commit.sudo.

2.2.0: all external tools updated to last version

All SCDB external tools (in external) have been updated to last version available. This includes svnkit, ant and Saxon.

Note: svnkit version provided is still 1.1.x, compatible with SVN client 1.4.x).

2.2.0: new ant target to generate VO parameters

SCDB 2.2.0 provides a new ant target, update.vo.config. This target allows to generate all the VO related templates from VO Identity Cards on CIC portal for all registered VOs.

2.2.0: panc upgrade to v8,

SCDB 2.2.0 includes a major upgrade of panc compiler (8.2.3). This version of panc is backward compatible with panc v7 but in v8, a few syntax constructs valid in v7 are now deprecated and must be fixed to ensure future compatibility with next major panc version, v9. Check panc released notes for more details about these changes.

2.1.5 : Script to generate OS errata added

A script to generate OS errata, src/utils/misc/rpmErrata.tpl has been added to SCDB tools.

2.1.5 : OS template generation tools fixed for RHEL and CentOS

OS template generation tools have been fixed to properly support non SL distribution. Support for Xen kernel has also been added. Options have been added to src/utils/misc/buildOSTemplates to support RHEL5-like distribuation layout.

2.1.5 : build-tag.pl detects is another instance is running

In previous version, build-tag.pl didn't detect another instance was already running and this often resulted in corruption of SVN cache leading to the need to check out SCDB again. This is fixed in version 2.1.5.

2.1.5 : PAN compiler upgraded to 7.2.9

PAN compiler has been upgraded to 7.2.9. This version mainly fixes a bug in self variable processing and is required by QWG Templates gLite-3.1.0-3 and gLite-3.0.2-14.

2.1.4 : New script misc/buildOSTemplates

A new script to ease generation of templates for an OS distribution is available, src/utils/misc/buildOSTemplates. Look at tool documentation? for more information.

2.1.4 : New script misc/updateComponent

A new script to help updating NCM components from CVS is available, source:SCDB/trunk/src/utils/misc/updateComponent src/utils/misc/updateComponent. Look at tool documentation? for more information.

2.1.4 : New script misc/createPackagesTemplate

Feature provided by src/utils/CAs/create_template.pl have been extended. The new script is misc/createPackagesTemplate. Look at CAs template? generation for more information.

src/utils/CAs/create_template.pl is now a wrapper to the new script.

2.1.3 : Several message changes

To enhance usefulness of messages displayed by SCDB Ant Tools, several messages have been changed. It can be an issue if you have some local tools doing pattern matching on these messages. This affects only messages produced by SCDB Ant Tools, not by PAN compiler.

Main changes are :

  • Line indicating a modified file during deploy : now both local and remote changes are checked at the same time and the report includes if the changes are local and/or remote.
  • If deploy fails, the last message is deploy aborted instead of tag aborted
  • The cluster name being compiled used to be the cluster path and the useful part was at the end of the line. Now the message has the cluster name first and the path to the cluster template after.

2.1.3 : Compiling only 1 cluster

With SCDB Tools 2.1.3, it is possible to compile only one cluster by adding the following option to ant command :

-Dcluster.name=name_of_cluster

name_of_cluster is the name of the directory under cfg/clusters.

Note : this option is forbidden during deploy.

2.1.2 : Support for repository/ namespace

With SCDB Tools 2.1.2, the default with update.rep.templates is to generate a template name that uses agreed repository namespace 'repository/). The namespace is prefixed to the template name as specified in the name= parameter in header comments. This is a change compared to behaviour in previous versions, where prefix repository_ was added in front of the template name.

To ease the transition to namespace, new version of SCDB tools preserve the name of an existing template, as defined in structure template line. Thus to produce a non namespaced repository template, just add the structure template line with the appropriate name or modify the line created by update.rep.templates. This will be preserved during future changes.

2.1.2 : New tag format requires a hook script upgrade

If you want to use the new hierarchichal tag format (e.g. yyyy/MM/dd-HH.mm.ss) introduced with SCDB 2.1.2, you need to update Subversion repository hook script and Quattor server script. Refer to SCDB installation instructions to know how to update them. In particular, take care to merge your site customizations into the hook script, as this script doesn't yet support an external configuration file.

2.1.0 : Subversion 1.4 requires a new httpd version (SL3)

If you choose to upgrade your Subversion client to 1.4 or later to be consistent with new SVNKit version, on SL3.x machines, you need to install a new version of httpd 2.0.46. You can found the new RPM at URL http://quattorsrv.lal.in2p3.fr/packages/os/sl305-i386/updates/httpd-2.0.46-61.1.ent.i386.rpm.

On SL4.x machines, the standard httpd version works properly with new Subversion client.

2.1.0 : SVNKit upgrade

SVNKit (former JavaSVN) has been upgraded to 1.1.2. This release is compatible with Subversion 1.4 client.

Before upgrading, you need to be aware that SVN 1.4 introduces a change in working area metadata format. As a consequence you cannot access/manage a working area with both a SVN 1.4 client and a previous version. There is no restriction in client/server versions combinations.

This may have a consequence on SCDB, depending on the client you use to do the SVN operations. This is because ant deploy task uses SVNKit and thus require a 1.4 working area metadata format. To solve any potention problems you need to upgrade the SVN client on the machine you use to administer SCDB (where you run ant) to version 1.4 or later.

As an alternative, you can use jsvn command provided with SVNKit instead of svn. The syntax is the same, refert to svn help for the command documentation. To use jsvn provided with SCDB tools, enter the following command :

external/svnkit/jsvn

2.0.0 : default compiler changed to panc v7

panc v7 (Java version of panc) is now the default compiler. Old compiler can still be used if necessary appending .c to ant task names, except for deploy.

Be sure to read instructions about running PAN compiler if you experiment any strange behaviour or slowness with the new compiler.

2.0.0 : configuration file for build-tag.pl

build-tag.pl is the script used to deploy configuration. It runs on the Quattor server. Version provided with 2.0.0 supports local customization through file /etc/build-tag.conf.

Use of new version supposes that you manually replace your existing script on the server with the new one provided in this release

Known Problems

In addition to known problems documented here, look at SCDB FAQ.

2.1.2 : update.rep.templates possible issue with empty repositories

SCDB tools v2 including 2.1.2 has a possible issue with empty repositories (#63). This happens if you removed from the associated template everything except comment header and structure template line. In this case, the template is not updated and the propeprty name is missing. This leads to an error in SPMA functions.

The easiest workaround is to also suppress the structure template line and update template name if the template is not namespaced.

This should be fixed in 2.1.3.

2.0.0 : issue with accentuated characters

The new PAN compiler (v7) encodes non ASCII characters in UTF-8. Unfortunatly, NCM components (particularly ncm-filecopy, and CCM are not prepared to handle properly UTF-8 encoded characters. The workaround is to remove these non ASCII characters from the templates (they are generally comments).

This problem also affects CCM before release 1.5.13. If after upgrade to SCDB 2.0.0, CCM complains about illegal characters, you need either to revert to previous version of SCDB and upgrade QWG templates first to 3.0.2-11 or later (see QWG Templates Release Notes), or to temporaly comment out all non ASCII characters in your templates (generally they are part of comments).

This will be fixed in a future release (#17).

Change Log

Changelog not available

Last modified 12 years ago Last modified on Jun 27, 2012, 10:59:21 AM