= Release Notes for SCDB = [[TracNav]] [[TOC(inline)]] SCDB is a set of related tools to administer Quattor templates. This page describes new features introduced, known problems and displays last entries from Subversion log. == Releases == The [source:SCDB/tags/pro production version] of SCDB Tools is can be downloaded from https://svn.lal.in2p3.fr/LCG/QWG/SCDB/tags/pro. ||= Date =||= Version =||= Description =|| || 27/03/07 || [milestone:SCDB-2.0.0 2.0.0] || panc v7 as default compiler || || 20/04/07 || [milestone:SCDB-2.1.0 2.1.0] || SVNKit upgrade (SVN 1.4 compatible) || || 04/05/07 || [milestone:SCDB-2.1.1 2.1.1] || panc v7.2.0 || || 21/05/07 || [milestone:SCDB-2.1.2 2.1.2] || New tag format, support for repository/ namespace || || 19/06/07 || [milestone:SCDB-2.1.3 2.1.3] || panc 7.2.3, support for compilation of one cluster, bug fixes || || 07/11/07 || [milestone:SCDB-2.1.4 2.1.4] || panc 7.2.4, fixed OS templates generation tools || || 12/02/08 || [milestone:SCDB-2.1.5 2.1.5] || panc 7.2.9, fixed OS templates generation tools || || 11/11/08 || [milestone:SCDB-2.2.0 2.2.0] || panc 8.2.3, checkdeps.py, sudo-based hook script || || 12/07/09 || [milestone:SCDB-2.3.0 2.3.0] || panc 8.2.8, quatview, hiearchical arrangement of clusters || || 23/08/09 || [milestone:SCDB-2.3.1 2.3.1] || Fixes for problems in [milestone:SCDB-2.3.0 2.3.0], rewrite of cluster group support, improvements to ant configuration || || 14/9/09 || [milestone:SCDB-2.3.2 2.3.2] || Fixed `update.vo.config` (completely broken in previous 2.3.x and buggy in 2.2) || || 15/01/10 || [milestone:SCDB-2.3.3 2.3.3] || (really!) fixed `update.vo.config` (bugs in VO group processing) || || 22/01/10 || [milestone:SCDB-3.0.0 3.0.0] || SCDB [milestone:SCDB-2.3.3 2.3.3] + SVN 1.5/16 support (SVN 1.4 no longer supported) || || 23/12/10 || [milestone:SCDB-3.1.O 3.1.0] || New VOConfigTask (full rewrite, same features) || || 14/01/11 || [milestone:SCDB-3.1.1 3.1.1] || VOConfigTask minor changes and fixes || || 02/03/11 || [milestone:SCDB-3.1.2 3.1.2] || panc 8.4.7 || || 21/04/11 || [milestone:SCDB-3.1.3 3.1.3] || Removal of obsolete/deprecated components (including `src/utils` symlink) || == Known Problems == === 3.2.0: xmldb format deprecated in panc v9.2 === #KB320Panc92 In panc v9.2, `xmldb` profile format has been deprecated. The new recommended format is `pan`. SCDB 3.2.0 has been updated to use this format by default. If the format to use is defined in a site-specific `quattor.build.properties`, you should remove this definition or update it. Generally there is such a `quattor.build.properties` on the Quattor server. === 3.1.0: new VOConfigTask backward compatibility === #KB310VOConfigTask As described in [#NF310VOConfigTask new features], SCDB [milestone:SCDB-3.1.O 3.1.0] provides a new version of `VOConfigTask` that implements a new algorithm for account name generation. This new algorithm is better at producing unique account names and is used by default. In an existing configuration, it is important to disable this new algorithm and to use the legacy one as changing account names requires a lot of planning and '''should not be attempted on the fly'''. To use the legacy algorithm, it is necessary to add in `quattor.build.properties` (located in the same directory as `quattor.build.xml`) the following line: {{{ voConfigLegacySuffixAlgorithm=true }}} '''Note that this is no longer required with SCDB [milestone:SCDB-3.1.1 3.1.1]. === 3.0.0 requires svnkit >= 1.2 === SCDB release [milestone:SCDB-3.0.0 3.0.0] is the first release to support `svnkit` 1.2 and later. Due to API changes in `svnkit`, it is not possible to use this SCDB version with previous versions of `svnkit`. If trying to do this, you'll get error messages like class or method not found. By default, SCDB [milestone:SCDB-3.0.0 3.0.0] is delivered with `svnkit` 1.3.x which is compatible with SVN 1.6. If you want to run it with SVN 1.5, you need to replace the `svnkit` version by 1.2.x, available on [http://svnkit.com svnkit site]. === 2.3.0/2.3.1: update.vo.config broken === Due to a mistake when SCDB 2.3.0 and 2.3.1 were packaged, the scdb-ant-utils task `VOConfigTask` implementing ant target `update.vo.config` is broken. There is no workaround, a fix will be released as soon as it is available. '''This problem has been fixed in SCDB release [milestone:SCDB-2.3.2 2.2.3].''' The only changed file is `external/scdb-ant-utils/lib/quattor.jar`. == New and Changed Features == === 3.2.0: VOConfigTask update === In SCDB 3.2.0, VOConfigTask, which allows to update VO templates for the authoritative operation portal, has been updated to support the new format of VO ID card and the new feed from the operation portal. === 3.2.0: panc upgraded to v9.2.0 === SCDB 3.2.0 features panc v9.2.0. See panc release notes for the detailed list of changes. In this version, the profile format used so far by SCDB (`xmldb`) has been marked depracated. Use `pan` format instead. see [#KB320Panc92 known problems] above. === 3.1.2: panc upgraded to 8.4 serie === Release [milestone:SCDB-3.1.2 3.1.2] of SCDB provides version 8.4.7 of the Pan compiler. Additionally this brings support for `.pan` as a valid template extension, in addition to `.tpl` and support for namespaced object templates. === 3.1.1: VOConfigTaks improvements === Release [milestone:SCDB-3.1.1 3.1.1] of SCDB contains several fixes and enhancements to VOConfigTask used to generate VO-related templates from VO ID cards. In particular: * VO card inconsistency where the only VOMS server available is not declared to support `voms-admin` is handled and the VOMS server is consider as supporting `voms-admin` (required by mkgridmap). * The account suffix generated by the new algorithm introduced in 3.1.0 is now added to VO template as attribute `suffix2`. This allows both legacy and new suffix to be present in the template and simplifies the migration. The option related to the use of the legacy algorithm is no longer needed and supported. === 3.1.0: full rewrite of VOConfigTask === #NF310VOConfigTask VOConfigTask has been rewritten from scratch to solve maintainability issues and to provide support for more VO parameters from the VO ID cards. This also provides improved consistency checks of the parameters in the VO ID card. The main visible changes are: * A VOMS server certificate is not added to the configuration if it is not valid or expired. As a result, a lot of previously declared `oldcert` are now removed. * `oldcert` is not added to the VO configuration if there is not an existing certificate valid and different from the one in the VO card. * A new algorithm has been implemented to produce unique account names for each different FQAN. A flaw in the previous algorithm tended to generate the same account name for several FQANs (it was based on FQAN length) and had to retry several times before finding a unique one, making the generated account name dependent on FQAN order. The new algorithm is used by default: to produce backward compatible templates, see [#KB310VOConfigTask known problems] above. === 3.0.0: update_yum_repos script added === A new script has been added into SCDB [source:SCDB/trunk/utils/misc miscellaneous scripts], [source:SCDB/trunk/utils/misc/update_yum_repos utils/misc/update_yum_repos]. This scripts allows to convert RPM repositories into YUM repositories and to maintain them (update with last content avaible). Use option `--help` for available options. This script is intended to be typically run as a cron job. === 3.0.0: group.name ant option renamed cluster.group === In early [milestone:SCDB-3.0.0 SCDB 3.0], ant option `group.name` was introduced to compile only a selected cluster group. For the sake of consistency for `cluster.name` that allows to select a specific cluster, this option has been renamed `cluster.group`. `group.name` is still accepted but may be removed in a future release. === 3.0.0: group.name option added to ant === A new option, `group.name`, has been added to ant to select a particular group of clusters, when this feature is enabled: {{{ ant -Dgroup.name=example }}} This option can be combined with `cluster.name` to select a particular cluster. === 3.0.0: new deployment scripts === SCDB release [milestone:SCDB-3.0.0 3.0.0] provides a new set of the deployment scripts, [source:SCDB/trunk/src/hooks/post-commit.py post-commit.py] and [source:SCDB/trunk/src/hooks/build-tag.py build-tag.py]. These scripts are a replacement for previous ones and are required to use a SVN server 1.5 or later. The options have changed. Both scripts use the same configuration file, `/etc/quattor-deploy.conf`. === 3.0.0: svnkit updated to 1.3 (SVN 1.6) === SCDB release [milestone:SCDB-3.0.0 3.0.0] provides support for SVN 1.6. This includes a new version of `svnkit`, 1.3, the version matching SVN 1.6 and an updated version of SCDB Ant Utils. Due to several changes in `svnkit` API, SCDB [milestone:SCDB-3.0.0 3.0.0] requires at least `svnkit` 1.2 (compatible with SVN 1.5) and is not compatible with `svnkit` 1.1.x (compatible with SVN 1.4). If you need to stay with `svnkit` 1.1.x, use SCDB [milestone:SCDB-2.3.2 2.3.2] instead. === 2.3.3: update.vo.config fixed === Version of VOConfigTask, used by `update.vo.config`, delivered with [milestone:SCDB-2.3.2 SCDB 2.3.2] still had issues with the handling of group and role combinations. [milestone:SCDB-2.3.3 SCDB 2.3.3] fixes all the remaining know issues. Note that the generated account suffix for each FQAN is changed for many FQANs. Refer to QWG templates [/ReleaseNotes/gLite-3.2 release notes] for more information on upgrading from previously generated templates. === 2.3.2: update.vo.config fixed === SCDB `ant` target `update.vo.config` was buggy in SCDB 2.2 series and [#a2.3.02.3.1:update.vo.configbroken completely broken] in SCDB 2.3 series. This has been fixed in SCDB [milestone:SCDB-2.3.2 2.3.2]. As a consequence of the fixes and some internal changes made to ensure that the account name associated with a given FQAN will never change, this may lead to some account name changes for some VOs. The creation of the new accounts will be properly handled by QWG templates but the NCM component used to manage accounts, `ncm-accounts`, is not able to removed the obsolete acccounts. It may be necessary (it is highly recommended on a CE) to remove these accounts, easily identifed as their name contains `fqan`. A template configuring a script to do it on every node where VO accounts are created is provided as an [source:templates/trunk/sites/example/site/misc/purge_fqan_accounts.tpl example]. To use it, define variable `GLITE_BASE_CONFIG_SITE` to the name of this template. === 2.3.1: ability to generate a list of all configured package repositories added === It is now possible, when updating templates describing package repositories (`ant update.rep.templates`), to generate an additional template, `repository/allrepositories.tpl`, that defines the variable `ALL_REPOSITORIES`. This variable is a nlist where the key is the repository name and the value the repository information as expected by `/software/repositories`. See SCDB [/wiki/Doc/SCDB/Usage#ManagementofPackageRepositoryTemplates users guide] for details. === 2.3.1: ant default configuration can be customized with quattor.build.properties === It is now possible to use file `quattor.build.properties` in SCDB instance root directory (the same directory as the one where `quattor.build.xml` resides) to customize default configuration values used by SCDB Ant tools. For details, look at SCDB [/wiki/Doc/SCDB/Usage#ConfiguringSCDBAntTools users guide]. === 2.3.1: fix for profiles-info.xml not properly updated === Due to changes introduced in SCDB [milestone:SCDB-2.3.0 2.3.0] for [#supportforhierarchicalarrangementofclusters initial support] of ''cluster groups'', `profiles-info.xml`, a file used by other Quattor components to know the list of existing node profiles on a deployment server, was updated too early and new profiles added where not relected in the list during the first deployment (#212). This problem has been fixed. See [/wiki/Doc/SCDB/Usage#ClusterGroups cluster groups] documentation to know how to enable a specific deployment area per cluster group. Note that this feature is not fully integrated with standard deployment method, based on `ant deploy`. === 2.3.1: cluster groups support changes === [#supportforhierarchicalarrangementofclusters Initial support] of hierarchical arrangement of cluster, introduced in SCDB [milestone:SCDB-2.3.0 2.3.0] has been rewritten for a better integration and renamed ''cluster groups''. It is now fully documented in SCDB [/wiki/Doc/SCDB/Usage#ClusterGroups users guide]. === 2.3.0: support for hierarchical arrangement of clusters === Using ant variable `cfg.siteprefix` (that must be defined in `quattor.build.properties`), it is possible to group clusters by ''sites'' in `clusters` directory. ''sites'' here doesn't refer to the usual `sites` directory but is a group name for clusters. === 2.3.0: panc upgraded to 8.2.8 === PAN compiler has been upgraded to version 8.2.8. In addition to fixes, this adds several new features: * Support for selective debug using ant variables `pan.debug.include` and `pan.debug.exclude`. Both variables can be defined with `-D` option in ant. Their value are a regexp matching a template namespace. * Ability to ignore some templates in dependency calculations. This is enabled by default for repository templates as normally a profile doesn't need to be rebuilt if there is not another change related to the repository template change (eg. a package replaced). === 2.3.0: src/utils renamed utils/ === All scripts and tools previously in `src/utils` have been moved to `utils`. A symlink `src/utils` has been created for backward compatibility. === 2.2.0: removal of obsolete tools ==) Some obsolete tools have been removed from SCDB: lld2pkgs, xmldb2hw, xml2pkgs. === 2.2.0: sudo supported as replacement for ssh === In [milestone:SCDB-2.2.0 SCDB 2.2.0], it is possible to use `sudo` instead of `ssh` to launch the deployment script (`build-tag.pl`) from the SVN hook script. This is currently implemented as an alternative version of the SVN post-commit script, [source:templates/SCDB/trunk/src/hooks/post-commit.sudo post-commit.sudo]. === 2.2.0: all external tools updated to last version === All SCDB external tools (in `external`) have been updated to last version available. This includes `svnkit`, `ant` and `Saxon`. ''Note: `svnkit` version provided is still 1.1.x, compatible with SVN client 1.4.x).'' === 2.2.0: new ant target to generate VO parameters === [milestone:SCDB-2.2.0 SCDB 2.2.0] provides a new `ant` target, `update.vo.config`. This target allows to generate all the VO related templates from VO Identity Cards on [http://www.gridops.org CIC portal] for all registered VOs. === 2.2.0: panc upgrade to v8, === [milestone:SCDB-2.2.0 SCDB 2.2.0] includes a major upgrade of panc compiler ([milestone:panc-8.2.3 8.2.3]). This version of panc is backward compatible with panc v7 but in v8, a few syntax constructs valid in v7 are now deprecated and must be fixed to ensure future compatibility with next major panc version, [milestone:panc-9.0.0 v9]. Check panc released notes for more details about these changes. === 2.1.5 : Script to generate OS errata added === A script to generate OS errata, [source:/SCDB/trunk/src/utils/misc/rpmErrata.tpl src/utils/misc/rpmErrata.tpl] has been added to SCDB tools. === 2.1.5 : OS template generation tools fixed for RHEL and CentOS === OS template generation tools have been fixed to properly support non SL distribution. Support for Xen kernel has also been added. Options have been added to [source:/SCDB/trunk/src/utils/misc/buildOSTemplates src/utils/misc/buildOSTemplates] to support RHEL5-like distribuation layout. === 2.1.5 : build-tag.pl detects is another instance is running === In previous version, build-tag.pl didn't detect another instance was already running and this often resulted in corruption of SVN cache leading to the need to check out SCDB again. This is fixed in version 2.1.5. === 2.1.5 : PAN compiler upgraded to 7.2.9 === PAN compiler has been upgraded to 7.2.9. This version mainly fixes a bug in self variable processing and is required by QWG Templates [milestone:gLite-3.1.0-3 gLite-3.1.0-3] and [milestone:gLite-3.0.2-14 gLite-3.0.2-14]. === 2.1.4 : New script misc/buildOSTemplates === A new script to ease generation of templates for an OS distribution is available, [source:SCDB/trunk/src/utils/misc/buildOSTemplates src/utils/misc/buildOSTemplates]. Look at [wiki:Development/Templates/Generated#OSTemplates tool documentation] for more information. === 2.1.4 : New script misc/updateComponent === A new script to help updating NCM components from CVS is available, [source:SCDB/trunk/src/utils/misc/updateComponent source:SCDB/trunk/src/utils/misc/updateComponent src/utils/misc/updateComponent]. Look at [wiki:Development/Templates/Components#InstallingaNewVersion tool documentation] for more information. === 2.1.4 : New script misc/createPackagesTemplate === Feature provided by [source:SCDB/trunk/src/utils/CAs/create_template.pl src/utils/CAs/create_template.pl] have been extended. The new script is [source:SCDB/trunk/src/utils/misc/createPackagesTemplate misc/createPackagesTemplate]. Look at [wiki:Development/Templates/Generated#TrustedCAsTemplate CAs template] generation for more information. [source:SCDB/trunk/src/utils/CAs/create_template.pl src/utils/CAs/create_template.pl] is now a wrapper to the new script. === 2.1.3 : Several message changes === To enhance usefulness of messages displayed by SCDB Ant Tools, several messages have been changed. It can be an issue if you have some local tools doing pattern matching on these messages. This affects only messages produced by SCDB Ant Tools, not by PAN compiler. Main changes are : * Line indicating a modified file during `deploy` : now both local and remote changes are checked at the same time and the report includes if the changes are local and/or remote. * If `deploy` fails, the last message is `deploy aborted` instead of `tag aborted` * The cluster name being compiled used to be the cluster path and the useful part was at the end of the line. Now the message has the cluster name first and the path to the cluster template after. === 2.1.3 : Compiling only 1 cluster === With SCDB Tools 2.1.3, it is possible to compile only one cluster by adding the following option to `ant` command : {{{ -Dcluster.name=name_of_cluster }}} `name_of_cluster` is the name of the directory under `cfg/clusters`. ''Note : this option is forbidden during `deploy`.'' === 2.1.2 : Support for repository/ namespace === With SCDB Tools 2.1.2, the default with `update.rep.templates` is to generate a template name that uses agreed repository namespace '`repository/`). The namespace is prefixed to the template name as specified in the `name=` parameter in header comments. This is a change compared to behaviour in previous versions, where prefix `repository_` was added in front of the template name. To ease the transition to namespace, new version of SCDB tools preserve the name of an existing template, as defined in `structure template` line. Thus to produce a non namespaced repository template, just add the `structure template` line with the appropriate name or modify the line created by `update.rep.templates`. This will be preserved during future changes. === 2.1.2 : New tag format requires a hook script upgrade === If you want to use the new hierarchichal tag format (e.g. yyyy/MM/dd-HH.mm.ss) introduced with SCDB 2.1.2, you need to update Subversion repository hook script and Quattor server script. Refer to [wiki:Download/SCDB#Installationofhookscriptandserverscript SCDB installation] instructions to know how to update them. In particular, take care to merge your site customizations into the hook script, as this script doesn't yet support an external configuration file. === 2.1.0 : Subversion 1.4 requires a new httpd version (SL3) === If you choose to upgrade your Subversion client to 1.4 or later to be consistent with new SVNKit version, on SL3.x machines, you need to install a new version of httpd 2.0.46. You can found the new RPM at URL http://quattorsrv.lal.in2p3.fr/packages/os/sl305-i386/updates/httpd-2.0.46-61.1.ent.i386.rpm. On SL4.x machines, the standard httpd version works properly with new Subversion client. === 2.1.0 : SVNKit upgrade === SVNKit (former JavaSVN) has been upgraded to 1.1.2. This release is compatible with Subversion 1.4 client. Before upgrading, you need to be aware that SVN 1.4 introduces a change in working area metadata format. As a consequence you cannot access/manage a working area with both a SVN 1.4 client and a previous version. There is no restriction in client/server versions combinations. This may have a consequence on SCDB, depending on the client you use to do the SVN operations. This is because `ant deploy` task uses SVNKit and thus require a 1.4 working area metadata format. To solve any potention problems you need to upgrade the SVN client on the machine you use to administer SCDB (where you run `ant`) to version 1.4 or later. As an alternative, you can use `jsvn` command provided with SVNKit instead of `svn`. The syntax is the same, refert to `svn help` for the command documentation. To use `jsvn` provided with SCDB tools, enter the following command : {{{ external/svnkit/jsvn }}} === 2.0.0 : default compiler changed to panc v7 === panc v7 (Java version of panc) is now the default compiler. Old compiler can still be used if necessary appending `.c` to ant task names, except for `deploy`. Be sure to read instructions about [wiki:Doc/compiler/running running PAN compiler] if you experiment any strange behaviour or slowness with the new compiler. === 2.0.0 : configuration file for build-tag.pl === [source:SCDB/trunk/build-tag.pl build-tag.pl] is the script used to deploy configuration. It runs on the Quattor server. Version provided with 2.0.0 supports local customization through file [wiki:Doc/SCDB/Server#DeploymentScript /etc/build-tag.conf]. Use of new version supposes that you manually replace your existing script on the server with the new one provided in this release == Known Problems == In addition to known problems documented here, look at [wiki:FAQ/SCDB SCDB FAQ]. === 2.1.2 : update.rep.templates possible issue with empty repositories === SCDB tools v2 including 2.1.2 has a possible issue with empty repositories (#63). This happens if you removed from the associated template everything except comment header '''and''' `structure template` line. In this case, the template is not updated and the propeprty `name` is missing. This leads to an error in SPMA functions. The easiest workaround is to also suppress the `structure template` line and update template name if the template is not namespaced. This should be fixed in 2.1.3. === 2.0.0 : issue with accentuated characters === The new PAN compiler (v7) encodes non ASCII characters in UTF-8. Unfortunatly, NCM components (particularly ncm-filecopy, and CCM are not prepared to [wiki:ReleaseNotes/panc#UTF-8encodingofnonASCIIcharacters handle properly UTF-8] encoded characters. The workaround is to remove these non ASCII characters from the templates (they are generally comments). This problem also affects CCM before release 1.5.13. If after upgrade to SCDB 2.0.0, CCM complains about illegal characters, you need either to revert to previous version of SCDB and upgrade QWG templates first to 3.0.2-11 or later (see QWG Templates [wiki:ReleaseNotes/gLite-3.0#gLite-3.0.2-11:upgradetopancv7recommended Release Notes]), or to temporaly comment out all non ASCII characters in your templates (generally they are part of comments). This will be fixed in a future release (#17). == Change Log == [[ChangeLog(SCDB/trunk,20)]]