15th Quattor Workshop - Bordeaux - 6-8/3/2013

Table of Contents

1. Update since last workshop
2. Pan Compiler Status and Roadmap
3. ncm-spma v3
4. Aquilon

Update since last workshop

2 releases made: who use them?

• Probably no site using them yet...

New ncm-spma is production ready: need to be tested

panc v10 released.

QWG

• grid templates: continued support, up-to-date with last MW versions
  • Mostly done by Guillaume and Jerome
  • Migration to GitHub still to be done
• StratusLab templates: available for new versions of the MW but insufficiently sync with GitHub
  • Discuss with Guillaume (main maintainer) a more regular contribution to GitHub: use branches if necessary

Aquilon

• Packaging still in progress: 2 repositories, mainly done at UGent and RAL
• RAL is already using it to manage the cloud machines and some other test machines
  • Still trying to figure out how to migrate the QWG machine types and database related things

Dashboard

• Still working on it: indexing added
  • Requires Python 2.7
• datawharehouse repo on GitHub

Solaris port working at MS

• Solaris 11
• Based on a fork made 1 year ago
  • Main changes are in network config and cron
  • No package management yet: everything installed from AFS before running components
  • Would like to merge back to the mainstream Quattor

QRD/QRC

• New QRD available and used at MS. Hope to publish it in one month.
  • VMWare (MS specific) and NetApp
  • Python
• Would like to do the NetApp management through QRD this year
  • Currently, only used at initial installation
• Also plans for switches

Quattor FS

• A few bug fixes and enhancements

JSON profiles

• Smaller, no need to escape

SCDB

• Pb with last scdb-ant-utils: changes related to JSON support?
• Try to work on it during the workshop

ncm-network

• Not part of the release
• Need to clarify status/future
• Check with Loic the status of the rewrite

Pan Compiler Status and Roadmap

v10.0 released

• Old, deprecated features are all removed
  • Session-related features removed (used only at CERN)
• Streamline, consistent options across CLI, Maven and Ant tasks
• Significant parts in clojure
• Speed: 13% in real time but 11% faster in CPU...
  • Probably unoptimal handling of disk I/O: to be investigated
• Not a drop-in replacement for 9.x due to the option changes
  • Need to ocordinate with a new release of SCDB (changes in quattor.build.xml) for SCDB users

v9.2 and 9.3 in maintenance mode

• Only serious bugs will be fixed

v10 roadmap

• Performance improvements
• Internal simplification for maintenance
• Flag for turning off escaping: if turned off, escape()/unescape() will be there but do nothing
  • Coordinate with CCM

Discussion

• Other wishes
  • Rename nlist into dict: will be first done as an alias
  • if_exists for files to be used with file_contents
  • Print template name as part of debug function
• Looking at possibilities for compile
  • Cal interested to see new people working on the compiler... but require a minimum amount of time to get used to the compiler

ncm-spma v3

Part of Quattor release since 13.02 but one important bug fixed in 13.03

• Do not upgrade if you are not happy with the concept presented!
• In production at U-Gent: happy with it
  • both SL5 and SL6
• Tests started at RAL and GRIF

Preserved features

• Ability to fix versions
• All changes in one transaction
• Use of proxies possible
  • Need to document recommended setting, in particular for repo metadata

Last AII version only capable of using YUM

SL5 support ready, including initial installation

• Requires a YUM upgade: RPMs availabe at U-Gent

If not caring about a specific version, a package in the configuration can be just an empty nlist()

Before upgrading

• Refactor the repositories to be sure they mirror the upstream repos
  • Do not throw random RPMs in upstream mirrors else risk of conflict
• Use separate repo for local RPMs
• Use one repo for each Quattor version

Repository hell/nightmare

• RPMfore is evil: many conflicting RPMs with SL ones
• Same for JPackage
• Use include_pkgs in the repository definition to restrict the packages that are considered in them

Conflicts hell

• YUM's obsoletes causing a lot of conflicts if there is an explicit RPM in the profile that is obsoleted.
  • By default ncm-spma disable YUM obsoletes
  • Exemple: perl-Compress-Lib required by ccm
• Real fix/cleanup is to remove explicit specification of dependencies that are not controlled by Quattor
• Another issue arises if a package is not specified with a specific version but one of its subpackage is
  • Eg. python without an explicit version but python-libs with one: clean up pan description

Repo priority hell

• If there is an explicit version required which is in a repo with a lower priority than another repo containing another version, YUM will not install the package and SPMA will fails
• Advice: do not use priority except for very specific cases

Snapshotting the repos is mandatory to avoid unexpected updates

• Moving from one snapshot to the other can be done either at the file system level (symlinks) or in the template defining the URL for the repo
  • URL can be build from a snapshod_id using the format() function

Future directions

• Remove backward compatibility (repository resolution) to improve compile time (30% saving)
• Allow for RPM signatures
• Support for the YUM fs-snapshot plugin: ability for YUM to do a snapshots of main filesystems before making any modification, allowing easier rollback
  • No experience yet

SCDB

• needs to reenable the recompile of profile if there is a change in a repo template
  • Change in the URL as part of the template handling, change in include_pkgs
• Improve update.rep.templates to handle properly include_pkgs

Discussion

• Rollback: doesn't guarantee a roll back to the exact previous state as only explicit versions will be rollback
• Not clear that when there is a set of explicit package (eg. main package, -lib, -devel...), only specifying the version for one of them is sufficient
• /etc/spma.conf is not used anymore. Only userpkgs is still taken into account by ncm-spma if defined in the profile
• New quattor releases can be used with the old ncm-spma

Aquilon

Agreement received from MS legal department to release it under an Apache2 license

• Need to figure out how to integrate features MS specifics as separate part

Seems that MS should be able to contribute directly to GitHub in the future but not immediately

Working on entitlement: reduce entitlement to change some particular parameters

• Declaration of parameters than can be changed from outside: will not protect against misusage/mistake of people with direct access to the templates

Integration with Active Directory Kerberos: working at RAL

• Just the matter of creating the keytab from AD

Test instances outside MS: RAL and U-Gent

Start an aquilon book.

• Cal agrees to setup a vanilla book based on pan docs
  • Docbook (XML): requires an appropriate editor, one free exists
• Where to put it as there is no file service in GitHub
  • Check if we could put it in stratuslab-srv1 and try to automate generation from Maven?
  • Keep using Source Forge?
• Any possibility to generate it in markdown format?
  • According to Luis a plugin should exist

Packaging: handled by Luis, hope to have an RPM ready by the summer

Web interface (from appliance): works at RAL (with a few mods)

• Add it to the Aquilon packages as an optional component