15th Quattor Workshop - Bordeaux - 6-8/3/2013

Table of Contents

1. Update since last workshop
2. Pan Compiler Status and Roadmap
3. ncm-spma v3
4. Aquilon
5. Solaris Port
6. Network Configuration Management
7. Development Process
   1. GitHub Migration
   2. Release Workflow
   3. Development Process
   4. Quattor Referecing in Social Networks
8. Actions
9. Conclusions

Update since last workshop

2 releases made: who use them?

• Probably no site using them yet...

New ncm-spma is production ready: need to be tested

panc v10 released.

QWG

• grid templates: continued support, up-to-date with last MW versions
  • Mostly done by Guillaume and Jerome
  • Migration to GitHub still to be done
• StratusLab templates: available for new versions of the MW but insufficiently sync with GitHub
  • Discuss with Guillaume (main maintainer) a more regular contribution to GitHub: use branches if necessary

Aquilon

• Packaging still in progress: 2 repositories, mainly done at UGent and RAL
• RAL is already using it to manage the cloud machines and some other test machines
  • Still trying to figure out how to migrate the QWG machine types and database related things

Dashboard

• Still working on it: indexing added
  • Requires Python 2.7
• datawharehouse repo on GitHub

Solaris port working at MS

• Solaris 11
• Based on a fork made 1 year ago
  • Main changes are in network config and cron
  • No package management yet: everything installed from AFS before running components
  • Would like to merge back to the mainstream Quattor

QRD/QRC

• New QRD available and used at MS. Hope to publish it in one month.
  • VMWare (MS specific) and NetApp
  • Python
• Would like to do the NetApp management through QRD this year
  • Currently, only used at initial installation
• Also plans for switches

Quattor FS

• A few bug fixes and enhancements

JSON profiles

• Smaller, no need to escape

SCDB

• Pb with last scdb-ant-utils: changes related to JSON support?
• Try to work on it during the workshop

ncm-network

• Not part of the release
• Need to clarify status/future
• Check with Loic the status of the rewrite

Pan Compiler Status and Roadmap

v10.0 released

• Old, deprecated features are all removed
  • Session-related features removed (used only at CERN)
• Streamline, consistent options across CLI, Maven and Ant tasks
• Significant parts in clojure
• Speed: 13% in real time but 11% faster in CPU...
  • Probably unoptimal handling of disk I/O: to be investigated
• Not a drop-in replacement for 9.x due to the option changes
  • Need to ocordinate with a new release of SCDB (changes in quattor.build.xml) for SCDB users

v9.2 and 9.3 in maintenance mode

• Only serious bugs will be fixed

v10 roadmap

• Performance improvements
• Internal simplification for maintenance
• Flag for turning off escaping: if turned off, escape()/unescape() will be there but do nothing
  • Coordinate with CCM

Discussion

• Other wishes
  • Rename nlist into dict: will be first done as an alias
  • if_exists for files to be used with file_contents
  • Print template name as part of debug function
• Looking at possibilities for compile
  • Cal interested to see new people working on the compiler... but require a minimum amount of time to get used to the compiler

ncm-spma v3

Part of Quattor release since 13.02 but one important bug fixed in 13.03

• Do not upgrade if you are not happy with the concept presented!
• In production at U-Gent: happy with it
  • both SL5 and SL6
• Tests started at RAL and GRIF

Preserved features

• Ability to fix versions
• All changes in one transaction
• Use of proxies possible
  • Need to document recommended setting, in particular for repo metadata

Last AII version only capable of using YUM

SL5 support ready, including initial installation

• Requires a YUM upgade: RPMs availabe at U-Gent

If not caring about a specific version, a package in the configuration can be just an empty nlist()

Before upgrading

• Refactor the repositories to be sure they mirror the upstream repos
  • Do not throw random RPMs in upstream mirrors else risk of conflict
• Use separate repo for local RPMs
• Use one repo for each Quattor version

Repository hell/nightmare

• RPMfore is evil: many conflicting RPMs with SL ones
• Same for JPackage
• Use include_pkgs in the repository definition to restrict the packages that are considered in them

Conflicts hell

• YUM's obsoletes causing a lot of conflicts if there is an explicit RPM in the profile that is obsoleted.
  • By default ncm-spma disable YUM obsoletes
  • Exemple: perl-Compress-Lib required by ccm
• Real fix/cleanup is to remove explicit specification of dependencies that are not controlled by Quattor
• Another issue arises if a package is not specified with a specific version but one of its subpackage is
  • Eg. python without an explicit version but python-libs with one: clean up pan description

Repo priority hell

• If there is an explicit version required which is in a repo with a lower priority than another repo containing another version, YUM will not install the package and SPMA will fails
• Advice: do not use priority except for very specific cases

Snapshotting the repos is mandatory to avoid unexpected updates

• Moving from one snapshot to the other can be done either at the file system level (symlinks) or in the template defining the URL for the repo
  • URL can be build from a snapshod_id using the format() function

Future directions

• Remove backward compatibility (repository resolution) to improve compile time (30% saving)
• Allow for RPM signatures
• Support for the YUM fs-snapshot plugin: ability for YUM to do a snapshots of main filesystems before making any modification, allowing easier rollback
  • No experience yet

SCDB

• needs to reenable the recompile of profile if there is a change in a repo template
  • Change in the URL as part of the template handling, change in include_pkgs
• Improve update.rep.templates to handle properly include_pkgs

Discussion

• Rollback: doesn't guarantee a roll back to the exact previous state as only explicit versions will be rollback
• Not clear that when there is a set of explicit package (eg. main package, -lib, -devel...), only specifying the version for one of them is sufficient
• /etc/spma.conf is not used anymore. Only userpkgs is still taken into account by ncm-spma if defined in the profile
• New quattor releases can be used with the old ncm-spma

Aquilon

Agreement received from MS legal department to release it under an Apache2 license

• Need to figure out how to integrate features MS specifics as separate part

Seems that MS should be able to contribute directly to GitHub in the future but not immediately

Working on entitlement: reduce entitlement to change some particular parameters

• Declaration of parameters than can be changed from outside: will not protect against misusage/mistake of people with direct access to the templates

Integration with Active Directory Kerberos: working at RAL

• Just the matter of creating the keytab from AD

Test instances outside MS: RAL and U-Gent

Start an aquilon book.

• Cal agrees to setup a vanilla book based on pan docs
  • Docbook (XML): requires an appropriate editor, one free exists
• Where to put it as there is no file service in GitHub
  • Check if we could put it in stratuslab-srv1 and try to automate generation from Maven?
  • Keep using Source Forge?
• Any possibility to generate it in markdown format?
  • According to Luis a plugin should exist

Packaging: handled by Luis, hope to have an RPM ready by the summer

Web interface (from appliance): works at RAL (with a few mods)

• Add it to the Aquilon packages as an optional component

Solaris Port

Revived by MS.

Profile schema is the same as for Linux/SL.

ccm, ncm-ncd, ncm-cdisp unmodified

• Potentially a problem with Kerberos handling in CCM

A few components incompatible as Solaris implementation is very different

• package management, network, cron...
• package management ignored for the time being
• How to handle them: just consider they will be fetched from different repositories or attempt to namespace them

Some components may just needs minor variants ue to different paths

• In this case, try to abstract paths from the component: retrieve them from some place
• Also the comand to restart daemons is pretty different
• Need to rediscuss how to properly implement this abstraction without creating a complex thing

Check if license allows to make an image available for testing Quattor Solaris.

• This would allow integration testing as part of the Quattor release process

Timeline: hopefully something ready for production by next workshop

• Already something working at MS but need to merge back the changes as they were made on Quattor 1 year ago.

Currently, not a lot of interest for the Solaris port at sites represented at the workshop...

Network Configuration Management

No progress since Budapest discussion one yar ago.

Next generation ncm-network goal: interface and routing configuration

• Basically what ip command does: ip layer configuration

Advanced features like MTU, TCP params should be in another components?

• Risk of conflict managing the same files?
• Be sure not to manage the same param in 2 components
• bonding is something that should probably be moved to a component dedicated to interface management

Minimal requirements for new ncm-network initial version

• Configuring IP address, basic routing information on all interfaces present in the machine
• Not necessarily all config parameters made persistent but at least all those that are necessary to have the network working and all those parameters that require a network restart

Investigate the possibility to inteface with NetworkManager daemon to do the network configuration.

• Check features available in the SL5 version or ability to upgrade it

Schema: review the current description under /system

• Making it component specific if we split network configuration between several components
• Iterate on the schema redesign to see how to split without duplication the configuration description into different subsets, eg. interfaces vs. IP

Actions

• Evaluate NetworkManager and its usability by Quattor
  • Dimitris
• Evaluate state of current rewrite and if it is possible to use it as a basis for next generation component
  • Nathan
• Schema discussions/improvements
  • First decide on what are the future components for network configuration

Development Process

GitHub Migration

Most Quattor components already migrated. Main missing parts are SCDB and QWG templates

• Aquilon: a clone of the SF repository because of MS restrictions to access GitHub
  • Still willing to move to GitHub as soon as it is possible
  • New Aquilon book decided: make it a subdirectory of the Aquilon repo
• spma and rpmt-py still here but not built by Jenkins
• SCDB and QWG in progress: what is in GitHub is only test repos
  • QWG repositories: renamed to something like template-library- . Suggestions welcome!
• Also missing QuattorFS
  • Currently in SF SVN
• SVN still enabled at SF: turn it off?
  • Seems painful to disable only write access: seems to require modifying entry for each user
• Repositories: make name as descriptive as possible, ensure a description is present
  • Check pb with QWG repositories where the description doesn't show up
  • Rename ncm-components-xxx into configuration-modules-xxx

Configuration module status

• ncm-network not migrated because of the rewrite in progress
• ncm-openldap in a bad shape but probably used by StratusLab
  • Cal will have a look

Almost every GitHub repository has a matching Jenkins job

• Several components with a good coverage in terms of unit testing: accessible from GitHub main page from the repository ('Passing' button)
• GitHub received a pull request after each commit in a Quattor Git repo

Component testing using prove

• No config file required, except to produce XML reports
• Every .t file present in component sources (under src/test) is executed

Git branches: prefer private forks rather than branches in main repositories

• Easily tracked if forks also hosted on GitHub through pull requests
  • A command line tool hub available to avoid going through the web interfaces
  • Specific comments possible for pull requests
• Changes pulled as one commit using a rebase
• Just require adding a remote in a personal repo to track several related repositories

Main pom file and build tools in a separate repository.

• Documentaton on how to make a new version of the pom file
• Pushing them to the central repository requires specific rights: only Cal and Luis

Release Workflow

Released pushed to YUM repo

• One repo per release
• Have one symlink pointing to the last release: to be used by sites who want to upgrade automatically to the last release

Manual step involved in making a release: create the release file

• Requires Jenkins to be green
• Currently only running the unit/functional tests but no test deployment
  • discuss the possibility to deploy VMs on StratusLab cloud: checks how to automate analysis of the installation/upgrade result
  • Keep AII as a separate thing in a first stage

Release numbering: everything renumbered according to the release number whether there was a component change or not

• Try to see if it is possible to generate a ChangeLog that will contain the list of commits since the last version
• After a release was made, maven will generate .1-SNAPSHOT

Non RPM files: continue to use SF as there is no file service in GitHub

• Must be done manually
• Improve the link documentation on quattor.org
  • For documentation, include html (or markdown) version into the quattor.org site

QWG: streamline the process of updating everything not specific to QWG to the last release

• Configuration modules
• AII
• Standard Quattor and pan templates
• Also involves some systematic testing

Development Process

Weekly meetings

• Almost not happened since 1 year
• Certainly useful to help making progress on main actions like the one discussed during this workshop
  • Not a complete consensus that a general meeting rather than focused meetings is really needed
• Time: would be good to allow MS Montreal people participation
  • Suggestion: Thursday 5pm CET
• Try to stay with Vidyo but ask CERN about the status of the "donald duck" issue affecting Luis
  • May also ask if there is a beta of the Skype bridge we could test

Next releases planned

• March: next week probably
• May

Better advertize releases.

Quattor Referecing in Social Networks

Ohloh: updated some months ago. To be checked.

Twitter: not much info pushed

• Not clear the added value for Quattor users
• Main goal: show the project is alive

Google+: some documents put by Luis

Actions

Pan compiler (Cal)

• nlist renaming into dict
• if_exists for files to be used with file_contents
• Print template name as part of debug function
• Optimize v10.0 performances to match v9.3

SCDB (Michel)

• New release(s) with JSON support and panc 10.0 new options
  • JSON support requires a new version of scdb-ant-utils (NotifyClientTask)
• Support for YUM
  • Documnent how to easily define if a repository template change must trigger a profile rebuild (pan.dep.ignore in quattor.build.xml?)
  • Correct handling of repository propery include_pkgs in upate.rep.templates

YUM-based ncm-spma (v3)

• More testing, cleanup of QWG repositories (see repository hell) (GRIF)

Aquilon

• Complete packaging (RAL)
  • Add appliance web interface as an optional component/package
• Start an Aquilon book (Nick/Cal/James/Dimitris)

Network configuration management (aka ncm-network rewrite)

• Evaluate NetworkManager and its usability by Quattor (Dimitris)
• Evaluate state of current rewrite and if it is possible to use it as a basis for next generation component (Nathan)
• Schema discussions/improvements
  • First decide on what are the future components for network configuration

Configuration modules (miscalleneous)

• Check ncm-openldap status: in particular check StratusLab modifications, if any

Quattor development process and tools

• Weekly meetings moved to Thursday 5 pm CET
  • Resume really weekly meetings...
• Shut down SVN access on SF
• Complete SCDB and QWG templates migration (Michel)
  • Rename QWG templates into template libraries
• Produce a ChangeLog in each RPM with the new commits part of the release (Luis)
• Streamline the process to update core components in template libraries

Conclusions

Next workshop: Bologna