Quattor Workshop - Budapest - 20/3/2012

Table of Contents

1. Community News
2. PAN Compiler
   1. Annotation Support
3. CCM
4. AII
5. Network Configuration
   1. ncm-network

​Agenda

Community web, development toos and processes

• Web landng page : move to Git
• IRC
• Twitter
• Quattor releases

Components

• Configuration status report
• Delayed execution
• Support for removal of escaping in Pan
  • If a flag is present on the profile, CCM could just do nothing when unescape is called

Community News

CERN looking at Puppet for their new infrastructure and possibly as a Quattor replacement on the future

• Report at a future Quattor Workshop

DESY still using Quattor, may try to reestablish contact

SLAC looking at Quattor and requiring some help.

• CNAF has contact

PAN Compiler

v8 frozen: 8.4.7 is the last version released

• No more support
• If still using, upgrade to v9!

v9 first versions

• Last released version is 9.1
  • RC1, RC2, RC3 released
  • 9.1 is RC3 + Windows-related fixes
• v8 deprecated features removed
• "panc nutshell book"
• "root" element for build info
  • May be an interesting feature to implement "profile cloning"
• various bug fixes

v9.2 recently released

• plan for removal of escaping: deprecate XMLDB, change tests to use pan xml
  • Enabled by a switch, the only option in v10
  • Profile may contain a flag saying if escaping has been used or not
  • Impact on SCDB
  • Componenent need update to be prepared to handle unescaped keys (values)
• Expanded maven support
  • Currently archetype, simple build
  • Future: Maven has a Pan interface, in addition to ant
• Support for JSON output format
  • Will enable transparent gziping almost for free in term of performance during compilation or even a positive one when IO is the limiting factor (eg. laptops)
  • Need to document how to enable gzip in downstream components
• Skeleton for updated panc command in clojure
  • More consistency in the options, simplified/cleaner code for processing options in the compiler

v9 series roeadmap

• Streamlined, simplified code with limited changes to functionality
• Gradual migration to clojure: CLI, template compilation, functions, output formats...
  • Can be done incrementally
• Limited inclusion of other libraries (e.g. Clojure, JSON)
  • Until now panc has been self contained
  • All included components have an Apache 2 (compatible) license

New requests from discussion

• Range starting with a negative number (Luis): may lead to some grammar ambiguity not trivial to fix
• panc as a service (Gabor): idea is to have a continuously running service to avoid the starting time cost when compiling only one profile
• Functions to convert IP addresses (v4 and v6) to long integer (and vice versa) to help with netmasking and other similar things (Gabor)
• Readd debugging information during dependency calculation to help identify the reason a dependency is not considered up to date (Michel)
  • Disappeared during v9 code refactoring
• Add an info() function in addition to the warning() one to avoid abuse of warning() (Gabor)
• function or pragma to activate debugging in a given template as an alternative to command line switches (Gabor)
  • Easier to use for people who are not Pan experts
  • May be great to say debug this template and all included templates...
  • Not easy to implement as compilation occurs before execution and debug code is removed in the debug flag is not set for the template: possible performance impact if checked during execution
• In include, support a list of template rather than just one (Ronald)
• Rename nlist to hash
  • Keep both as a function and deprecate nlist at some point
  • Add a section to the manual explaining the Pan vocabulary and how it matches the vocabulary in other languages

Discussion but no agreement on how to allow structure templates with rights to modify other part of the configuration than the one they are assign to or to pass parameters to normal templates...

• Several goals: make things cleaner than using global variables (variable scope), make easier to non expert to use pan, entitlement (restrict a user to use one specific template to achieve a specific configuration action)...

Annotation Support

2 possible approaches for building the output files produced by annotation processing

• Hierarchy based on input templates hierarchy: related to layout of input files
  • Makes input options more complex as some specific options are required just for annotation
• Namespace : conflict between similar namespace in different template sets
  • Requires a wrapper to handle possible conflicts.

Decision : move annotation processing outside of panc compiler to a separate tool

• Will reuse the compilation module of panc
• Makes having distinct options easier: keep the current behaviour
• Will put less constraint on further extensions of the tool

CCM

Problems

• Scalability because of the backend storage format: mainly MS and Gent
• Difficult to support new backend format like JSON
• Supports only http URL: makes testing difficult
• Remove of unescape() function

MS contributions in the last years

• CDB_File format x3 faster
  • Require an additional perl module: perl-CDB-File
• Kerberos support for encryption

Recent work

• JSON support: quite advanced, able to run components from the JSON file
  • Still need to test/Validate the results
• Support of new protocols for downloading profiles: ​file://, ​ftp://, ​ssh://
  • But the underlying modules may not support all of them

Status

• Rewritten modules have a large number of unit tests
• Introduces new dependencies

JSON support issues

• Some data types are lost: impossible to distinguish long and double
  • And Perl doesn't distinguish them internally from string...: means that you cannot know if the information was a string or a number
  • Pb doesn't exist with language distinguishing thems
• No way to ask JSON about the type of a data
• In fact getTree() already loose this information

unescape() removal

• Will check the flag saying that the profile has been generated without escaping
• IF the flag is set, will do nothing: just return the argument unmodified
  • No modification required to components, except if they use their private version of unescape()

Requests:

• Have distinct exit code in ccm-fetch for failures related to network errors (Gabor)
  • May allow to switch between different profile sources

AII

Known limitations

• SL6 issues
• LVM and reinstallation
• Large installation disk support
• Scalability: lack of large installation

MS additions/improvemnents

• Profile caching: Profiles are cached for future reuse for better performances, using CDB_FIle format for improved performances
• Fine-grained locking: per-profile instead of globally
  • Allow to run aii-shellfe in for several nodes
  • Allow monitoring of stalled installation by looking at lock timestamp
• Partition alignment based on some provided information
  • Not really working with SL4
• Parallel installation support: only partially merged as it requires PRocess::Parallel which is MS-specific
  • Some alternative may be found
• New options: -firmware and -livecd to boot alternate images in a way similar to rescue
  • Target of these options defined in node profile

*Full RHEL6/SL6 support

• Schema made more flexible to add/remove mandatory or deprecated options
• GPT used for partition tables, using parted
  • Anaconda doesn't die with large disks, much faster
• Built with Maven tools

New issues

• Code quality is becoming a problem: many contributors, few tests, spaghetti code
  • May require a significant rewrite with unit tests
• Kickstart file more and more difficult to read....
  • Debugging of Kickstart %pre and %post scripts can be very difficult...
  • May redirect everything to a file in /tmp and ensure all commands are logged

Pending issues/requests

• Implement deferred reboot during installation based on a profile property: needed by MS
• DHCP plugin by MS: would be great to have it release to the community
  • Replacement for aii-dhcp able to do more things looking at the whole profile information (like adding info required by Windows machine)
• MS saw problems with machines where boot disk is renamed after some events (change) on the machine or OS upgrade
  • Would like to think at a mechanism to give an abstract name to disks and do late binding but difficult to implement into Kickstart

MS Stanley still considering reviving the Solaris port, including installation port.

Network Configuration

ncm-network

Among the problems

• Network restart during reconfiguration: potentially a problem on diskless systems
• Code very difficult to maintain

Rewrite requirements

• Separate each part of the configuration: interface, routing...
• Reconfiguration without network restart
• Support all current features
• Add all the features that RH scripts support
• Easy path from old to new component if possible
• Use ip command rather than ifconfigand route

Routing configuration: rely on ip route rather than ADRESS/NETMASK/GATEWAY

• Convert root definition to a nlist: key is the subnet usual notation xx.xx.xx.xx/nn (nn being the number of bits in the netmaks)
• Policy routing contributed by Gabor
• Multipath routing: done by ip route nexthop

Status

• Main missing part is the routing... Loic expects to have some time to work on it in the coming months...

IPv6 support required: no real reason to manage it in a separate component, need to add it

Network schema

• May require some refactoring: some information are improperly placed under /system/network and we need to support more information interface specific
  • An idea is to move this kind of information into the hardware description part and have the network configuration module looking there for this part of the information, based on the interface name

Other issues

• loopback interface is not configured by network configuration module and is ignored. This doesn't allow to define aliases on it (Gabor)