Quattor Workshop - UAM (Madrid) - 29-30/10/07

Table of Contents

1. Site Reports
   1. CERN
   2. LAL
   3. NIKHEF
   4. GRID-Ireland
   5. PIC
   6. UAM
   7. Morgan-Stanley (Nick Williams)
   8. Philips
   9. CNAF
2. Core Components
   1. PAN Compiler - C. Lommis

Site Reports

CERN

Main instance : ~6100 nodes (+ 1K nodes in 6 months)

• Template "de-optimization" : had to get rid of most value('/xxx/yyy.tpl') for moving to namespaces. Result is perf slower by a factor of 2.
• Still using panc v6 : perfs not satisfactory with panc v7 first tests. No time for extensive test. Current compilation time is 20' on a 2 dual-core system (16 GB of memory).
• CDB : still using a flat namespace, moving slowly to namespaces. Optimizing for multi-core machines...
• SPMA + SWrep : extended authentication added (Krb5), many bug fixes and improvements
• CDB2SQL : no progress
• CCM : authenticated mode added
• Initial installation : still using PrepareInstall instead of AII : interacts with both CDB and CDBSQL to get machine parameters and some CERN specific components (LAN db, SINDES...). Installation done by AIMS. Very successfull but a little bit hugly to maintain.

Quattor development activities

• Release management related projets (ETICS integration, Savanah...) on hold
• Default template set has been moved to namespace and build tools updated
• Maintenance of a few core modules : CDB2SQL, SPMA, SWrep, CDB...

Other activities :

• Xen-based virtualization based on ncm-xen and ncm-filesystems : development of a Xen Hypervisor.
  • Integration between Quattor and SLS in progress

LAL

See ​http://indico.cern.ch/materialDisplay.py?contribId=10&sessionId=0&materialId=slides&confId=20479

About using Squid to cache RPM at every site : Grid-Ireland experience is that they are some drawbacks if you have something outdate in the cache. Can be difficult to clear all caches. Prefer to use rsync at deployment time to ensure that every site repository is up to date.

NIKHEF

NIKHEF-ELPROD : ~150 nodes, strong increase expected

• Also an installation testbed with 15 nodes

Currently using AII for initial installation and Quattor for configuration of OS and MW

• OS configured with NCM components
• MW configured with ncm-yaim : frequent patches required to YAIM

Quattor components used :

• Moved to panc v7 : faster than v6.
• AII, SPMA
• SCDB without Subversion (using CVS for versionning)... Only Ant tools are used.

Issues :

• PAN compiler performances as a significant increase of number of nodes is expected
• Scaling problems when deploying to 130 nodes simultaneously : try to monitor update results with Ganglia.

Future :

• XEN virtualization
• Nagios setup under Quattor

GRID-Ireland

Quattor managing 18 sites with a total 400 nodes

• Single CVS moving to SVN
• Replicated SW repositories at each site (rsync)
• 3 deployment servers : production, tests, e-Learning

Recent developments :

• 17 sites reinstalled with Quattor/Xen : fully automated PXE installation of hosts and guests
• Migration to SVN : integration in progress with local web deployment tool. Realized the high number of unused, obsolte templates.
• New compute resources : ~140 Condor VMs

QWG usage :

• "Pointer" to particular QWG revision (by rev number), checked out semi-automatically. Plan to pull QWG templates via svn:externals
• Real sites containing clusters, ability to select sites. Local target compile.sites.
• Plan to use Stijn's dummy WN template

Issues :

• Desesperatly need monitoring integrated to detect failed deployments. At least documentation about currently existing solutions.
• int.eu.grid site backed out of Quattor due to conflicts with APT/YAIM
• Getting started with Quattor is still difficult

PIC

Several Quattor servers :

• Quattor01 : still Quattor 1.1, CDB
• Quattor02 : Quattor 1.3, SCDB, QWG glite-3.1
• Quattor03 : will replace Quattor01/02 soon with QWG glite-3.0 and glite-3.1 (mainly WNs and UIs at first).

Local developments :

• ncm-snmp : needed for Nagios monitoring
• Local command to hide SCDB/CDB differences : gettpl, ...

UAM

2 grid clusters : UAM-LCG2 (150 nodes) and GVMUAM-LCG2 (300 nodes)

• Quattor 1.2, SCDB, QWG 3.0.2-x (not the most recent), AII, SWrep
• Still configuring d-Cache manually : considering moving to Quattor

1 non grid cluster still managed using CDB

• Different set of people managing this cluster, used to CDB

Morgan-Stanley (Nick Williams)

Not yet using Quattor, looking at it as one possible solution for Morgan Stanley needs.

Currently using an home made product called Aurora

• Quite good but designed at beginning of 90s
• Defines all distributed services and host configuration : how machine is configured, how to access apps, ...
• Designed with the ability to restore a machine in a previous state in 20 minutes
• Based on an homogeneous view of the ressources, in particular through use of AFS : nothing installed locally

Current configuration is 20K Unix servers, includes "grid systems" (pool of nodes running the same app) with home-gown MW.

• Aurora designed for 5K machines
• Need to integrate risk management, an important part of financial business (failover...)
• Users asking for custom configurations, quicker and more agile.
• Machine organized into bucket : ~2500 machines sharing most of their configuration settings.
• A campus made of multiple buckets, a region made up of datacenters.
• Significant work done at night to synchronize configurations

Core of the new Aurora must be :

• A configuration system
• An entitlements system : control of rights to do an action
• Move off AFS if possible : RPM ?
• Virtualization
• Work done by Unix Engineering group (20 people, based in London)
• Support for Linux and Solaris

Philips

Restarting work on Quattor : new persons found.

Current effort is to transfer Quattor knowledge to system management to get more people involved.

• Quattor 1.3, SCDB, QWG templates 3.1

Issues :

• How to revert to a previous tag
• What actions to perform when new QWG templates are out

CNAF

Still using Quattor 1.2 withouth SCDB/QWG : lack of time to look at it.

• SLC 4 support has been added, mainly 32-bit
• Quattor 1.3 update still planned, will use namespace at the same time (lot of CNAF specific templates to update)
• Using last AII version : no problem so far...

Evaluating Xen : some issues using pypxeboot

Training of CNAF staff still a problem : Quattor often considered as the responsible for deleting what was done manually...

• Very time consuming

Candidate for hosting next Quattor workshop...

Core Components

PAN Compiler - C. Lommis

Current status :

• v6 deprecated and frozen : no bug fixes or enhancements. Still used by default by CDB.
• v7 is the production version (7.2.6) : (almost) 100% backward compatible, very limited enhancements. Used by default by SCDB.
• v8 : development version. First version with language-level changes compared to v6. Expect a production version before Christmas

v7 performance :

• Faster than v6 for almost everybody, except CERN. Rudimentary profiling added to v7 to help investigate the problem.
• Better memory management : can make huge differences with large number of templates
• Generic performance tests done :
  • v7 significantly faster than v6 (x5-10) for almost every operation, except variable self reference test where v7.2.5 was 50x slower than v6. Fixed in 7.2.6 where perf is comparable with v6.
  • Many operation slower on multi-core than on single-core : requires some investigation.
  • Perf on CERN profiles is better with panc v7.2.6 than with v6 (x1.5 to x2). Memory use is around 2 GB for 920 profiles (probably much less than panc v6 but difficult to get exact consumption for v6).
• Possibility to explicitly set the number of thread to use but all tests showed the best performance is obtained with the number of thread matching the number of cores (default behaviour).

Planned language changes :

• v7 : foreach statement, bind statement (replacement for one form of type), bit and misc. functions, some deprecated keyworkds
• v8 :
  • Enhancements : format function, i18n support, perf and include logging, automatic variable for current template name, simple DML and copy-on-write optimization (only copy changed parts), finalfor structure templates.
  • Deprecated : lowercase automatic variables, type synonym for bind.
  • Unsupported : define, description and descro keyword, delete statement
• v9 : deprecated include with literal name (important simplification of grammar), removal of lowercase automatic variables and type as a synonym for bind
• v10 : removal of include with literal name
• v11 : include DML without braces

Other requests :

• Cast operator : bring lot of complications. Delayed for now.
• Forced assignment : change the type of variable. Risk of undetected errors. Delayed for now.

Roadmap discussion :

• Deprecate include with literal name in v8 : the change required is compatible with panc v6. Agreed.
• Introduce include DML without braces at the same time former include syntax is unsupported (v9).