Changes between Version 2 and Version 3 of Meetings/Workshops/20061018

Timestamp:

Oct 21, 2006, 11:37:51 PM (18 years ago)

Author:

/C=FR/O=CNRS/OU=UMR8607/CN=Michel Jouvin/emailAddress=jouvin@…

Comment:

--

Meetings/Workshops/20061018

@@ -1 @@
-                result['accounts']['users'][role_user] = merge(nlist('uid', vo_params[vo]['base_uid']+role_num,
-                                                                     'groups', list(vo_group),
-                                                                     'comment', 'VO '+vo_name+' '+role['description'],
-                                                                     'createKeys', vo_params[vo]['create_keys'], 
-                                                                     'createHome', vo_params[vo]['create_home'],
-                                                                     'poolDigits', vo_params[vo]['pool_digits'],
-                                                                     ),
-                                                                role_user_home);
+= Quattor Worshop - DESY - 18-20/10/06 =
+[[TracNav]]
+
+[[TOC(inline)]]
+
+Agenda : https://indico.desy.de/conferenceTimeTable.py?confId=64&showDate=all&showSession=all&detailLevel=contribution&viewMode=parallel
+
+== Site Reports ==
+
+=== CERN ===
+
+Mainly increase in the number of nodes managed by Quattor : #4000
+
+Quattor Solaris support dropped
+
+CEN specific activities :
+ * CDB profile authentication/encryption : not yet deployed, need to solve the issue of a host cert expiring or in a CRL to avoid dead lock. Idea is to use 2 URLs for downloading the config, with an automatic failover it first one fails.
+ * Manage Xen and VirtualPC with Quattor
+ * Namespaces : urgently need, want to agree on namespace layout before. Will take time to implement in 20K templates
+ * test, new and pro area to be provided using loadpaths and ACLs
+ * Integration of Quattor with SL (Service Level Status)
+ * Impact of SL5 (using FC5/6) on ncm-components
+ * SINDES maintenance : not really part of Quattor but mainly used in this context.
+
+
+=== DESY ===
+
+200+ systems in production in the GRID infrastructure
+ * Local HERA experiments
+ * CMS and Atlas (DESY T2)
+ * VOs that are part of EGGE
+
+Still using Quattor 1.1
+ * CDB, AII, SWREP, SPMA
+ * Still interest for SCDB but no time yet...
+ * Middleware installation is done using YAIM (yaimexec ?)
+
+Current issues :
+ * How to keep CA and middleware RPMs up to date ? How to use YAIM and SPMA in conjunction ?
+ * PAN compile time increasing with the number of machines (currently 5-6 minutes for 200 nodes)
+ * Time to get fixes for YAIM bugs reported in Savanah
+ * Infrequent cdispd aborts : seems to be known at CERN, related to CCM
+
+
+=== BEGrid ===
+
+Central SCDB + SWrep
+ * SCDB : Certs+ACLs. Not everybody allowed to edit everything.
+ * Goal :
+
+Current configuration :
+ * 4 sites with LCG, 2 with gLite
+ * Restarted from scratch (new repository) with gLite
+
+Additions to QWG templates :
+ * SE_dCache
+ * Lemon server with Oracle
+ * Use of IPMI for /system/hardware
+ * Ganglia server and client
+ * All pwd and sensitive information into one template
+
+Problems using SLC with Quattor/QWG
+
+Tests done with WNs in VMware under WXP, managed by Quattor.
+
+Changes to AII :
+ * SINDES/AII integration : some templates needed to be changed
+ * Install kernel in ks rather than ks-post-install : easier to use alternative kernels
+
+Work on improving bulk compile of a large number of worker nodes
+ * Compile a dummy WN
+ * In real WNs, include the compiled profile and redo part of the configuration
+ * Doesn't fit well with QWG templates, difficult to say wich part to reinclude and which part to ignore, probably need something inside PANC
+
+
+=== CNAF ===
+
+Early adopter of Quattor : both experienced QWG templates and YAIM
+
+Currently using Quattor for initial installation (AII)
+ * Use http repositories instead of SWrep
+ * Since LCG 2.7, moved to YAIM with ncm-yaim
+
+Pretty well accepted by farm managers
+ * Especially storage guys
+
+Started to implement our own namespaces
+ * Mainly to achieve machine category segmentation, in particular to control access to templates.
+ * Would like some discussion on use of namespaces in standard templates
+
+Would be useful to have all templates needed to install a basic SL system.
+
+More of documentation on basic Quattor components (PAN, LC libs...) would help dissemination.
+
+
+=== NIKHEF ===
+
+T1 for LHC and involved in several national projects (BIG GRID, VL-e
+
+2 "sites" :
+ * Production : ~180 nodes. Significant increase expected.
+ * Installation testbed : ~15 nodes
+
+Quattor usage : CVS + ant/panc
+ * OS : CentOS 3
+ * panc 5.0.6
+ * Only generic components used
+
+gLite : moved to ncm-yaim
+ * Initial installation via Quattor
+
+Issues :
+ * 64-bit SW installation
+ * Compiler performance
+
+
+=== Philips Eindhoven ===
+
+Part of research division uses grid
+ * A few test systems installed with Quattor
+ * Links with NIKHEF
+
+
+=== PIC ===
+
+250 nodes running SL3 + 2 CASTOR nodes
+
+Use ncm-yaim.
+ * Problem with the hardcoded list of variables that can be configured with ncm-yaim. Made some changes, not filled into Savanah. Pb fixed 6 months ago by Ronald.
+
+Deploying Quattor 1.2 + SCDB
+
+
+=== Irish Grid ===
+
+Entire irish grid managed with Quattor
+ * 18 sites, 200 nodes : all the nodes centrally managed from 1 site (Dublin)
+ * 1 Quattor database
+ * CVS SCDB, HTTP RPMs
+ * 95% of nodes are Xen VMs
+ * Had our own hierchical model : EGEE->GI->Site. Started to move to standard QWG layout/hierarchy soon with gLite 3.0.2.
+ * Expertise is spreading throughout the group
+
+Moving non Grid servers to Quattor : 64-bit, SL4, Xen...
+
+Integration with automatic VM creation tool for building testbeds.
+
+Have spent a lot of time keeping up to date with changes in QWG structure.
+
+
+=== UAM ===
+
+Quattor used for installation + configuration of 3 clusters, using 3 Quattor servers.
+ * Use CDB with last QWG templates
+
+Cluster UAM-LCG2
+ * Part of a distributed T2
+ * 130 WNs
+ * QWG LCG templates
+ * Issue with update snc with other sites
+
+Cluster GVMUAM-LCG2
+ * 500 nodes, mainly PCs for student lectures
+ * Used for different topic
+ * Installed with QWG templates
+ * Must preserve existing partitions
+ * No full control of DHCP
+ * Network pretty slow
+
+Cluster WS
+ * User desktops : #40
+ * Template layout based on organization : departement, group...
+ * Home made templates
+ * Software for desktops
+ * Components to configure desktop services like printers, X11
+
+
+== Experience with ncm-yaim - R. Starink ==
+
+QWG templates until 2.6.0 : several difficulties, mainly due to the lack of genericity
+ * Took 4-8 weeks to incorporate local changes to a new QWG release
+ * Backward compatibility between release
+ * Complex structure
+
+Move to YAIM with ncm-yaim with LCG 2.7
+ * YAIM used only for config, install with SPMA
+ * YAIM variables created from templates
+ * Activate YAIM on each machine
+
+Setting YAIM variables into templates very similar to writing a pro_lcg2_config_site.tpl
+ * Issue with new version of YAIM requiring variables not supported by ncm-yaim
+ * Explicit list of supported variables comes from the ncm-yaim schema that bring the advantage of validation (instead of using a plain filecopy).
+
+Building RPMs list. Several solutions attempted :
+ * Dependencies from gLite meta packages
+ * From an APT repository
+
+Need for YAIM local functions required :
+ * Security : no shared pool accounts for SGM users
+ * Central db for DPM and LFC
+ * Shared gridmapdir, shared home dirs, accounts on LDAP : not supported by YAIM
+ * Developped nikhef-yaim-local to be installed (via SPMA) or updated after gLite-yaim. Close contacts with YAIM developpers.
+
+Satisfied by the change :
+ * Easier to maintain, less overhead, less dependent on external party (QWG)
+ * Still some surprise with YAIM...
+ * Shorter time for deployment of a new release (1 week)
+ * No experience in changing a node type without reinstalling
+
+
+== RPM Dependency Hell - Stijn Weirdt ==
+
+RPM management by Quattor involves several steps :
+ * Create/update repository with tools in cvs/utils
+ * Create base repository contents with cvs/utils
+ * Keep repository up to date
+ * Test deployment : nothing to help here
+
+Other tools existing :
+ * apt : no bi-arch support
+ * yum : the best presently, not supporting all RPM options
+ * smart : try to support everything, still buggy
+ * All these tools share in common the idea of RPMs metadata stored in some db that can be accessed without accessing RPMs.
+
+SPMA : works well but some limitation : in particular inability to test deployement without installing RPMs as there is no metadata.
+
+RPM repositories :
+ * SWrep : should have a "rsync url" option to keep local repository up to date.
+ * OS distros : could use existing mirrors to avoid duplication locally. Would require some kind of metadata to produce the local templates required.
+ * Initial loading of OS templates : may rely on comps.xml to find what is needed or on some metadata.
+ * Keeping repository up to date : OS update metadata parsing ? 
+
+RPM testing before deployment : test all the dependency from the administrator machine before deployment
+ * Should be very fast (30-60s per machine) : require metadata
+ * Fake installation not fast enough
+ * Injection of all RPMs in a rpmdb + rpm -i
+ * Problems with first tests on 64-bit : yum fails for unknown reasons, rpmdb doesn't support correctly bi-arch.
+
+
+== Specific Use Cases ==
+
+=== Diskless Systems - M. Shroeder ===
+
+2 possible setups :
+ * RAM disk : the whole system in a large file loaded at boot time
+ * NFS mount : a small image loaded from network at boot, other FS through NFS, mainly readonly (and shareable)
+
+Read Hat's way
+ * PXE + NFS mount
+ * Clone the server system
+ * One snapshot for each client : non shared files, writable files
+
+Quattor usage in this context :
+ * Configure RH tools (pxeos, pseboot) via quattor templates and components : ncm-diskless_server
+ * Kickstart for server install and cloning : only install base system
+ * server and its clone configured separatly : chrooted for clients
+ * Client configuration cloned in 2 parts : 1 common to all clients (done on the clone), 1 specific for each client (1 profile / client)
+
+Clone is not really a real machine : cannot receive CDB modifications notification
+ * Has to fetch new profiles via cron
+ * ncm components run on the server but must not impact the server
+ * Client filesystem is read-only but to run a component on a client, need to create some files and the ability to modify existing ones
+ * Not clear if we want to support several clones (several configurations) per server
+
+Current experience : 2 test clusters (2 and 8 clients)
+ * Clients in a private network without access to CDB/SWrep
+ * SPMA cannot be run on the client : establishing a matrix of components that can run on the clients
+
+
+=== Quattor and XEN - S. Child ===
+
+Main problem is the grub component :
+ * Need support for multiple boot
+ * XEN is the kernel, Linux kernel and initrd are "modules"
+ * New version with this support now checked in but problem found at CERN ?
+
+Started ncm-xen :
+ * Write configuration files for individual VMs
+ * Should also write base Xen configuration
+ * Will set up links for automatic start of domains
+ * Will check in 0.1 soon... Still not mature !
+
+GridBuilder : web based interface (Ajax based) for creating and managing VMs
+ * http://gridbuilder.sourceforge.net, developper by Dublin Trinity College (Author : S. Childs)
+ * LVM allows fast creation of COW FS images
+ * Database of VMs and images
+ * Quattor used for configuration. Still small amount of pre-configuration :
+   * Configure network on filesystem images
+   * Fetch Quattor profile
+
+Possible improvements in Quattor integration
+ * Automatic generation of node profiles from user supplied description in GridBuilder
+ * Support for Colinux, a version cooperation with Windows for sharing of HW ressources (memory...)
+ * Condor pool
+
+
+== PAN Compiler Update ==
+
+C version : implementation frozen, only major bugfixes, v6.0.3
+ * Performance improvements in last version : compression removed, defaults processing improved, speed and memory consumption as good or better than before
+ * Added "session" directory to improve interface for CDB
+
+Java version : still in development, all major parts functionning
+ * Limited alpha available, first beta mid December, Production January
+ * Main part missing : built-in functions
+ * Validation suite is complete
+ * License : probably Apache2 to be consistent with EGEE-II
+ * Source in QWG SVN repository : https://svn.lal.in2p3.fr/LCG/QWG
+ * Backward-compatibility : as much as possible, may be some incompatibilities for some very unused features
+ * Require Java 1.5+
+ * Compilation and packaging : ant
+ * Parser (build) : JavaCC 4.0
+ * Unit testing : JUnit 4.1
+ * Base64 encoding/decoding (build & run) : classes available from Apache/W3C, probably incorporate them directly in the code base.
+
+Syntax changes :
+ * Bit operators
+ * Unary plus for symmetry with unary -
+ * Octal, hex accepted everywhere : ranges, path...
+ * Limits allowed on record statements
+ * 'bind' statement added for binding a path to a type, in replacement of one form of 'type' (this 'type' usage will be deprecated)
+ * 'return' allowed where functions are (not very useful, grammar simplification)
+ * Warning could be issued for deprecated usage
+
+Other changes :
+ * Stricter syntax checking at compile time
+ * Generation of "object" files (binary form of a syntax checked template) to avoid recompilation of an unchanged template
+ * ant tasks will be the primary interface to the compiler (no "binary") but wrapper scripts will be provided for command line
+
+Incompatibilities :
+ * 'bind' is now a keyword
+ * OBJECT, SELF, ARGV ARGC defined to conform to best practices for global varaible and avoid conflict (at grammar level) between object keyword and object variable
+ * No pointers to properties : now 'x = y[0] = 0; y[0] = 1' now leaves x==0 (currently x is also set to 1)
+
+Emphasis for first release : verifying functionality and measuring performances
+ * In particular evaluate cost/benefit for object files
+
+Future changes after initial release :
+ * Parallelization : compilation of templates, building of configuration trees
+ * Remove deprecated features : lowercase global variable, deprecated form of 'type', 'define' keyword
+ * Addition of string functions : uppercasing, lowercasing, push, pop
+ * More default types : XMLSchema, port...
+
+Missing features/pending bug reports :
+ * Unescape strings in traceback produced in error msg
+ * Add a file existence test operator
+ * Add an argument to matches() to allow to pass global options (like in Perl)
+
+== QWG Templates ==
+
+From discussion : need to think about explicit support for CDB
+ * Probably mainly the matter of defining load paths in an optional template in replacement for cluster.build.properties
+
+
+== CDB/SCDB Update ==
+
+== CDB Update ==
+
+New features since last workshop :
+ * Namespaces
+ * X509 and Krb authentication
+ * ACLs with namespace support
+ * Client/server improvement through session metadata
+
+State management through metadata :
+ * Problem was session directories used for both data and state
+ * Clear separation with specific metadata for state control : better and earlier detection of commits
+
+Parallel compilation of templates :
+ * All profiles compiled with one command doesn't scale : too long, to much memory
+ * Whole set of templates divided into several subsets (without dependencies) compiled separatly and in parallel on several processor/machines
+
+Smarter rollback and commit :
+  * Currently rolling requires rollback all the mods in the sesion, commit can screw up previous modifications committed but not in the session directory
+ * Now allow selective rollback and interactive commit
+
+Handling of dead revisions : problem related to CVS backend
+ * A removed template is no longer in CVS, restoring from backup requires a lot of manual cleanup. Look at SVN as a new backend ?
+
+New authentication for CDB moved to a separate library and now used in all components (SWrep in particular)
+ * Doesn't require to install CDB to use other components, only the library
+
+Other project status :
+ * CDB as a Web service : still any interest ? Not sure...
+ * Fine-grained CDB locking with faire queuing : really required
+ * Concurrent compilation of (non object) templates  : too complex, wait for new compiler..
+ * mod_perl : no further investigation, mod_fastcgi probably a better solution.
+
+Open issues :
+ * CVS desn't scale : a problem with 24k templates. Possible solutions : perl based CVS ? Subversion ? XML database ?
+ * Relocatability : difficult to port to other systems, testing requires a full installation or specific privileges
+
+=== SCDB Update ===
+
+See slides.
+
+Suggestion :
+ * To avoid a full rebuild after repository templates update, ignore these templates when evaluation if a node profile must be recompiled (this is how it is handled within CDB).
+
+
+== Quattor Core Modules Update ==
+
+Several ongoing developpement at CERN, of general interest.
+
+CDB2SQL :
+ * Rewrite with multithreaded Python and a fast XML parsing library. Sould have no CERN dependency, only Oracle dependency (but should not be difficult to add support for other RBMS)
+
+autoconf :
+ *  In contact with ETICS to use their framework for configuration and automatic build
+ * Will remain possible to run Quattor build tools outside of ETICS (mainly need to define --LOCALDIR)
+
+CCM :
+ * Add support for a failover profile, in case the URL in --profile is not available
+ * Problems observed causing ncm-cdispd to crash
+
+wassh2 : improvement of wassh (parallel ssh), interfacing with CDB
+ * CDB plugin done using a plugin : CERN uses Oracle/cdb2sql
+ * Will be part of Quattor 1.3
+ * Non CERN testers welcome
+
+Notification systems : have the ability to trig execution of a NCM component on one or several nodes without logging to them (even with wassh)
+ * Basically one command : notify_host myhost component
+ * 'component' is a keyword translated on the target host via a configuration file
+ * Current version with lots of CERN dependencies, plan to reengineer it and release it as part of Quattor
+
+
+== AII ==
+
+Since last workshop, various bug fixes and some enhancements :
+ * Support for rescue images
+ * Support for alternative device naming schemes
+
+Work in progress :
+ * Separate site configuration from component configuration
+ * Error handling
+ * Complete partitioning scheme
+ * Documentation
+ * Schema change for block devices
+
+On the todo list :
+ * SPMA proxy start
+ * Support for SINDES
+
+Separating site and component configuration :
+ * Idea if to have pro_software/declaration_component_aii really only related to AII component. Will provide a function aiiconfig(name,disk) to return actual configuration
+ * pro_config_aii_OSNAME : OS / arch specific configuration
+ * pro_aii_config_site : site specific information
+
+Expected incompatibilities in new version :
+ * Separation of configuration
+ * Change in schema for generic block devices
+
+Remark : could add a property to /software/components/osinstall/options to select if initial installation is done with DHCP or final address and improve the template accordingly
+
+
+== New Schema for Block Devices ===
+
+Need to add support for SW RAID, new kinds of block devices, filesystem mount options.
+ * Also need to align naming for HW Raid and SW Raid
+
+New schema proposal : /system/blockdevices/[disk|md|lvm|hwraid]
+ * disk : 1 entry per disk, almost all information optional, mainly partitions 
+ * md/hwraid : basically the same, add information about raid members, raid level, stripe size...
+ * lvm : allow to use more sophisticated LVM scheme that current functions (LVM vg splitted over several HW raid...)
+
+Seems ok but need to check :
+ * AII compatibility : in particular KS template
+ * Ability to represent multi-pathed devices
+ * Is the md/hwraid distinction relevant : may be just keep a property in a common schema
+
+== Namespaces ==
+
+See German presentation. Basically everybody agrees. Just a few details :
+ * What's in pan/ and what's in quattor/
+ * Have all type defined in one place as some will move to the compiler
+ * Hardware : may be cards is not needed, for ram/ use bank.tpl
+ * Components : upgrade quattor build tools to be able to automatically produce namespaced version from non namespaced source, call declaration.tpl schema.tpl and define default values there in the future, add the ability to build RPM-less components (automatically insert perl script into template).
+
+Clusters :
+ * CDB relies on clusters and subclusters
+ * Could be added in SCDB
+ * Rename site/ to config/
+ * Difficult to agree on the whole layout between CDB/SCDB as the concepts are different
+
+Sites :
+ * Rename site/ to config/
+ * Difficult to agree on the whole layout between CDB/SCDB as the concepts are different
+
+OS templates :
+ * Change rpmlist to rpms
+ * Rename templates describing groups to groupname.tpl
+ * Renamme os/ namespace to config/
+
+Standard variables : no real need to agree on variables, need to agree only on the schema
+
+OS/arch naming : originally os_arch, QWG use os-arch, no real need to agree as this is not in the schema
+
+
+== Wish List, Roadmap... ==
+
+Documentation :
+ * Provide user guides in addition to specification for all components
+ * Move quattor.org to Twiki (except home page ?)
+ * Have a short installation guide (not 80 pages...)
+ * List of available components, with a very short explanation and the recommended production version (may come from CVS tags, updated manually if necessary)
+ * Tutorial : differenciate between old and recent tutorials
+
+Open Issues (from Savanah) :
+
+== Conclusions ==
+
+Next meeting : Trinity College, target date : mid-march
+
+
+TBD :
+ * Add an option to osinstall for using DHCP at installation time and merge LAL and standard KS template