Context Navigation

20091104

Timestamp:: Nov 5, 2009, 4:18:01 PM (16 years ago)
Author:: /O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=Michel Jouvin
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

Meetings/Workshops/20091104

-              v2
+              v3
 Need to discuss how to enforce annotations for high-level services (features) and produce some tools to easily access this information.
+== Other Developments ==
+=== Quattor Asset Database - C. Tryantafyllydys ===
+AUTH needed a tool to ease everyday tasks for new sysadmins and allow to feed other tools with Quattor data
+ * Eg. configure a monitoring system from Quattor
+QAD main characteristics:
+ * Written in Ruby
+ * A few schema extensions:  /monitiroing, network "parent" (switch it depends on), rack
+   * Need to be merged with CERN similar extensions
+   * /monitoring is an alternative to the LEMON /monitoring
+ * Not yet decided: access to SVN? read or write?
+   * Currently using a post-commit hook to synchronize QAD db with SCDB deploys: impact on performance
+   * Would like to remove this in the future and rely on XML but some high level information is currently missing: need to extend the schema
+ * Allow to change some node characteristics: OS, IP... and redeploy
+ * From the config db, can list nodes attached to a specific switch: rely on configured information
+ * Can list all the VMs attached to a host (using the enclosure definition)
+Need to make progress on using CDB2SQL back-end for producing/maintaining db and use it in all tools that need a build a db from the XML
+ * Rename XML2SQL
+   * CERN will provide an update on the status and the best version to start from
+ * May look at document-oriented dbs, liek CacheDB
+=== Remote Configuration with Quattor - N. Williams ===
+''Disclaimer: doesn't currently work with SCDB (because of dependency over CDB notification).''
+Goal: apply Quattor benefits to boxes that cannot run Quattor client by using delegation
+ * A Linux box will acts as a delegate where  a configuration module will execute appropriate configuration commands
+ * Combination of AII and and CCM/NCM/NCD
+Current implementation:
+ * quattor-remote-dispatcher (QRD): a tool running on Linux box and receiving CDB notification messages. It acts as a replacement for listend, cdispd, ccm-fetch, ncm-ncd
+   * Configuration allows to define which part of the configuration are listened and what is the command to run
+   * Can use different commands on different sets of nodes
+   * Can define constraints on some part of the configuration to do different things based on some configuration state (for example state=build or production)
+ * quattor-remote-configure: an AII equivalent allowing to produce a new configuration for the managed box and notify the remote dispatcher
+   * Configuration of the managed device is  under /software/components
+   * The actual component to use is defined by /system/components/namespace: default is NCM::NCD:: but can be defined to something specific to a device, eg. ESX, Netapp...
+Used to manage virtualization in Aquilon (for a WMware Hypervisor)
+ * VMs are not associated with actual hosts (handled for example by VMware) but with clusters
+   * A cluster is a  group of hosts running an hypervisor. There is an object template for each cluster.
+   * Each virtual host (machine running an hypervisor) has also an object profile that allows its configuration out of the box
+ * VMs are managed as normal machines
+Plan to release this as soon as it is polished but will not be able to release the specific components as they are very MS specifics and sometimes rely on non public APIs.
+=== Virtualisation - D. O'Callaghan ===
+Currently used Hypervisors configured with Quattor:
+ * Xen: TCD, NIKHEF, CERN, AUTH
+ * OpenVZ: UAM
+ * VMware: MS
+Also in use but not managed by Quattor:
+ * KVM: CNAF
+ * Hyper-V: CERN
+ VM cluster managers of interest: Platform/VMO, OpenNebula
+Issues and requests:
+ * Use libvirt, common configuration module for Linux-base hypervisors
+   * Could look at what is done by Puppet or other config tools
+ * Creation of images
+QWG templates for basic virtualisation: mainly used at TCD (also in Senegal!)
+ * Rely on ncm-xen, would be great if we had support KVM, probably using libvirt
+=== Integration with Monitoring Systems - C. Tryantafyllydys ===
+Porblem: current Nagios configuration in QWG relies on everything being described (in particular probles) in the configuration. But EGEE and OAT developped NCG to do it more dynamically on a specific node. How to take advantage of this?
+NCD: generic tool to define Nagios configuration based on context
+ * 2 main basic modules/entry points: NCG::SiteSet and NCG::SiteInfo::
+ * Also several internal modules to define probes to use, ... that would benefit to receive information from Quattor
+   * Exemple: configure all probes for a CE if the node is configured as a CE
+Integration between NCG and XML doesn't scale as it is far too long. Need to pre-process data and this is done with QAD.
+ * Currently more a proof of concept than a ready-to-use tool
+Discussion:
+ * Need to figure out how to use NCG to define services without defining hosts and rely on Quattor for host definitions of hosts managed by Quattor
+ * Create a small working group with a specific mailing list: quattor-monitoring
+ * TCD will commit their change to the generic templates: no specific change
+ * NIKHEF will document the change they had to make to generic templates to identify additional customizations needed
+ * CNAF: currently the storage group has its own way of configuring Nagios with filecopy, difficult to change even if convinced
+ * RAL interested but probably not in the short term
+ * ULB/BEGrid started to look at it
+ * Guillaume will fix the Nagios example in QWG repository
+ * Potential scalability pb if nagios server profile need to depend on all the profiles for all the machines it monitors
+== QUEST Proposal Status - M. Jouvin ==
+See [http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=2&resId=1&materialId=slides&confId=67632 slides].
 == Improving Quattor's Accessibility - M. Jouvin ==
 …
    * One proposal is CMDB for ''Configuration Management DB''. Need more thinking
    * May be not really an issue anymore with CERN left as the only site using CDB...
+== QUEST Proposal Status - M. Jouvin ==
+See [http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=2&resId=1&materialId=slides&confId=67632 slides].
+== Other Developments ==
+=== Quattor Asset Database - C. Tryantafyllydys ===
+AUTH needed a tool to ease everyday tasks for new sysadmins and allow to feed other tools with Quattor data
+ * Eg. configure a monitoring system from Quattor
+QAD main characteristics:
+ * Written in Ruby
+ * A few schema extensions:  /monitiroing, network "parent" (switch it depends on), rack
+   * Need to be merged with CERN similar extensions
+   * /monitoring is an alternative to the LEMON /monitoring
+ * Not yet decided: access to SVN? read or write?
+   * Currently using a post-commit hook to synchronize QAD db with SCDB deploys: impact on performance
+   * Would like to remove this in the future and rely on XML but some high level information is currently missing: need to extend the schema
+ * Allow to change some node characteristics: OS, IP... and redeploy
+ * From the config db, can list nodes attached to a specific switch: rely on configured information
+ * Can list all the VMs attached to a host (using the enclosure definition)
+Need to make progress on using CDB2SQL back-end for producing/maintaining db and use it in all tools that need a build a db from the XML
+ * Rename XML2SQL
+   * CERN will provide an update on the status and the best version to start from
+ * May look at document-oriented dbs, liek CacheDB
+=== Remote Configuration with Quattor - N. Williams ===
+''Disclaimer: doesn't currently work with SCDB (because of dependency over CDB notification).''
+Goal: apply Quattor benefits to boxes that cannot run Quattor client by using delegation
+ * A Linux box will acts as a delegate where  a configuration module will execute appropriate configuration commands
+ * Combination of AII and and CCM/NCM/NCD
+Current implementation:
+ * quattor-remote-dispatcher (QRD): a tool running on Linux box and receiving CDB notification messages. It acts as a replacement for listend, cdispd, ccm-fetch, ncm-ncd
+   * Configuration allows to define which part of the configuration are listened and what is the command to run
+   * Can use different commands on different sets of nodes
+   * Can define constraints on some part of the configuration to do different things based on some configuration state (for example state=build or production)
+ * quattor-remote-configure: an AII equivalent allowing to produce a new configuration for the managed box and notify the remote dispatcher
+   * Configuration of the managed device is  under /software/components
+   * The actual component to use is defined by /system/components/namespace: default is NCM::NCD:: but can be defined to something specific to a device, eg. ESX, Netapp...
+Used to manage virtualization in Aquilon (for a WMware Hypervisor)
+ * VMs are not associated with actual hosts (handled for example by VMware) but with clusters
+   * A cluster is a  group of hosts running an hypervisor. There is an object template for each cluster.
+   * Each virtual host (machine running an hypervisor) has also an object profile that allows its configuration out of the box
+ * VMs are managed as normal machines
+Plan to release this as soon as it is polished but will not be able to release the specific components as they are very MS specifics and sometimes rely on non public APIs.
+=== Virtualisation - D. O'Callaghan ===
+Currently used Hypervisors configured with Quattor:
+ * Xen: TCD, NIKHEF, CERN, AUTH
+ * OpenVZ: UAM
+ * VMware: MS
+Also in use but not managed by Quattor:
+ * KVM: CNAF
+ * Hyper-V: CERN
+ VM cluster managers of interest: Platform/VMO, OpenNebula
+Issues and requests:
+ * Use libvirt, common configuration module for Linux-base hypervisors
+   * Could look at what is done by Puppet or other config tools
+ * Creation of images
+QWG templates for basic virtualisation: mainly used at TCD (also in Senegal!)
+ * Rely on ncm-xen, would be great if we had support KVM, probably using libvirt
+=== Integration with Monitoring Systems - C. Tryantafyllydys ===
+Porblem: current Nagios configuration in QWG relies on everything being described (in particular probles) in the configuration. But EGEE and OAT developped NCG to do it more dynamically on a specific node. How to take advantage of this?
+NCD: generic tool to define Nagios configuration based on context
+ * 2 main basic modules/entry points: NCG::SiteSet and NCG::SiteInfo::
+ * Also several internal modules to define probes to use, ... that would benefit to receive information from Quattor
+   * Exemple: configure all probes for a CE if the node is configured as a CE
+Integration between NCG and XML doesn't scale as it is far too long. Need to pre-process data and this is done with QAD.
+ * Currently more a proof of concept than a ready-to-use tool
+Discussion:
+ * Need to figure out how to use NCG to define services without defining hosts and rely on Quattor for host definitions of hosts managed by Quattor
+ * Create a small working group with a specific mailing list: quattor-monitoring
+ * TCD will commit their change to the generic templates: no specific change
+ * NIKHEF will document the change they had to make to generic templates to identify additional customizations needed
+ * CNAF: currently the storage group has its own way of configuring Nagios with filecopy, difficult to change even if convinced
+ * RAL interested but probably not in the short term
+ * ULB/BEGrid started to look at it
+ * Guillaume will fix the Nagios example in QWG repository
+ * Potential scalability pb if nagios server profile need to depend on all the profiles for all the machines it monitors
+ * Commands: wrapper for quattor commands on managed nodes and on AII server developped by David as an action after last workshop
+   * https://sourceforge.net/apps/mediawiki/quattor/index.php?title=Command_Renaming
+   * `quattor` command as a wrapper over ncm-ncd, ccm-fetch, ncm-query...
+   *  `quattor-installer` as a wrapper over aii-xxx
+     * For AII, should not distinguish between local and remote (`installfe` and `shellfe`)
+   * In both cases, symlinks created for each subcommands to allow to use shell completion (à la git)
+   * Currently deployed by a template: David will try to build 2 separate RPMs (1 for managed nodes, 1 for AII server)
 == Miscellanous ==