Enabling Grids for Escienc E Quattor Installation and
Enabling Grids for E-scienc. E Quattor : Installation and Configuration Management M. Jouvin (LAL-Orsay) jouvin@lal. in 2 p 3. fr Grid Administration Training Montpellier, February 2009, 4 -5 www. eu-egee. org EGEE-III INFSO-RI-222667
Agenda Enabling Grids for E-scienc. E • • • Why Quattor ? Architecture Configuration Description Serveur et client Installation initiale Administration GRIF Example Quattor usage status Proposed configuration for tutorial Useful links EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Why Quattor ? Enabling Grids for E-scienc. E • Quattor developed EDG WP 4 – EDG : need a management tool for large farms – Initial installation AND configuration changes – Existing tools do one or the other § Examples : Kickstart, Imaging, APT… – Based on CERN (difficult) experience with management of a large number of machines § Crucial to have a unique and uniform description of the whole configuration § Be able of doing atomic installation of several packages • Installation must be reproducible – Quick reinstallation of a machine after a failure – Be able to ensure 2 machines are identical (WNs) • Linux (RH, rpm) • No dependency on grid middleware EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Main Characteristics Enabling Grids for E-scienc. E • Machine configuration stored in a versioned database – Reproducible installation : will reuse the same configuration – Active configuration is updated by modifying configuration database – From initial installation to machine death… § Initial installation with standard tool (Kickstart) – Configuration versioning allows an easy rollback • Service configuration reuse – Different types of machines can share a service configuration – Easier maintenance : only one place to modify to update a service configuration • Configuration consistency checked before deployment to lower risk of mistakes and broken configuration – Configuration information checked against a schema – Existence of packages to deploy EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor Workflow Enabling Grids for E-scienc. E EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor (Core Components)… Enabling Grids for E-scienc. E • Quattor = 1 database + 1 deployment server – Configuration database is a SVN repository • Configuration organized in templates – Description of final configuration, not how to do it § Language: PAN – Template inheritance – Hardware and software • Deployment server: dhcpd + tftpd – Server is notified after a configuration database change – Notification of clients to trig a reload of their configuration – Possible to have several servers attached to the same configuration database (multi-site configuration) – No Quattor specific service EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
… Quattor (Core Components) Enabling Grids for E-scienc. E • Client : Configuration Cache Manager (CCM) – Receive a configuration change notification (cdp-listend) and download new configuration (command ‘ccm-fetch’) § Profile update may be forced by manually executing ‘ccm-fetch’ • Client : Node Configuration Manager (NCM) – Process ncm-cdispd – Implements configuration changes – Based on “components” (plug-ins) § 1 component manages 1 “service” (NIS, accounts, Globus…) § A component is a Perl script with a particular API – ‘ncm-query’ alllows to display node configuration • Quattor management can be done from any platform (Unix, Windows, Macintosh) – PAN compiler (Java) + client SVN + ant available on all platforms – May use GUI like Eclipse EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor Other Components Enabling Grids for E-scienc. E • Package deployment : SPMA – Implemented as a NCM component – Developed in the framework of Quattor § Atomic modification of installed packages (rpmt) § Relies on an explicit list of package version/arch to install • Allows upgrade and downgrade § Check package dependencies before deployment – APT and YUM are also supported but less integrated § Generally not used at Quattor sites • Initial installation : AII – Initial installation of OS and Quattor client § Configuration and package deployment done via NCM components – Use standard installation tool (Kickstart/Anaconda) + PXE § DHCP, TFTP and KS configured from configuration database – Managed with command ‘aii-shellfe’ EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Configuration and Templates Enabling Grids for E-scienc. E • Machine configuration used by various Quattor components is a XML file – 1 profile per machine, downloaded with http • Machine profile built from a high-level configuration description : templates – Templates use PAN language (declarative-like) – Templates are compiled to produce the machine profile – Standard and generic templates available to configure OS and g. Lite § QWG Templates – 1 profile generally uses many templates § Templates are similar to Russian puppets… § In QWG, several hundreds of templates used by all machines, most of them shared between machines and services EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
PAN Language… Enabling Grids for E-scienc. E • Declarative language – Describes final state – Mainly made of value assignments to configuration paths or variables § A value may be based on another configuration path/variable – Non procedural: impossible to define conditional block • Allow to define functions – Written in PAN language – Produce value to be assigned to configuration paths/variable – Validation functions used to check validity of values at the end of compilation EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
… PAN Language Enabling Grids for E-scienc. E • Hierarchical organization of information – Exemples : § « /hardware/network/interfaces/eth 0/ip » = « 134. 158. 88. 34 » ; • Strong typing of values – Configuration schema defines allowed resources and values § Similar to a structure/class definition § 1 standard schema proposed – Type conversion functions available • Support for variables – Can be used as target of assignements and as input to compute other configuration values – Heavily used by QWG templates to customize generic templates EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor SCDB Enabling Grids for E-scienc. E • Versioned database of PAN templates – No versioning of machine profiles : always possible to rebuild from templates • Based on Subversion – No Quattor specific components except a few ant tasks – Fine-grained access control based on SVN ACLs • SCDB operations implemented as ‘ant’ tasks – Written in pure Java : usable on all platforms • 3 main operations – compile : compile locally all machine profiles affected by template changes § Dependency management, incremental compilation – deploy : deployment of new configuration (machine profiles) § Profiles compiled on server, then clients are notified § Occurs only if compilation was successful and local copy of repository in synch with the repository itself – update. rep. templates : refresh templates associated with RPM repositories EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
NCM Components Enabling Grids for E-scienc. E • Plug-ins of ncm-ncd managing one part of the configuration – Each component manages « /software/components/componentname » by default – Possible to define dependencies between components § E. g. SPMA to install the component • Must be written in Perl • Many components available – core : general components (in QWG standard/components) § chkconfig, named, filecopy… – lcg : configuration of g. Lite services (in QWG grid/version/components) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor Client (SPMA) Enabling Grids for E-scienc. E • NCM component responsible for package deployment – Started by ncm-ncd – Configuration stored in templates : « /software/packages » – Configuration done with a set of dedicated PAN functions : pkg_add(), pkg_repl(), pkg_del()… § Check that needed packages are “present” in a RPM repository – Possible but difficult to use APT or YUM instead of SPMA § Not supported in QWG templates – Use rpmt to implement atomic modification of a list of RPMs and check dependencies before applying modifications § rpmt : written in Python, based on standard RPM libraries/module § No modification is done if any dependency problem in the transaction • Upgrade and downgrade (rollback) – Based on an explicit selection of version to install – Ability to install different versions on different machines from the same repository EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Quattor AII Enabling Grids for E-scienc. E • AII : Automated Installation Infrastructure • PXE (DHCP+TFTP) + Kickstart configured from machine profile using ‘aii-shellfe’ command – DHCP (ISC) : may be shared between Quattor and non Quattor usage § Entries managed by Quattor must be in one dedicated file – TFTP : No exclusive usage by Quattor required – Kickstart : may be customized via AII hooks but normally standard configuration is enough • Re-installation may be triggered by reboot – Always boot PXE – Select between boot or reinstallation with ‘aii-shellfe’ EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Administration with Quattor Enabling Grids for E-scienc. E • Template editions (site specific templates) – Most of the templates (>95%) are standard and should not be modified • Compile and deploy new configuration – ant, svn commit, ant deploy – Eclipse (IDE) may be used to edit, compile and deploy • Initial installation – Generate Kickstart+DHCP+TFTP configuration : § aii-shellfe --configure node – Installation at next reboot : § aii-shellfe --install node – Reboot using PXE (normally configured in BIOS as first device) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
MW Update Enabling Grids for E-scienc. E • Download templates for the new version and merge into local configuration – Tool directory-sync (https: //trac. lal. in 2 p 3. fr/LCGQWG/wiki/Download/QWGTemplates /Install) • ‘ant deploy’ – Recompile les profils – Notifie les clients qu’un nouveau profil est disponible EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
OS Update Enabling Grids for E-scienc. E • Download templates for the new version into local configuration • Update OS version used by a node in template os/version. db (site specific) • escape("grid 01. lal. in 2 p 3. fr"), "sl 450 -x 86_64", • ‘ant deploy’ – Recompile profiles – Notify clients that a new configuration (profile) is available EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Complex Configuration Enabling Grids for E-scienc. E • Resources are organized in « cluster » – Set of machines sharing configuration parameters – Machines in a cluster may run different OS version/arch – Not necessarily connected to a cluster HW • Support for multi-site configurations – E. g. GRIF = 6 sites with specificities – Every cluster may belong to several sites § E. g. : local site + GRIF • 1 template may be redefined in smaller scope – Search path defined per cluster § E. g. Standard, GRIF, site local, cluster – GRIF : +95% common templates § 10 -20 specific templates per cluster or site (except profiles) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
SCDB Example : GRIF Enabling Grids for E-scienc. E (15) cluster. pan. includes=sites/ipno/**/* sites/grif/**/* os/sl 305 -i 386/**/* standard/**/* (2500) (4) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Example of Templates. . . Enabling Grids for E-scienc. E • All profiles for one specific machine types are identical object template profile_ipnls 2005; include machine-types/wn; # Add repositories include repository/config; • WN type defined based on other standard templates template machine-types/wn; … # VO configuration variable CONFIGURE_VOS = true; variable CREATE_HOME ? = undef; variable NODE_VO_PROFILE_ENV = true; # Include base configuration of a g. Lite node include { 'machine-types/base' }; # Include WN components include { 'glite/wn/service' }; # Add site specific configuration, if any include { return(WN_CONFIG_SITE) }; … EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
… Exemple de Templates Enabling Grids for E-scienc. E • Base configuration of a g. Lite machine unique template machine-types/base; … # profile_base for profile structure include { 'quattor/profile_base' }; # NCM core components include { 'components/spma/config' }; include { 'components/grub/config' }; # hardware include { 'hardware/functions' }; "/hardware" = if ( exists(DB_MACHINE[escape(FULL_HOSTNAME)]) ) { create(DB_MACHINE[escape(FULL_HOSTNAME)]); } else { error(FULL_HOSTNAME + " : hardware not found in machine database"); }; # common site machine configuration include { 'pro_site_config' }; # File system configuration. variable FILESYSTEM_LAYOUT_CONFIG_SITE ? = "site/filesystems/glite"; variable FILESYSTEM_CONFIG_SITE ? = "site/filesystems/glite"; # Select OS version based on machine name include { 'os/version' }; # Load g. Lite version information include { 'defaults/glite/version' }; … EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Site Parameters Enabling Grids for E-scienc. E • Site configuration defined with variables template site/glite/config; … # MYPROXY CONFIGURATION -------------------------variable PX_HOST ? = "myproxy. grif. fr"; variable GRID_TRUSTED_BROKERS ? = list( "/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid 09. lal. in 2 p 3. fr", "/O=dutchgrid/O=hosts/OU=nikhef. nl/CN=boswachter. nikhef. nl", "/O=dutchgrid/O=hosts/OU=nikhef. nl/CN=bosheks. nikhef. nl", "/O=GRID-FR/C=FR/O=CEA/OU=DAPNIA/CN=node 27. datagrid. cea. fr", "/O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=node 08. datagrid. cea. fr", ); # MON BOX PARAMETERS (R-GMA), Apel variable MYSQL_PASSWORD ? = "wp 6_saclay"; variable MON_HOST ? = "node 06. datagrid. cea. fr"; variable APEL_ENABLED ? = true; variable APELDB_PWD ? = "lhc 4 G"; # OTHER SERVICE LOCATIONS ------------------------variable LFC_HOSTS ? = nlist( "grid 14. lal. in 2 p 3. fr", nlist('alias', 'lfc. grif. fr'), ); variable GRIDICE_SERVER_HOST ? = MON_HOST; # RB / WMS --------------------------------variable RB_HOST variable WMS_HOST … EGEE-III INFSO-RI-222667 ? = "node 04. datagrid. cea. fr"; ? = "wms. grif. fr"; Quattor - M. Jouvin 25/09/2020
SCDB : Tracing Changes… Enabling Grids for E-scienc. E • Change log easy to browser with a web-based SVN client (e. g. Trac) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
. . . SCDB : Tracing Changes Enabling Grids for E-scienc. E EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Enabling Grids for E-scienc. E Who is Using Quattor ? • Usage growth – 50+ sites, in Europe and Asia § Several group of sites managed with one DB (Ireland, Belgium) – Several quattorized countries § France, Spain, Ireland, Belgium • Wide variety in size, most using QWG templates – CERN : ~5000 nodes – ~500 nodes : NIKHEF, CNAF, PIC, GRIF – Many smaller sites, e. g. SN-UCAD (4 machines + 2 VMs) • Main usage is grid management but growing interest for other usages – Internal servers, desktops, virtual machines (Xen) EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Tutorial Proposed Configuration Enabling Grids for E-scienc. E • • • 1 Quattor server 1 CE 2 WNs 1 SE 1 BDII Start with dedicated HW for each machine and demonstrate use of VMs if enough time EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
Documentation and Links Enabling Grids for E-scienc. E • Quattor official site : http: //quattor. org – On Source. Forge • QWG web site : http: //trac. lal. in 2 p 3. fr/LCGQWG – Documentation of grid templates and their customization – SCDB § Initial installation of SCDB : https: //trac. lal. in 2 p 3. fr/LCGQWG/wiki/Download/SCDB • Scientific Linux : http: //www. scientificlinux. org EGEE-III INFSO-RI-222667 Quattor - M. Jouvin 25/09/2020
- Slides: 28