PES Platform Engineering Services Agile Infrastructure Project Overview

PES Platform & Engineering Services Agile Infrastructure Project Overview: Status and How to use it Vítor Gouveia, vitor. gouveia@cern. ch IT-PES-PS CERN IT Department CH-1211 Geneva 23 Switzerland www. cern. ch/it

Platform & Engineering Services • • Agenda Agile Infrastructure (AI) Overview Current Status of the AI project Schedule of the AI project How to use the AI : – Creation and configuration of VMs • Current support • Questions ? Agenda Overview Status Schedule Demo Reports problems Questions 2/19

Platform & Engineering Services Agile Infrastructure (AI) Overview • CERN IT is changing strategy for machine provision and configuration. • Cloud Infrastructure as a Service (Iaa. S) – virtual delivery of computing resources in the form of hardware, networking, and storage services. • It may also include the delivery of operating systems and virtualization technology to manage the resources. • The goal of the Agile Infrastructure Project is to: – Provide and efficient and flexible infrastructure service which enables new applications and supports existing ones while exploiting increased capacity and multi-site facilities within the current man power resources Agenda Overview Status Schedule Demo Reports problems Questions 3/19

Platform & Engineering Services Agile Infrastructure (AI) Overview • User documentation being written at: – http: //information-technology. web. cern. ch/book/cernpuppet-development-user-guide • Internal documentation about the AI project: – https: //twiki. cern. ch/twiki/bin/view/Agile. Infrastructure/Web. Home • Agile Infrastructure Components: – Configuration: • Puppet, Foreman, Hiera – Provision: • Open. Stack – Other Services • GIT, Koji Agenda Overview Status Schedule Demo Reports problems Questions 4/19

Platform & Engineering Services Agile Infrastructure (AI) Overview • Agile Infrastructure Configuration Components: – Puppet • Open source configuration management tool like Quattor • Declarative language to describe system configuration • Puppet programs are called “manifests” – Conceptually like a cdb template • The core of the Puppet language is the resource declaration file { 'testfile': path => '/tmp/testfile', ensure => present, mode => 0640, content => "I'm a test file. ", } A singleton collections of resources in puppet is called class • Tutorial about puppet: http: //docs. puppetlabs. com/learning/ • CERN puppet user guide (work in progress): http: //information-technology. web. cern. ch/book/cern-puppetdevelopment-user-guide Agenda Overview Status Schedule Demo Reports problems Questions 5/19

Platform & Engineering Services • Agile Infrastructure Configuration Components (Cont. ): – Puppet ( Cont. ) • Facter – Puppet tools which examines our system and normalizes all the information about it » creates a set of variables and passes them off to puppet – Source of information about the machines (CPUs, RAM, network etc. ) – Functionally similar to CDB hardware/network profiles, but provides information dynamically from various data sources – Hiera • Data store used by puppet • hierarchical-based lookup system Agenda Overview Status Schedule Demo Reports problems Questions 6/19

Platform & Engineering Services Agile Infrastructure (AI) Overview • Agile Infrastructure Configuration Components (Cont. ): – Foreman • Current web front-end for Puppet and Open. Stack • Groups hosts (node) into hostgroups of similar configuration • Hostgroup – Essentially the same a cdb cluster – In a hostgroup several features are aggregated together – Ex: Cluster of a LAMP server ( should include the puppet classes responsible for apache, mysql and php) – Can be hierarchical Ex: 1 st level ~= cdb cluster, 2 nd/3 rd level ~= cdb customization • Real time information about hosts status (node) • Rebuild machines from scratch • High-level machine configuration (~= CDB profile) – Possible to configure a hostgroup or just a single host https: //judy. cern. ch/users/login Agenda Overview Status Schedule Demo Reports problems Questions 7/19

Platform & Engineering Services • Agile Infrastructure Provision Component: – Open. Stack (Hamster) • Software in use for the CERN private cloud. – Principal organizational structure • Tenants – Resource container for a project/VO/… Tenant users can freely create volumes, VMs etc. within a certain quota of total Disk, Cores, RAM, etc… https: //hamster. cern. ch/dashboard 8/19

Platform & Engineering Services Agile Infrastructure (AI) Overview • Other Services – GIT • Distributed version control system • GIT in the AI project: – GIT branches map to Foreman environments ( ~= cdb stages) – Inside a branch it is possible to find manifests and modules Modules: • Collection of configuration components Ex: apache, afs, ssh, sssd • Similar to cdb services and cdb components • Can use other modules • Stored in repository root in modules/ Agenda Overview Status Schedule Demo Reports problems Questions 9/19

Platform & Engineering Services • Other Services (Cont. ) – GIT ( Cont. ) Manifests: • They are specific for each hostgroup • Should not be reusable • May reference several modules • Conceptually similar to cdb cluster/customization templates • Describe which modules/puppet classes/puppet resources should be applied to a set of machines (hostgroup) Ex: lxplus, lx. Batch • Stored in repository root in manifest/ https: //gitgw. cern. ch/gitweb/ Agenda Overview Status Schedule Demo Reports problems Questions 10/19

Platform & Engineering Services • Other Services (Cont. ) – KOJI • Creates RPMs and publish to YUM repositories • Instead of the swrepsrv Agenda Overview Status Schedule Demo Reports problems Questions 11/19

Platform & Engineering Services Current Status of the AI project • Pre-Production Service running: – Open. Stack Essex, Foreman 1. 0 and puppet 2. 7 • Several IT services running on the AI – EX: SLC 6 Batch machines • Deploy puppet configured VMs on Open. Stack – Configuration and Management through web portal (Foreman) – VM creation still requires using command-line scripts for the time being Agenda Overview Status Schedule Demo Reports problems Questions 12/19

Platform & Engineering Services Schedule of the AI project • Deploy into production: – End of January 2013 • Folsom (Openstack) • suitable for implementing the 'Cattle‘ – machines that may die without impact on production Ex: batch machines – Not suitable yet for ‘pet’ machines • Even less if High-availability is necessary – Ex: VOBOX – Eventually Openstack will be integrated in Foreman (together with Puppet) • It will be possible to create and manage VMs through Foreman • Only one website! – Performance and stability will improve over time Agenda Overview Status Schedule Demo Reports problems Questions 13/19

Platform & Engineering Services How to use the AI • From now on you can start using the AI: – Attention: Everything is in pre-production – Things will change!! • What are you going to see? – How to create a VM in the AI – How to configure it More info: https: //twiki. cern. ch/twiki/bin/view/Agile. Infrastructure/VOCGetting. Started Agenda Overview Status Schedule Demo Reports problems Questions 14/19

Platform & Engineering Services How to use the AI • Mailings lists and egroups to join: – lxvoadm-Authorized-Users • Gives Access to [aiadm 02. cern. ch | aiadm 03. cern. ch] server • Aiadm is for AI what lxvoadm is for Quattor – puppet-users • CERN puppet discussion – ai-vobox-migration-discussion: • Gives you access to create VMs in the AI project • Best way to provide feedback Agenda Overview Status Schedule Demo Reports problems Questions 15/19

Platform & Engineering Services 16/19

Platform & Engineering Services Current support • This is not a production service so official support lines don't exist yet. • All Feedback about the VOBox migration to AI should be reported through the mailing list ai-vobox-migration-discussion Agenda Overview Status Schedule Demo Reports problems Questions 17/19

Platform & Engineering Services Current support • Upcoming meetings… – Topics to be covered • • • GIT, branches Partitions + resize Monitoring + custom sensors Koji + RPMs Accounting etc… – Starting from January every two weeks… • Thursdays afternoon Agenda Overview Status Schedule Demo Reports problems Questions 18/19

Platform & Engineering Services Questions? ? ? 19/19