ASIS et le projet EU Data Grid EDG

ASIS et le projet EU Data. Grid (EDG) Germán Cancio IT/FIO

Outline u ASIS u EDG reminder WP 4 and software distribution u Comparison EPFL visit, 14/7/03 German Cancio IT/FIO - n° 2

ASIS: reminder u ASIS: stands for ‘Application Software Installation Service’ n u Package generation: n u u n Adding/removing/upgrading packages (HAPPI, ASIStm) n Repository replication (ASISlcm) Client access (over shared file system) Configurable agent (ASISwsm/p) Repository contents at CERN: n u assisted multiplatform application compilation (HAPPI) Repository management: n u Started 1993; joint project between CERN/IT and EPF Lausanne CERNLIB, CASTOR, Compilers, GNU tools, La. Te. X, editors… Focus on providing an uniform application layer on multiple UN*X platforms EPFL visit, 14/7/03 German Cancio IT/FIO - n° 3

ASIS at CERN: evolution … Clients 3/03: 3710 Clients 10/99: 1680 EPFL visit, 14/7/03 German Cancio IT/FIO - n° 4

… today’s changed environment… Since ASIS was designed… u RISC n phaseout: Only 2 CERN-IT supported Un*x OS (Linux & Solaris) u Large scale farms (LXBATCH scaling up to O(10 K)) u Mature n RPM (RH Linux), PKG (Solaris) u Large n and reliable vendor package managers number of default packages shipped with OS distros RH 73: ~ 700 packages, Solaris 8: ~ 410 packages u Availability n of large local disks Packages don’t have to be ‘links only’ to application servers EPFL visit, 14/7/03 German Cancio IT/FIO - n° 5

… and its impact on ASIS u ASIS cannot easily handle ‘foreign’ packages (packages not generated by ASIShappi) n n More and more applications already bundled with Linux distros -> no need for generating them with ASIS RH 61: 510 packages provided via ASIS, RH 73: 160 u ASIS is built around a shared file system between repository, configuration database and clients. n File system scalability issues u. Inside the ASIS project, the decision taken was to minimize further development efforts, and to participate in the design of the EDG fabric management work package. EPFL visit, 14/7/03 German Cancio IT/FIO - n° 6

EDG WP 4 EPFL visit, 14/7/03 German Cancio IT/FIO - n° 7

EDG WP 4 u WP 4 is the ‘fabric management’ work package of the EU Data. Grid project. u Objective: n To develop system management tools for enabling the deployment of very large computing fabrics […] with reduced sysadmin and operation costs. u Includes solutions for n automated from scratch node installation n node configuration/reconfiguration n software storage, distribution and installation n storing, maintaining and retrieving configuration information. EPFL visit, 14/7/03 German Cancio IT/FIO - n° 8

WP 4 tools used for SW distribution (I) SWRep (Software Repository): u Client-server u Universal n n toolsuite for the management of software packages repository: Extendable to multiple platforms and package formats (RHLinux/RPM, Solaris/PKG, … others like Debian dpkg) Multiple package versions/releases u Management (“product maintainers”) interface: n ACL based mechanism to grant/deny modification rights n Current implementation using SSH u Client n access: via standard protocols HTTP (scalability), but also AFS/NFS, FTP u Replication: EPFL visit, 14/7/03 using standard tools (eg. rsync) German Cancio IT/FIO - n° 9

WP 4 tools used for SW distribution (II) Central Configuration Database (CDB): u Common store for configuration information n u …including what software packages to deploy from which repository on which nodes Configuration information can be arranged in templates: Linux Base packages CC packages EDG/LCG m/ware LXBATCH lxbatch 444 lxbatch 445 lxbatch 446 u Possible to create template combinations/hierarchies to match service structures u Each template can be maintained (using a GUI) by a different person u Configuration information is validated and kept under version control using transactions EPFL visit, 14/7/03 German Cancio IT/FIO - n° 10

WP 4 tools used for SW distribution (III) Software Package Management Agent (SPMA): u Runs on every target node u Configurable u Multiple u Plug-in n u Can n n repositories can be accessed (eg. division/experiment specific) framework allows for portability System packager specific transactional interface (RPMT, PKGT) manage either all or a subset of packages on the nodes Useful for add-on installations, and also for desktops Configurable policies (partial or full control, mandatory and unwanted packages, conflict resolution…) u Addresses n locally or via CDB scalability Packages can be stored ahead in a local cache, avoiding peak loads on software repository servers (simultaneous upgrades of large farms) EPFL visit, 14/7/03 German Cancio IT/FIO - n° 11

WP 4 SW distribution architecture inventory Repository A packages (RPM, PKG) CDB CLI Mgmt API Packages config GUI http nfs afs ftp Repository B Client nodes SUE/ NCM SPMA. cfg cache EPFL visit, 14/7/03 SPMA German Cancio IT/FIO - n° 12

Comparison EPFL visit, 14/7/03 German Cancio IT/FIO - n° 13

Comparison: Software Repository Management access: u Both ASIStm and WP 4’s SWRep provide authentication, and authorization mechanisms (using ACL’s) Client access: u ASIS: u WP 4: shared file system required. Can be any of HTTP, NFS, AFS, FTP Replication: u The ASIS replication tool (ASISlcm) replicates both the repository contents and the default configuration. u WP 4 does not require a specific tool for repository replication: standard tools (eg. rsync) already exist. EPFL visit, 14/7/03 German Cancio IT/FIO - n° 14

Comparison: Software update client u SPMA n can do everything ASISwsm(p) can do Almost same design team… u SPMA can do more: n Manage all packages (system and applications) n Designed to use the system packagers (RPM, PKG) n Multiple repository access protocols n Cache management u Configuration: n In ASIS, only one default configuration profile n WP 4 CDB: any number of profiles, version control u Configuration n n GUI: ASIS provides a end-user GUI (tkwsm) The WP 4 CDB GUI (Pan. GUIn) is targeted towards sysadmins, but end-user GUI feasible EPFL visit, 14/7/03 German Cancio IT/FIO - n° 15

Comparison: Package generation: u ASIShappi can build applications for any UN*X platform. u Application building is not addressed by WP 4. n EDG uses the standard RPM build mechanisms n No equivalent build mechanism available for Solaris EPFL visit, 14/7/03 German Cancio IT/FIO - n° 16

Status and Documentation u> 700 nodes using SPMA at CERN (LXPLUS and LXBATCH) n To become >1000 before the end of the summer u Software Repository: HTTP based server clustering solution n Load balanced using DNS round robin n Replication done using rsync u Two u WP 4 n home page: http: //cern. ch/hep-proj-grid-fabric u WP 4 n platforms: RH 73 and RH Enterprise Server 2. 1 installation task: http: //cern. ch/wp 4 -install EPFL visit, 14/7/03 German Cancio IT/FIO - n° 17