http cern chganga Ganga Status and Outlook K

http: //cern. ch/ganga Ganga Status and Outlook K. Harrison (University of Cambridge) 16 th Grid. PP Meeting Queen Mary, University of London, 27 th-29 th June 2006 28 June 2006

People/groups behind Ganga • Ganga is an ATLAS/LHCb joint project to develop a Grid user interface • Strong support from UK (PPARC/Grid. PP) and EU (EGEE/ARDA) • Current core team: – F. Brochu (Cambridge), U. Egede (Imperial), J. Elmsheuser (München), K. Harrison (Cambridge), H. C. Lee (ASCC), D. Liko (CERN), A. Maier (CERN), J. T. Moscicki (CERN), A. Muraru (Bucharest), A. Soroko (Oxford), C. L. Tan (Birmingham) • Contributions past and present from many others 28 June 2006 2

Applications ATLAS applications LHCb applications Other applications File catalogues User interface for job definition and management Remote repository Local repository Ganga job archives 28 June 2006 Metadata catalogues Ganga monitoring loop Tools for data management Data storage and retrieval • Ganga has built-in support for ATLAS and LHCb • Component architecture allows customisation for other user groups Ganga in sixty seconds Experiment-specific workload-management systems Local batch systems Distributed (Grid) systems Processing systems (backends) 3

Ganga job abstraction • A job in Ganga is constructed from a set of building blocks, not all required for every job Application Job Backend Input Dataset Output Dataset Splitter Merger 28 June 2006 What to run Where to run Data read by application Data written by application Rule for dividing into subjobs Rule for combining outputs 4

Framework for plugin handling Ganga. Object Executable 28 June 2006 ISplitter -exe -env -args IDataset User IApplication System Example plugins and schemas Plugin Interfaces • Ganga provides a framework for handling different types of Application, Backend, Dataset, Splitter and Merger, implemented as plugin classes • Each plugin class has its own schema IMerger -CE -requirements -id -status -reason -actual. CE -exitcode IBackend LCG 5

Applications and backends • Running of a particular Application on a given Backend is enabled by implementing an appropriate adapter component or Runtime Handler – Can often use same Runtime Handler for several Backend: less coding LHCb Experiment neutral Gauss/Boole/Brunel/Da. Vinci (Simulation/Digitisation/ Reconstruction/Analysis) Local PBS Executable ATLAS Athena. MC Athena (Production) (Simulation/Digitisation/ Reconstruction/Analysis) LSF OSG PANDA Nordu. Grid LHCb WMS US-ATLAS WMS 28 June 2006 Implemented Work in progress 6

Job repository • Job repository provides for storage and retrieval of job representations • User can choose to work with repository on local filesystem, or with repository on remote server that has certificate-based authentication – Implementation makes use of AMGA database interface AMGA interface for local database AMGA interface for remote database • API for local and remote repositories is the same, with CVS-like possibilities for job commit, checkout and update • Also have support for selections, bulk operations, and fast retrieval of summary data 28 June 2006 7

Job monitoring • Job monitoring is multi-threaded – Can set different refresh rate for different Backends • Actions initiated in monitoring threads include updating of job status in repository, and output retrieval for completed jobs 28 June 2006 8

Ganga Command-Line Interface in Python (CLIP) • CLIP provides interactive job definition and submission from an enhanced Python shell (IPython) – Especially good for trying things out, and understanding how the system works # List the available application plug-ins list_plugins( “application” ) # Create a Da. Vinci job to be submitted to DIRAC j = Job( application = “Da. Vinci”, backend = “Dirac” # Set the job-options file j. application. optsfile = “my. Opts. txt” # Submit the job j. submit() # Search for string in job’s standard output !grep “Selected events” $j. outputdir/stdout 28 June 2006 9

Ganga scripting • From the command line, a script my. Script. py can be executed in the Ganga environment using: ganga my. Script. py – Allows automation of repetitive tasks • Scripts for basic tasks included in distribution # Create an Athena job to be submitted to LCG ganga make_job Athena LCG test. py # Edit test. py to set Athena properties, then submit job ganga submit test. py # Query status, triggering output retrieval if job is completed ganga query Approach similar to the one traditionally used when submitting to a local batch system 28 June 2006 10

Ganga Graphical User Interface (GUI) • GUI consists of central monitoring panel and dockable windows • Job definition based on mouse selections and field completion • Highly configurable: choose what to display and how Scriptor Job details Logical Folders Job Monitoring 28 June 2006 Job builder Log window 11

Bringing Ganga to the users • Since July 2005, have had three Ganga tutorials for LHCb and two for ATLAS, in various locations CERN, September 2005 Cambridge, January 2006 Bologna, June 2006 • Approach of Grid. PP-supported LHCb-UK Software Course (January 2006), with Ganga/Grid session integrated in more-general course, very successful – Attract users who wouldn’t otherwise be considering the Grid • Ganga tried out by 100+ people, with positive feedback – “Very handy way to organise job submission” (ATLAS user) – “Clever and nicely designed” (LHCb user) • Small but growing group of people regularly using Ganga (also from a laptop) 28 June 2006 12

Successes in distributed analysis • Success of undergraduate project students in running LHCb analyses using the experiment’s distributed-analysis system reported in Grid. PP news item • System is based on LCG (Grid infrastructure), DIRAC (workload management layer and Ganga (user interface) • Together, project students and others in LHCb-Cambridge processed more than 75 million simulated beauty events over three-month interval • Fraction of jobs completing successfully averaged about 92% Did he say • Extended periods with success rate >95% 75 Excellent demonstration that Ganga allows million? physics analyses to be run easily on the Grid by people with no knowledge of Grid technicalities 28 June 2006 13

Ganga beyond ATLAS and LHCb • In EGEE, Ganga is used as submission engine and monitoring system for the DIANE job-distribution framework • Ganga/DIANE combination adopted for a number of applications • Geant 4 regression tests performed for major releases (twice per year) • Use of Grid in search for Search for differences in drugs against avian flu widely simulation results reported • Ganga/DIANE adopted for • About one eighth of jobs submitted using Ganga/DIANE running these tests on the Grid First use December 2005 Job statistics from Ganga 28 June 2006 • ITU Regional Radio Conference held in Geneva, May-June 2006 • Required real-time optimisation of evolving plan for sharing frequencies between 120 countries Maximise number of satisfied requests Minimise interference • Ganga/DIANE used to run optimisation jobs on the Grid 14

Conference contributions: July 2005 - June 2006 • AHM 2005(Nottingham, UK, September 2005) – Ganga user interface for job definition and management (K. Harrison) – Distributed analysis in the ATLAS experiment (C. L. Tan) AHM 2005 • (Milano, Italy, September 2005) – Ganga user interface for job definition and management (D. Liko/K. Harrison) • (Mumbai, India, February 2006) – Ganga: a Grid user interface (K. Harrison) – Experience with distributed analysis in LHCb (U. Egede) • ISGC 2006 (Taipei, Taiwan, May 2006) – Ganga: a job management and optimising tool for job submission to the Grid (A. Maier) 28 June 2006 15

Conference contributions: coming attractions • • (Geneva, Switzerland, July 2006) – Using Python in the Development of a Grid user interface for distributed data analysis (A. Soroko) AHM 2006(Nottingham, UK, September 2006) – Ganga: a Grid user interface for distributed analysis (A. Soroko) – Distributed analysis in the ATLAS experiment (C. L. Tan) 28 June 2006 16

Conclusions • Excellent progress with Ganga development since redesign (early 2005) • Wealth of functionality has been implemented – Support for Applications and Backends of interest to ATLAS and LHCb • Work in progress on ATLAS-specific Backends: PANDA and Nordu. Grid – Possibilities for working at the command line, with scripts, and through a graphical interface – Job monitoring, local/remote repository, job splitting, and more • Work on data handling delayed because of uncertainties in the experiments, but is now one of the top priorities • Several highly successful Ganga tutorials have been held: more to come • Ganga has allowed high-statistics LHCb physics studies to be performed on the Grid by people with no knowledge of Grid technicalities • Ganga used for a range of applications beyond ATLAS and LHCb 28 June 2006 17