ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on

ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on behalf of the TAG developers group ATLAS Software and Computing Week, 4~8 April, 2011

TAGS • TAGS are produced as files. These files are uploaded to a database by the Tier 0. – General (with links to TAG monitoring): https: //atlas-tz-monitoring. cern. ch/ – TAG Task Lister (upload and post. Tag. Upload tasks): https: //atlas-tz-monitoring. cern. ch/tag. Interface. html – TAG Dataset Lister: https: //atlas-tz-monitoring. cern. ch/tag. Lister. html • Details tracked in the TASK database – Data distribution, mapping to datasets, sites, etc. – Service installations and usages • Other metadata is also needed for a meaningful event selection. – COMA: triggers, periods, streams, etc. Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

BACKBONES—Infrastructure development Responsible: Florbela Viegas, Elizabeth Gallas and Elisabeth Vinek • Databases (Oracle) Routine uploads of TAGs and other metadata info (TASK and COMA [COditions Meta dat. A]) • Topology CERN and several Tier-1 and Tier-2 sites (BNL, TRIUMF, DESY, PIC, RAL, and …) are hosting different sets (or subsets) of TAG data with or without overlaps and replicates of CATALOG and COMA data • Data Volume and Complexity Grow/evolve and pose technical challenges on code flexibility and scalability, affect all the services Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 3

TASK COMA Event Metadata Collection Registry Event-File Map Sites File Registry Service Registry DDM TASK Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 Runs POOLTAG COnditions Metadat. A

ELSSI Services Suite • Web Tools – i. ELSSI – Run. Browser (used by i. ELSSI) • Web Services – GUID Lookup (used by Pathena) – Trigger decoder (used by i. ELSSI, Event. Selector in Athena) – Token Extraction (used by i. ELSSI, Ganga) – Skimming – Lumirange calculation (used by i. ELSSI, being factorized) –… • https: //twiki. cern. ch/twiki/bin/view/Atlas/ELSSIServices. Suite Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

i. ELSSI--NOT just a browser 1. Does provide browsing functions ü Metadata info on full streams ü Triggers/Runs ü Physics metadata from AOD ü Event flags, e. g. Higgs candidate 2. Performs statistics on these metadata ü Are there enough dileptons with 2 forward jets to do an analysis requiring that? ü What is the overlap of the muon and egamma streams and what are the properties of those events? ü What is the data reduction if I want to build a DAOD stream of only the digamma events that pass triggers this and that? Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 6

i. ELSSI--NOT just a browser Also provides an integrated, full-chain service by: 1. interconnecting with multiple DBs and sites (setting the right parameters for db handshakes) 2. Integrating GRL, JO files with event selection/skim 3. building efficient queries (valid attributes, data streams, triggers and runs/periods) 4. interfacing with ELSSISuite services 5. building inputs to Athena or DA tools such as Ganga or Pathena. 6. Submitting grid jobs on selected events to produce ntuples or skims of AOD, ESD or RAW. Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 7

Trigger decoder service • Map a trigger name to the appropriate bit in the TAG trigger words • Provide the SQL fragment (e. g. , operation="bitand(EFPassed. Trig. Mask 0, power(2, 1))") for programs using the TAG database • Clients: – i. ELSSI when querying triggers on the database – Event. Selector in Athena--TAG file analyses to decode the trigger words in the TAG files. • trigger. Decoder. Url: https: //atlas-tagservices. cern. ch/tagservices/trigger. Decoder Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Trigger decoder service • Usage: – From a browser: trigger. Decoder. Url/get. Decoded. Trigger. Info. php? trignms=EF _e 10_loose&runrange=162882&phys_raw=raw&filenam e_tag=data 10_7 Te. V&ami_tag=f 287_m 588_p 250_p 160 – From a command line or within a program: curl -s -S --key server_hotkey --sslv 3 --cert server_hotcert -k --url "trigger. Decoder. Url/get. Decoded. Trigger. Info. php? trignms=E F_e 10_loose&runrange=162882&phys_raw=raw&filename _tag=data 10_7 Te. V&ami_tag=f 287_m 588_p 250_p 160" Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Token Extraction service • Constructs a ROOT collection of selected events which can then be retrieved from the /afs space at CERN or directly from the web. • Times can vary between seconds and minutes depending on the size of your selection (can be checked beforehand in i. ELSSI). • A link to the results will be emailed to you or it will also appear in i. ELSSI when the job completes. The email will contain all the information on the inputs and outputs of the job which can be browsed or wget-ed. – Configuration (in xml) – Log files – Root Collection • At the end of the Extract results, it gives instructions for running the command line or portable client version of Extract. Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Token Extraction service invoked by i. ELSSI Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Skimming service • https: //twiki. cern. ch/twiki/bin/viewauth/Atlas/Skimming. Service#Running_a_Manu al_Skim • Skim the events selected in your ROOT collection • Supply with status updates as it progresses through skim configuration, submission and completion (or failure). • Create scripts and send them to the Ganga Service • The skim does two jobs: – ‘Tag Prepare’ job finds (using a guid lookup) the datasets the selected events belong to. – ‘Tag Run’ job. The chosen AOD Athena JO is then run on the LCG on these datasets. The Tag run job is submitted to the Grid. • Email notice with the name of the resulting dataset stored in dq 2 (e. g. user 09. thomasdoherty. ganga. users. thomasdoherty. ganga. 7. 2010041 3/) will be made available with a dq 2 -get. Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Skimming service invoked by i. ELSSI Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

TAGs for analysis? • TAGs have proven extremely useful in commissioning and in early collision data • TAGs discovered and quantified the scope of lumi blocks being split across multiple files in 2010 reprocessings – Problematic for cross-section calculation • TAGs discovered the event duplication issues coming from TDAQ in early 2011 collision data • If you have used event picking, you have used TAGs • TAGs have identified anomalous and/or problematic events • TAGS have been useful in understanding the overall cross-stream statistics of our data • TAGs provide a quick means of checking whether your analysis needs to process the debug stream • TAGs for analysis? Needs involvement of PAT group Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

SUMMARY ELSSISuite services have been developed on top of an integrated worldwide database system to provide extended selection possibilities You are invited to more TAG related talks at the DB session this afternoon Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 QIZHI ZHANG 15

Thank you! Supporting slides or a live i. ELSSI demo Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Period-based selection-1 1 2 3 4 5 1 2 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 17

period-based selection-2 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 18

Count per run/per trigger-1 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 19

Count per trigger-2 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 20

Stream mapped with runs Without run range Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Perform queries of different project-1 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Perform queries of different project-2 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Perform queries of different project-3 D F Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011 E

Synchronized editing-1 B C A Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Synchronized editing-2 C Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Download GUIDs into a text file-1 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011

Download GUIDs into a text file-2 Qizhi Zhang, Software and Computing Week, 4 -8 April, 2011