Data Periods in COMA Elizabeth Gallas Oxford February
Data Periods in COMA Elizabeth Gallas - Oxford February 14, 2011 Feb 2010 Elizabeth Gallas - COMA / Run. Browser
Outline Purpose TODAY: talk about ATLAS Data Periods (entry, storage, reports and services) § What is a “Data Period” ? What is “COMA” ? § § Future implementation § § § Current implementation Schema proposal What has been done so far ? What is needed ? (interfaces: Entry, Report, Services) Timescale Other unresolved issues: § § § Loading issues Production location Replication plan Feb 2010 Elizabeth Gallas - COMA / Run. Browser 2
What is a Data Period ? § Data Period is a set of ATLAS Runs grouped for a purpose. § § Used in ATLAS data processing, assessment, and selection … Each Period is designated a name … concatenation of § Project name (i. e. ‘data 09_900 Ge. V’) § Period name (i. e. ‘C 1’, ‘C 2’, ‘C’, ‘All. Year’ …) § § But proceeded sometimes with ‘period’ (i. e. ‘period. C 1’) § propose dropping this in DB implementation Current implementation: § Resides in files on afs at CERN (next slide) § Edited by hand by Data Prep Coordination (experts) § Expert human entry is unavoidable … many factors Structure evolved over last year with experience § Many systems read these AFS files including COMA loading … other systems use them from COMA. § § Plan described here: § § Use COMA as DB Storage behind current/future Data Periods Define entry and report services to provide Period info Feb 2010 Elizabeth Gallas - COMA / Run. Browser 3
/afs/cern. ch/atlas/ www/GROUPS/ DATAPREPARATION/ Data. Periods Data Period AFS files to date data 10_7 Te. V. period. All. Year. runs. list data 10_7 Te. V. period. A 1. runs. list data 10_7 Te. V. period. A. runs. list data 10_7 Te. V. period. B[1 -2]. runs. list data 10_7 Te. V. period. B. runs. list data 10_7 Te. V. period. C[1 -2]. runs. list data 10_7 Te. V. period. C. runs. list data 10_7 Te. V. period. D[1 -6]. runs. list data 10_7 Te. V. period. D. runs. list data 10_7 Te. V. period. E[1 -7]. runs. list data 10_7 Te. V. period. E. runs. list data 10_7 Te. V. period. F 1. runs. list data 10_7 Te. V. period. F 2. runs. list data 10_7 Te. V. period. F. runs. list data 10_7 Te. V. period. G[1 -6]. runs. list data 10_7 Te. V. period. G. runs. list data 10_7 Te. V. period. H[1 -2]. runs. list data 10_7 Te. V. period. H. runs. list data 10_7 Te. V. period. I[1 -2]. runs. list data 10_7 Te. V. period. I. runs. list Feb 2010 Each file contains simply a list of Run numbers [n-m] == range of files Note: period C contains all Runs of period C 1 and C 2; data 09_2 Te. V. All. Year. runs. list data 09_900 Ge. V. All. Year. runs. list data 10_7 Te. V. period. Vd. M[1 -3]. runs. list data 10_7 Te. V. period. Vd. M. runs. list data 10_900 Ge. V. period. B. runs. list data 10_hi. period. J[1 -7]. runs. list data 10_hi. period. J. runs. list Elizabeth Gallas - COMA / Run. Browser 4
What is COMA ? COMA: short name for “COnditions Metadat. A”: § part of general effort to consolidate/relate ATLAS Metadata. 3 Components of the COMA Project: § COMA Database Tables § A collection of selected/derived Run/LB-wise Conditions § § Source: mostly ATLAS Conditions DB and Trigger DB Originally, built to support dynamic queries of TAG DB Now, also a standalone system with interfaces (below) Run. Browser Package: § run. Browser. Report – A report interface for COMA tables § § But AMI now has access … can also report … run. Browser – interface for finding Run/LB of interest § run. Browser is another related interface in development NOTE: in diagrams, all table and view names start w/’COMA_’ (i. e. RUNS table name is COMA_RUNS) Feb 2010 Elizabeth Gallas - COMA / Run. Browser 5
SOURCES COMA Schema Overview RUN_TYPES STREAM_LBN_EVENTS LOAD STATUS EXPLAINS MASKS STREAM_EVENTS RUN_MAGNETS* d te a l e r d Perio RUN_FILLS* PERIOD_OBS PERIODS SHORTNAMES EVENTS RUN_PERIODS FILLS* TRANSITIONS Master_Keys LBN_TIMES RUNS Run_Agg_PS RUN_LUMS LBN_LUMS LUM_DETS COOL_CONNECTIONS MK_Triggers VIRTUAL_FLAGS Trig_Agg_PS Prescale_Keys VFLAG_DETS MKT_Physics PS_Sets MKT_Others Trig_Stream STREAM Feb 2010 LVL 1_PSK LVL 1_PS DETSTATUS CODES HLT_PSK HLT_PS DETECTORS Elizabeth Gallas - COMA / Run. Browser GROUPS 6
2010 implementation 1. RUNS RUN_INDEX 3. PERIOD_RUNS P – PERIOD_NAME P – RUN_INDEX 2. PERIODS P – PROJECT_PERIOD_NAME FILENAME_TAG PP_FILE_TIME PP_START_TIME PP_END_TIME PP_RUN_STRING PP_DESC PERIOD_RANK Current Periods in COMA Current COMA Tables related to Periods: 1. RUNS: table of all COMA Run Numbers 2. PERIODS: table of period names to date 3. PERIOD_RUNS: Runs in each Period 4. PERIOD_OBS: attempt to archive old records when periods changed (or catch human error) New implementation goals: § Make {entry, storage, report} more robust. § Some periods: are a group of periods § § 4. PERIOD_OBS P – CPO_INDEX PROJECT_PERIOD_NAME FILENAME_TAG PP_FILE_TIME Feb 2010 Want each piece of information in the schema only once and more robustly § § Eliminate: PP_RUN_STRING, PERIOD_RANK Need better system for obsolete entries § § Want new tables to reflect this hierarchy Allow flexibility to data entry Improve indexing; better track insert/updates; § AMI needs to know when entries change Elizabeth Gallas - COMA / Run. Browser 7
1. RUNS Current Proposed: Data Periods in COMA (tables) (same) RUNS P -- RUN_INDEX 3. PERIOD_RUNS PERIOD_P 1_TO_RUNS (P_LEVEL = 1) P – PERIOD_NAME P – RUN_INDEX PF – P_INDEX PF – RUN_INDEX (P_LEVEL = 2, 3) PERIOD_DEFS P – P_INDEX F – PDOM_NAME 2. PERIODS P – PROJECT_PERIOD_NAME FILENAME_TAG PP_FILE_TIME PP_START_TIME PP_END_TIME PP_RUN_STRING PP_DESC PERIOD_RANK So 3 levels of periods (can be used like RANK): P_LEVEL = 1 is a Period, 2 is Period Group, 3 is Parent Group. Feb 2010 PERIOD_PHI_TO_PLOS PF – PHI_INDEX PF – PLO_INDEX P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS (add reference to GRLs) PERIOD_GRLS PF – P_INDEX P – GRL_NAME PGRL_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE (add ‘Domain’ dimension and people and roles for Period Entry) PERIOD_DOMAINS PERIOD_DOMAIN_ROLES P – PDOM_NAME PDR_INDEX PDOM_NAME PDR_USER PDR_ROLE PDOM_DESC Elizabeth Gallas - COMA / Run. Browser 8
Definitions PERIOD_DEFS P – P_INDEX F – PDOM_NAME P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD_DEFS table § Current Periods are from Data. Prep domain § PDOM_NAME = “DP” § Contains Period definitions (P_LEVEL=1, 2, 3) § P_LEVEL = 1 (a set of Run Numbers) § § P_PROJECT P_PERIOD P_DESC P_STATUS § P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE § § § A Period Group is a set of Level 1 periods e. g. : ‘D’ contains ‘D 1’-’D 6’ Names: alpha with no number P_LEVEL = 3 § § § Feb 2010 P_LEVEL = 2 § § e. g: ‘D 1’, ‘D 2’, . . . ‘D 6’ … Names: alpha followed by a number Parent Period Group: set of Level 2 periods e. g. : ‘All. Year’ contains ‘A’, ‘B’, … ‘I’ P_DESC: written description P_STATUS: ‘locked’ or ‘unlocked’ § ‘locked’ == no more members can be added Elizabeth Gallas - COMA / Run. Browser 9
New implementation status § New COMA Period Tables, Views (next slide) have been defined § In ATLAS_TAGS_METADATA_SG@INT 8 R § Table/Column descriptions: https: //gallas. web. cern. ch/gallas/Table. Doc_COMA. html but some minor modifications not checked in yet … sorry § § § Legacy data has been loaded: Old New tables Some new run. Browser. Reports show results (my php dev area): § r. BR: Data Period Report input menu § § r. BR: Multi-Run Report for Period J (project ‘data 10_hi’): § § https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run. Browser/trunk/ run. Browser. Report/run. Browser. Report. php? pn=J%&fnt=data 10_hi r. BR: Front page for General run. Browser. Reports: § § https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run. Browser/trunk/ run. Browser. Report/r. BR_Period_Report. php https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run. Browser/trunk/ run. Browser. Report/run. Browser. Report. html r. BR Documentation: § § https: //twiki. cern. ch/twiki/bin/viewauth/Atlas/Conditions. Metadata https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run. Browser/trunk/ run. Browser. Report/r. BR_Documentation. html not updated yet wrt new period implementation Feb 2010 Elizabeth Gallas - COMA / Run. Browser 10
RUNS P -- RUN_INDEX PERIOD_DEFS Proposed: Data Periods and Period Groups (tables and views) PERIOD_P 1_TO_RUNS PF – P_INDEX PF – RUN_INDEX P – P_INDEX F – PDOM_NAME P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE V_PERIOD_RUNS PDOM_NAME P_LEVEL P_INDEX P_PROJECT P_PERIOD RUN_INDEX PHI_LEVEL PHI_INDEX PHI_PROJECT PHI_PERIOD P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS V_PHI_PLOS PERIOD_PHI_TO_PLOS PF – PHI_INDEX PF – PLO_INDEX PLO_PROJECT PLO_PERIOD_GRLS V_P 3_RUNS PF – P_INDEX P – GRL_NAME PGRL_STATUS PDOM_NAME P – PDOM_NAME P 3_INDEX P 3_PROJECT P 3_PERIOD PDOM_DESC PDOM_NAME P 2_INDEX P 2_PROJECT P 2_PERIOD P 1_INDEX P 1_PROJECT P 1_PERIOD RUN_INDEX PERIOD_DOMAINS PERIOD_DOMAIN_ROLES PDR_INDEX PDOM_NAME PDR_USER PDR_ROLE Feb 2010 V_P 2_RUNS V_P 1_RUNS Elizabeth Gallas - COMA / Run. Browser 11
Open Issues § Schema includes: Opportunity for other Period Domains § Right now, all Periods in PDOM_NAME = DP (“Data Preparation”) § § Do other groups want to define sets of runs § § § e. g. alignment? create Domain = AL (“Alignment Group”) PERIOD_GRLs (table) § Current TWiki includes Period to GRLs, so I added this table § This could evolve to include more tables for DQ/GRLs PERIOD_REQUIREMENTS (column ? ) § A new column or table in the schema (not existing now) § § In all interfaces, this domain is the default (when unspecified) Each P_LEVEL=1 period will in future have a set of Conditions requirements (i. e. stable beams, measured luminosity) … from which candidate Runs for that Period can be generated. I am unsure what form it will take … postpone until more information is in place … just basic entry is needed for now w/o this feature 3 tables (%_TO_% and %_GRLS): primary keys w/ multi-column index § AMI require these to have (additional) single-column index? Any other suggestions for schema ? Feb 2010 Elizabeth Gallas - COMA / Run. Browser 12
Period Information Service Interfaces Feb 2010 Elizabeth Gallas - COMA / Run. Browser 13
Period Info Service: needed by other systems Various systems use Data Period information: § From COMA: TAG services using COMA can continue to use COMA § From AFS files: Need to provide the same information to those systems (put the AFS files behind us) § Create py. AMI (or other ? ) service … Steps: § Agree on the schema § § Contact developers: run. Query, Data. Quality, Data. Prep … (more ? ) Agree on basic services: input, output and format 1. “Period to Run service”: § Input Project, Period Name § Output: list of Run Numbers 2. “Run to Period service”: § Input: Run Number (and possibly P_LEVEL) § Output: list of Period names containing that Run § § § Feb 2010 Develop these services with existing data Give instructions to system how to use new service. Timescale: ASAP ! Give other systems time to adjust before mid-march 2011 collisions Elizabeth Gallas - COMA / Run. Browser 14
Entry Interfaces Feb 2010 Elizabeth Gallas - COMA / Run. Browser 15
Data Period Entry Interfaces PERIOD_DEFS 1. Insert a new Data Period definition (PERIOD_DEFS table) § P – P_INDEX F – PDOM_NAME § (P_PROJECT_PERIOD = P_PROJECT + ‘. ’ + P_PERIOD) P_LEVEL ( 1, 2, 3 ) P_PROJECT_PERIOD P_PROJECT P_PERIOD P_DESC P_STATUS P_CREATE_USER P_CREATE_DATE P_MODIFY_USER P_MODIFY_DATE User authorization determines Domain (PDOM_NAME) User enters: § P_Level, Project, Period name, description § 2. Initial status is ‘unlocked’ Add members to an existing Period 1. Level 1: Add Runs to Periods (PERIOD_P 1_TO_RUNS) 2. 3. Runs must be in COMA … provide users with a list of available Runs Level 2: Add Periods to Groups (PERIOD_PHI_TO_PLOS) Provide user with Level 1 Periods candidates Level 3: Add Period Groups to Parent Periods (PERIOD_PHI_TO_PLOS) Provide user with Level 2 Period candidates 3. Update a Data Period definition (PERIOD_DEFS) 1. 2. User can change P_DESC (description) at any time User can set P_STATUS from ‘unlocked’ to ‘locked’ Timescale: Before mid-March First 2011 collisions 4. Associate a Period with a GRL (PERIOD_GRLS) 1. Would be good to integrate with GRL system information for integrity Feb 2010 Elizabeth Gallas - COMA / Run. Browser 16
Entry Interface Issues § Data Period Entry users must be experts § Experts must be aware of naming restrictions § § If those restrictions can be coded, specify them Certification options (2): 1. Need grid certificate on browser … in ATLAS VO 2. AMI login can identify users with particular roles § Special writer account ? § Feb 2010 For Data Period related tables in COMA Elizabeth Gallas - COMA / Run. Browser 17
User Report Interfaces Feb 2010 Elizabeth Gallas - COMA / Run. Browser 18
Reporting Services for Users § run. Query § § run. Browser and run. Browser. Report § § Will continue to include Period information (eg next slide) § Adjust to new schema A preliminary run. Browser. Report contains information as in the TWiki Data. Periods (next slide) § § § switch from AFS file usage to new Data. Period service utility I developed this to see the data entered § Assist with schema development, facilitate communication Could be maintained … better optimized, organized … more information like luminosity, events added The other option is to have this in AMI pages Feb 2010 Elizabeth Gallas - COMA / Run. Browser 19
RBR Multi-Run Report: Contains: basic information for the runs matching the input criteria Input criteria: printed in the header Click on a specific Run link: RBR Single Run report Click on duck: RBR documentation Links in Table column headers: Short description of column Hover link: Indicates what will happen Click on SMK link: RBR Trigger Super Master Key Report Feb 2010 Elizabeth Gallas - COMA / Run. Browser 20
Period Documentation Front Page Purpose: Generate Period documentation for chosen input criteria Click on -/+ links: Expands to show The Periods Period Groups, Parent Periods § https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run Browser/trunk/run. Browser. Report/r. BR_Period_Report. php Feb 2010 Elizabeth Gallas - COMA / Run. Browser 21
Example Period Document (like TWiki) Prototype ! Format can be improved… § https: //voatlas 69. cern. ch/tagservices/dev/users/gallas/Run Browser/trunk/run. Browser. Report/r. BR_Period_Report. php? pd=DP&fnt=data 10_hi Feb 2010 Elizabeth Gallas - COMA / Run. Browser 22
Other unresolved issues § COMA Run loading -- now more time critical § § § Data Periods must be directly loaded in the COMA master COMA must contain a Run to be included in a Data Period ! COMA loading cron executes daily on ATM@INT 8 R § § INT 8 R is an integration database at CERN Loading program: latest Runs in the TAG Catalog Schema § Usually within a day of Tier-0 processing, soon enough? § § § Modify Run loading to use Tier-0 DB rather than Catalog ? Exports COMA tables to ATLARC then TAG remote sites Production location issue: § TAG Catalog recently moved to ATLR § COMA should move there too § § Revise replication plan ? § § Feb 2010 critical service should not be on the archive database ATLARC. Data Prep should be mindful that Period replication not automatic I don’t think we can use Oracle streams … schema is evolving Elizabeth Gallas - COMA / Run. Browser 23
Thanks and Apologies § § Thanks to Beate and Solveig for useful discussions. Apologies for § Things I forgot § § I have some notes not cross checked to write slides … After the meeting, I propose to update slides § § § Belated meeting announcement Belated sending of links § Was hoping to send § § And let you know I have done so more finalized schema Links in production locations More complete documentation BUT this schema is in development § Feb 2010 Good point to get feedback now ! Elizabeth Gallas - COMA / Run. Browser 24
Backup Feb 2010 Elizabeth Gallas - COMA / Run. Browser 25
run. Browser. Report: General Principles run. Browser. Report Purpose: Report interface for COMA § § § Display what COMA knows about Runs § Useful for experts/users (also for COMA loading and cross checks) Provide links to more detailed information (in other systems) § run. Query, AMI, Trigger, Data Quality …reports using COOLCherry. Py Provide reports distinctive from other systems § (Avoid duplication of effort) § Display COMA ‘derived’ information § § Not available in other systems run. Browser. Report php uses GET methods: § Inputs are not complex, user can modify URL to get report they want, put links into TWiki or other documentation https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § Currently: ELSSI and run. Browser generate links to run. Browser. Reports for selections in those systems … links in AMI ? ? ? Feb 2010 Elizabeth Gallas - COMA / Run. Browser 26
run. Browser. Report: Reports available (example links) RBR Run Report (Run 161562): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § General Run information, AMI tags, DQ, Prescale Evolution, Triggers § Trigger tables show derived Run-integrated prescale, passthrough RBR Master Key Report (Key 877): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? smk=877 § Runs using this SMK, physics/commissioning chains, Level 1 items RBR Chain Report (Chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 11_etcut § § Summary: SMK, Runs, Signature appearances of this chain Runs with this chain ‘active’ RBR Chain Wildcard Report (chain name matching string EF_g 1%): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 1% § Summary: SMK, Runs with this chain, activation in Runs RBR Prescale Report (Run 162882 Prescales for chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=162882&cn=EF_g 11_etcut § Derived chain activity over the Run and LB-wise prescales Feb 2010 Elizabeth Gallas - COMA / Run. Browser 27
COMA run. Browser. Report Index page § § INDEX: https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. html GUIDE: https: //atlas-tagservices. cern. ch/RBR/r. BR_Documentation. html “Index” page: § Enter any input quantity, § IE: a Run number § IE: a Chain Name § Click on submit (or return key) Links at bottom of every page: RBR reports use “GET” methods: Modify the URL to change input RBR reports contain links No need to use index page for every query Links lead you to other COMA reports other external reports AMI, run. Query, Trigger. DB… Feb 2010 Bottom of each page 28 Elizabeth Gallas - COMA / Run. Browser
RBR: Single Run Overview General Run info § Period, LB range, Date … § Links to COMA reports Links to external reports § Yellow sections expand (5): § AMI tags for this Run § § In the TAG Catalogue DQ LBSUMM assessments COOL tagged/locked Active Streams Prescale Evolution § PS Key changes Trigger section § HLT summary § HLT subsections § § § Open this section in next slide… Feb 2010 § § Physics Commissioning Level 1 Items Elizabeth Gallas - COMA / Run. Browser 29
- Slides: 29