Raw Wallclock in APEL John Gordon STFCRAL WLCG

Raw Wallclock in APEL John Gordon, STFC-RAL WLCG Accounting Task Force, 15 th June 2017 John Gordon

• Raw Wallclock Time is the only additional requirement for APEL that has been agreed to date. • Mention has been made of multiple benchmarks but more clarity is needed and perhaps some limitation as APEL cannot see how to accommodate an open set of benchmarks. • These slides relate only to Wallclock. John Gordon

Outline • • Current State Options Preferred Option Other Work John Gordon

Current State • APEL receives data from a number of sources • • • CREAM-CE via Apel Client Other CEs directly (eg ARC CE) Extracted from site databases as jobs (? ? ) Extracted from site databases as summaries (CERN, NIKHEF) Extracted from e-infrastructure databases (Nordu. Grid, OSG) APEL sends summaries of all data to the accounting portal for visulaisation. For some of this data the wallclock value has been scaled internally by the underlying batch system and does not reflect the true wall time. This is independent of the publishing method. User jobs internally report raw cpu time and wallclock as seen by the cpu and not the batch system. For comparison with user-recorded data the experiments have asked APEL to record the real wallclock time. John Gordon

• The APEL Job Record schema has fields for Start. Time, and End. Time, as well as Wall. Duration • So, Raw. Wall. Duration can be calculated as End. Time-Start. Time • This could be done on the server for all jobs for which APEL holds job records. • Summaries don’t contain individual start and end times, just the sum of Wall. Duration so a change will be required on the client. John Gordon

Option 1 a) Extend the Job. Record to include an extra field Raw. Wall. Duration b) Extend the Summary Record to include the sum of Raw. Wall. Duration c) Summarising will sum the values from Job Records. a) Summarising on the client will include this sum in the summaries sent by sites. b) Summarising on the server will sum over job records sent. John Gordon

Option 2 a) Extend the summary record to include Raw. Wall. Duration b) Fill this at summary time by Raw. Wall. Duration=sum(End. Time-Start. Time) c) This works on client and server who both use the same summary code. No change to Job Record d) New client software only required at sites who send summaries. Server does it all for sites who send jobs (the majority) John Gordon

Preferred Option • Option 2 John Gordon

Other Work • Once the summary records contain Raw. Wall. Duration the portal will need to accept them • … and present the data. • Assumption is that the data will be provided as a metric in addition to the current Wall. Duration and Normalised. Wall. Duration. • Will not change any existing reports. • Sites cutting their own summaries will need to extend the schema with Raw. Wall. Duration and fill it. – CERN, OSG, Nordu. Grid • Sites cutting their own Job Records need do nothing – ARC John Gordon

Summary • A solution looks feasible • APEL would prefer to modify the summary process and not change job records • The majority of sites would need to do nothing. – Sites publishing summaries would need an updated client. – VOs who value this data could persuade their sites to update client. John Gordon