Tools for performance reports production and its automation

  • Slides: 23
Download presentation
Tools for performance reports production and its automation Performance Taskforce 7 -12 -2007 SFT

Tools for performance reports production and its automation Performance Taskforce 7 -12 -2007 SFT group Meeting Danilo Piparo 1

Contents • • Introduction: what’s a performance report? The CMSSW Performance Taskforce standard candle

Contents • • Introduction: what’s a performance report? The CMSSW Performance Taskforce standard candle Profilers Profile data elaboration – Perfreport and evolution • The automation through relvalreport_v 2 7 -12 -2007 SFT group Meeting Danilo Piparo 2

The performance report Df A set of informations pertaining an aspect of one application’s

The performance report Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way 7 -12 -2007 SFT group Meeting Danilo Piparo 3

The performance report Process Df A set of informations pertaining an aspect of one

The performance report Process Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way 7 -12 -2007 SFT group Meeting Danilo Piparo 4

The performance report Profiler Process Df A set of informations pertaining an aspect of

The performance report Profiler Process Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way 7 -12 -2007 SFT group Meeting Danilo Piparo 5

The performance report Profiler Process Df A set of informations pertaining an aspect of

The performance report Profiler Process Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way 7 -12 -2007 SFT group Meeting Danilo Piparo Profile 6

The performance report Profiler Process Df A set of informations pertaining an aspect of

The performance report Profiler Process Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way Profile analysis tool 7 -12 -2007 SFT group Meeting Danilo Piparo 7

The performance report Profiler Process Df A set of informations pertaining an aspect of

The performance report Profiler Process Df A set of informations pertaining an aspect of one application’s performance, stored and accessed in a suitable way Profile analysis tool Report 7 -12 -2007 SFT group Meeting Danilo Piparo 8

Profilers Profiler Process Profile analysis tool Report 7 -12 -2007 SFT group Meeting Danilo

Profilers Profiler Process Profile analysis tool Report 7 -12 -2007 SFT group Meeting Danilo Piparo 9

The choice of the process • • • Not trivial problem: several candles for

The choice of the process • • • Not trivial problem: several candles for one application? Measure of disk, CPU and memory usage of CMSSW QCD Pt 20 -30 Ge. V – Performance Task Force Standard Candle Performance measurement shares with release validation several aspects New tool to tailor candles and perform Relval Py. Release. Validation 7 -12 -2007 • Introduce needed flexibility: tailor candles • Perform CMSSW Rel. Val • First use of Python for configuration (test and improve) SFT group Meeting Danilo Piparo 10

Profilers - 2 • • • Ig. Prof: Memory (different aspects) and Speed Val.

Profilers - 2 • • • Ig. Prof: Memory (different aspects) and Speed Val. Grind. FCE: Number of instructions, full callstacks (R. Moser, Performance Taskforce) Memcheck: Memory leaks Time Report: Cms. Run specific – time spent by modules No Process … Edm. Size: Size on disk of objects stored in rootfiles … But Skimming make. Skim. Driver. py: • Python for cms. Run configuration • Wrapper scheme • Select from command line which cathegory has to be skimmed (AOD, AODSIM …) e. g. : make. Skim. Driver. py –i myinfile. root –o myskimmedfile. root –n 100 –datatype AOD … Soon part of Configuration/Eventcontent … 7 -12 -2007 SFT group Meeting Danilo Piparo 11

Profile analysis and reports Profiler Process Profile analysis tool Report 7 -12 -2007 SFT

Profile analysis and reports Profiler Process Profile analysis tool Report 7 -12 -2007 SFT group Meeting Danilo Piparo 12

Profile analysis and reports • • • Perfreport: Ig. Prof, Valgrind. FCE, Edm. Size

Profile analysis and reports • • • Perfreport: Ig. Prof, Valgrind. FCE, Edm. Size Valgrind. Memcheck. Parser: Memcheck Timereport. Parser: Timereport Ig. Prof. Analysis: Ig. Prof Memory Simple. Memcheck. Parser: Memory growth HTML reports Perfreport (PR): a tool for performance report production • Developed within the Performance Taskforce (R. Moser) • Html Reports • Graphics production • Regression between different profiles • Customisable filters in XML format • … Even More … 7 -12 -2007 SFT group Meeting Danilo Piparo 13

Perfreport Evolution 2 1 KCachgrind: - No saving PR 1: PR 2: + HTML

Perfreport Evolution 2 1 KCachgrind: - No saving PR 1: PR 2: + HTML Reports 3 PR 3: + Ig. Prof, Edmsize + Plugins: Front. Page - Only Valgrind + Regression + Db filling - KDE + Name Resolution + Server side web application + Full call stacks 7 -12 -2007 SFT group Meeting Danilo Piparo 14

Perfreport Evolution – PR 1 7 -12 -2007 SFT group Meeting Danilo Piparo 1

Perfreport Evolution – PR 1 7 -12 -2007 SFT group Meeting Danilo Piparo 1 15

Perfreport Evolution – PR 2 2 7 -12 -2007 SFT group Meeting Danilo Piparo

Perfreport Evolution – PR 2 2 7 -12 -2007 SFT group Meeting Danilo Piparo 16

Perfreport Evolution – PR 3 3 7 -12 -2007 SFT group Meeting Danilo Piparo

Perfreport Evolution – PR 3 3 7 -12 -2007 SFT group Meeting Danilo Piparo 17

Relvalreport_v 2 • Common interface to profilers and profile crunchers • Configuration/Py. Release. Validation/test

Relvalreport_v 2 • Common interface to profilers and profile crunchers • Configuration/Py. Release. Validation/test • Batch mode • Used also by simulation and Rel. Val teams Profilers Relvalreport_v 2. py Report relvalreport_v 2. py –c ‘cms. Run mycfg. cfg’ … To know more: SWGuide. Relval. Report 7 -12 -2007 Crunchers Profilers SFT group Meeting Danilo Piparo 18

Summary • • • Large variety of profilers and profile analysers Perfreport: wide range

Summary • • • Large variety of profilers and profile analysers Perfreport: wide range of report types and functionalities Relvalreport_v 2: excellent tool for automated execution of performance benchmarks and report production 7 -12 -2007 SFT group Meeting Danilo Piparo 19

Backup 7 -12 -2007 SFT group Meeting Danilo Piparo 20

Backup 7 -12 -2007 SFT group Meeting Danilo Piparo 20

XML candles files 7 -12 -2007 SFT group Meeting Danilo Piparo 21

XML candles files 7 -12 -2007 SFT group Meeting Danilo Piparo 21

A concrete example - 1 scramv 1 project CMSSW_1_7_0 cd CMSSW_1_7_0 eval `scramv 1

A concrete example - 1 scramv 1 project CMSSW_1_7_0 cd CMSSW_1_7_0 eval `scramv 1 run -sh` export CVSRSH=ssh export CVSROOT=: kserver: cmscvs. cern. ch: /cvs_server/repositories/CMSSW cd src cvs co -r. V 03 -04 -06 FWCore/Parameter. Set cvs update -A FWCore/Parameter. Set/python/Config. py Fixed Soon! scramv 1 b cd - ; cp -r ~dpiparo/public/Luca/test_skim_py. ; cd test_skim_py. /make. Skim. Driver. py -i MU-_10_ALL. root -o MU-_10_ALL_skimmed. root --outputcommands AODSIMEvent. Content -n 1 FEVTEvent. Content RECOEvent. Content • Just one event. . AODEvent. Content FEVTSIMEvent. Content • Event content could have been: RECOSIMEvent. Content AODSIMEvent. Content The blocks defined in FEVTSIMANAEvent. Content. cff RECOSIMANAEvent. Content (checked at RUNTIME!) AODSIMANAEvent. Content Comments: 7 -12 -2007 SFT group Meeting Danilo Piparo 22

A concrete example - 2 What’s going on? # Test Extensively the new Edm.

A concrete example - 2 What’s going on? # Test Extensively the new Edm. Size feature MU-_10_ALL. root @@@ Edm_Size. FEVTSIMEvent. Content, 1 @@@ MU-_Edm. Size. FEVTSIM MU-_10_ALL. root @@@ Edm_Size. RECOSIMEvent. Content, 1 @@@ MU-_Edm. Size. RECOSIM MU-_10_ALL. root @@@ Edm_Size. AODSIMEvent. Content, 1 @@@ MU-_Edm. Size. AODSIM MU-_10_ALL. root @@@ Edm_Size. FEVTSIMANAEvent. Content, 1 @@@ MU-_Edm. Size. FEVTSIMANA MU-_10_ALL. root @@@ Edm_Size. RECOSIMANAEvent. Content, 1 @@@ MU-_Edm. Size. RECOSIMANA Rootfile Edm. Size identifier. <Event. Content>, nevts name or command meta See Backup for XML version! 7 -12 -2007 SFT group Meeting Danilo Piparo 23