ALICE Offline Tutorial PART 3 PROOF Alice Core

ALICE Offline Tutorial PART 3: PROOF Alice Core Offline 5 th June, 2008

PROOF Parallel ROOT Facility Interactive parallel analysis on a local cluster Parallel processing of (local) data (trivial parallelism) Fast Feedback Output handling with direct visualization Not a batch system PROOF itself is not related to Grid Can access Grid files The usage of PROOF is transparent The same code can be run locally and in a PROOF system (certain rules have to be followed) PROOF is part of ROOT 2

PROOF Schema Client – Local PC root ana. C Result stdout/result ana. C Remote PROOF Cluster root node 1 root Data Result Data node 2 root node 3 Proof master Proof slave root node 4 Result Data 3

Event based (trivial) Parallelism 4

Terminology Client Your machine running a ROOT session that is connected to a PROOF master Master PROOF machine coordinating work between slaves Slave/Worker PROOF machine that processes data Query A job submitted from the client to the PROOF system. A query consists of a selector and a chain Selector A class containing the analysis code In ALICE we use the Analysis Framework, therefore a Ali. Analysis. Task is sufficient Chain A list of files (trees) to process (more details later) 5

ALICE Analysis Framework Transparent access to all resources with the same code Usage: Local, Ali. En grid, CAF/PROOF Transparent access to different inputs ESD, AOD, Kinematics tree (MC truth) Allow for „scheduled“ analysis Common and well tested environment to run several tasks Defines a common terminology 6

How to use PROOF The analysis framework is used Files to be analyzed are put into a chain TChain Analysis written as a task Ali. Analysis. Task The same analysis like in the local case can be used in a parallel environment If additional libraries are needed, these have to be distributed as a "package" Input Files (TChain) Analysis (Ali. Analysis. Task) Output 7

Ali. Analysis. Task Classes derived from Ali. Analysis. Task can run locally, in PROOF and in Ali. En "Constructor" Create. Output. Objects() Connect. Input. Data() Exec() once on your client once on each slave for each tree for each event Terminate() 8

Class TTree A tree is a container for data storage It consists of several branches Tree Branch x y z Branch Set of helper functions to visualize content (e. g. Draw, Scan) Branches Compressed point x x x x Branch These can be in one or several files Branches are stored contiguously (split mode) When reading a tree, certain branches can be switched off speed up of analysis when not all data is needed File x x y y y y y z z z z z 9

TChain A chain is a list of trees (in several files) Normal TTree functions can be used Draw(. . . ), Scan(. . . ) these iterate over all elements of the chain Selectors can be used with chains Process(const char* selector. File. Name) After using Set. Proof() these calls are run in PROOF Chain Tree 1 (File 1) Tree 2 (File 2) Tree 3 (File 3) Tree 4 (File 3) Tree 5 (File 4) 10

Merging The analysis runs on several slaves, therefore partial results have to be merged Objects are identified by name Standard merging implementation for histograms available Other classes need to implement Merge(TCollection*) When no merging function is available all the individual objects are returned Result from Slave 1 Slave 2 Merge() Final result 11

Ali. Analysis. Manager – PROOF mode PROOF AM>Start. Analysis(“proof”) TSelector task. N On O 2 On AM AM AM task 1 task 2 task 3 task 1 task 2 O task. N task 3 task 1 task 2 task. Ntask 3 task 2 task. N task 3 O task. N task 3 task. N Outputs task 3 O 2 Worke r Worke AM r Worker AM r Inputs task 2 O 1 Outputs task 1 Input list Input chain AM Slave. Begin Process() () Slave. Terminate( e() ) Output list Ali. Analysis. Selector Inputs Analysis Manager O 1 Master Outputs My. Analysis. C PROOF Inputs Outputs CLIENT 12

Workflow Summary Chain Analysis (Ali. Analysis. Task) Input Tree 1 (File 1) proof Tree 2 (File 2) Tree 3 (File 3) proof Tree 4 (File 3) Tree 5 (File 4) proof 13

Workflow Summary Analysis (Ali. Analysis. Task) proof Output Merged Output 14

Packages PAR files: PROOF ARchive. Like Java jar Gzipped tar file PROOF-INF directory • • BUILD. sh, building the package, executed per slave SETUP. C, set environment, load libraries, executed per slave API to manage and activate packages Upload. Package("package") Enable. Package("package") 15

CERN Analysis Facility The CERN Analysis Facility (CAF) will run PROOF for ALICE Prompt analysis of pp data Pilot analysis of Pb. Pb data Calibration & Alignment Available to the whole collaboration but the number of users will be limited for efficiency reasons Design goals 500 CPUs 100 TB of selected data locally available 16

Evaluation of PROOF Test setup since May 2006 40 machines, 2 CPUs each, 200 GB disk Tests performed Usability tests Simple speedup plot Evaluation of different query types Evaluation of the system when running a combination of query types Goal: Realistic simulation of users using the system 17

Hands-On Getting ready. . . Run a task that accesses ESD Locally PROOF Modify it. . . Run a task that accesses MC PROOF Reading log files, resetting session, etc. 19

Warm up Preconditions • From your lxplus account, do: wget http: //aliceinfo. cern. ch/Offline/Activities/Analysis/C AF/proof-tutorial. tgz gtar xvzf proof-tutorial. tgz Set up environment source /afs/cern. ch/alice/caf-lxplus. sh v 4 -14 Release • NB: This works only for the tutorial machines, i. e. SLC 4 i 686 How to enable it on LXPLUS is explained later in the tutorial Check ROOT Start it. Does it show ROOT version 5. 21/01? 20

Files to be used Create. ESDChain. C: Creates a chain from a list of file names ESD 82 XX_30 K. txt: List of PDC 07 files distributed on the CAF AF-v 4 -12. par: Par archive for PDC 07 data and analysis framework Ali. Analysis. Task. Pt. {cxx, h}: Task that creates an uncorrected p. T spectrum from ESD tracks Ali. Analysis. Task. Pt. MC. {cxx, h}: Task that creates an p. T spectrum from the MC particles 21

Run a task locally Start ROOT Try the following lines and once they work add them to a macro run. C (enclose in {}) Load needed libraries g. System->Load("lib. Tree"); g. System->Load(“lib. Geom”); g. System->Load("lib. VMC"); g. System->Load(“lib. Physics”); g. System->Load("lib. STEERBase"); g. System->Load("lib. AOD"); g. System->Load("lib. ESD"); g. System->Load("lib. ANALYSIS"); Add the Ali. Root include path (only needed for local case) g. ROOT->Process. Line(". include $ALICE_ROOT/include"); 22

Run a task locally (2) Create the analysis manager mgr = new Ali. Analysis. Manager("mgr"); Create the analysis task and add it to the manager g. ROOT->Load. Macro("Ali. Analysis. Task. Pt. cxx++g"); • "+" means compile; "g" means debug task = new Ali. Analysis. Task. Pt; mgr->Add. Task(task); Add the ESD handler (to access the ESD) Ali. ESDInput. Handler* esd. H = new Ali. ESDInput. Handler; mgr->Set. Input. Event. Handler(esd. H); Add the lines to the macro run. C 23

Run a task locally (3) Create a chain g. ROOT->Load. Macro("Create. ESDChain. C"); chain = Create. ESDChain("ESD 82 XX_30 K. txt", 10); Attach the input (the chain) c. Input = mgr->Create. Container("c. Input", TChain: : Class(), Ali. Analysis. Manager: : k. Input. Container); mgr->Connect. Input(task, 0, c. Input); Create a place for the output (a histogram: TH 1) c. Output = mgr->Create. Container("c. Output", TH 1: : Class(), Ali. Analysis. Manager: : k. Output. Container, "Pt. root"); mgr->Connect. Output(task, 0, c. Output); Enable debug (optional) mgr->Set. Debug. Level(2); Add the lines to the macro run. C 24

Run a task locally (4) Initialize the manager mgr->Init. Analysis(); Print the status (optional) mgr->Print. Status(); Run the analysis mgr->Start. Analysis("local" , chain); Add the lines to the macro run. C After running look at the output and check the content of the file Pt. root 25

run. C 26

Package Management Connecting to the PROOF cluster TProof: : Open("lxb 6046"); Managing packages Upload (= copy to the cluster) • g. Proof->Upload. Package(“AF-v 4 -14"); Enable (= compile) • g. Proof->Enable. Package("AF-v 4 -14"); Clean (= remove) • • g. Proof->Clear. Package("AF-v 4 -14"); Known issue on AFS: Removal may fail. Try again after few seconds… Clean all (in case some libraries are messed up) • g. Proof->Clear. Packages(); 27

Running a task in PROOF Copy run. C to run. Proof. C Add connecting to the cluster 20 files TProof: : Open("lxb 6046") Replace the loading of the libraries with uploading the packages g. Proof->Upload. Package("AF-v 4 -14") g. Proof->Enable. Package("AF-v 4 -14") Replace the loading of the task with g. Proof->Load("Ali. Analysis. Task. Pt. cxx++g") Replace in Start. Analysis "local" with "proof" 200 files Run it! Increase the number of files to 200 28

run. Proof. C 29

Progress dialog Query statistics Abort query and view results up to now Abort query and discard results Show log files Show processing rate 30

Looking at the task Constructor Called once when the task is created Input/Output is connected Connect. Input. Data (usually does not need to be changed) Called once per tree on each slave Connect f. ESD pointer Create. Output. Objects Called once per slave Create histograms Exec Called once per event Track loop, tracks are counted, histogram filled, output "posted" Terminate Called once on the client (your laptop/PC) Histogram read back from the output stream, visualized, saved to disk 31

Changing the task Add a |h| < 0. 5 cut Float_t eta = track->Eta(); if (TMath: : Abs(eta) > 0. 5) continue; 32

Changing the task (2) Add a second plot: h distribution Header file (. h file) • Add new member: TH 1 F* f. Eta; // eta distribution Constructor • • Initialize member: f. Eta(0) Add second output slot: Define. Output(1, TH 1 F: : Class()) Create. Output. Objects • Create histogram f. Eta = new TH 1 F("f. Eta", "#eta distribution", 20, -2, 2); Exec • • • Get h like in previous example Fill histogram: f. Eta->Fill(eta); Post output: Post. Data(1, f. Eta) 33

Changing the task (3) Terminate Read histogram from the output slot f. Eta = dynamic_cast<TH 1 F*> (Get. Output. Data(1)); Introduce an if statement if the object was retrieved if (!f. Eta) { Printf("ERROR: f. Eta was not found"); return; } Draw the histogram new TCanvas; f. Eta->Draw. Copy(); Copy run. Proof. C to run. Proof 2. C and change: Add second output slot c. Output 2 = mgr->Create. Container("c. Output 2", TH 1: : Class(), Ali. Analysis. Manager: : k. Output. Container, "Pt. root"); mgr->Connect. Output(task, 1, c. Output 2); 34

Read Monte Carlo tracks Use task Ali. Analysis. Task. Pt. MC. {h, cxx} Copy run. Proof. C to run. Proof. MC. C Change Ali. Analysis. Task. Pt to Ali. Analysis. Task. Pt. MC Add access to the MC event handler = new Ali. MCEvent. Handler; mgr->Set. MCtruth. Event. Handler (handler); Change output filename to Pt. MC. root Run it! 35

run. Proof. MC. C 36

Looking at the MC task Very similar to ESD track case Instead of looping over content of f. ESD, MC event is retrieved by Ali. MCEvent. Handler* event. Handler = dynamic_cast<Ali. MCEvent. Handler*> (Ali. Analysis. Manager: : Get. Analysis. Manager() ->Get. MCtruth. Event. Handler()); if (!event. Handler) { Printf("ERROR: Could not retrieve MC event handler"); return; } Ali. MCEvent* mc. Event = event. Handler->MCEvent(); if (!mc. Event) { Printf("ERROR: Could not retrieve MC event"); return; } 37

Reading log files When your task crashes You can access the output via the PROOF progress window In rare casrs you have to restart the ROOT session Reading output from last query Open ROOT Get a PROOF manager object mgr = TProof: : Mgr("lxb 6046") Get the log files from the last session logs = mgr->Get. Session. Logs(0) Display them logs->Display() Search for a special word (e. g. segmentation violation) logs->Grep("segmentation violation") Save them to a file logs->Save("*", "logs. txt") 38

Some Goodies. . . Resetting environment TProof: : Reset("lxb 6046") Do not put this in your macro, if really needed, call it manually in a root session Compile with debug Load("<task>+g") Create a package from Ali. ROOT make ESD. par 39

CAF use from LXPLUS "Any" ROOT version is ok Recommended version for LXPLUS runs SLC 4 on x 86_64 cd /afs/cern. ch/alice/caf source caf-lxplus. sh v 4 -12 -Release (bash shell) • v 4 -12 -Release is the Ali. Root version that is enabled at the same time More information on http: //aliceinfo. cern. ch/Offline/Activities/Analysis/CAF Please join the mailing list alice-projectanalysis-task-force@cern. ch by going to http: //listboxservices. web. cern. ch/listboxservi ces/ 40

Backup 41

PROOF Installation Install ROOT with PROOF enabled (default) More information: http: //root. cern. ch Configuration (see next slides) xrootd config file: xrd. cf PROOF config file: proof. conf Start xrootd service Requires unprivileged user account 42

xrd. cf ## Load the Xrd. Proofd protocol: xrd. protocol xproofd: 1093 /opt/root/lib. Xrd. Proofd. so ## Set ROOTSYS xpd. rootsys /opt/root ## Working directory for sessions xpd. workdir /pool/proofbox 43

xrd. cf (2) ## xpd. resource static [<cfg_file>] [ucfg: <user_cfg_opt>] [wmx: <max_workers>] [selopt: <selection_mode>] xpd. resource static /etc/proof. conf wmx: -1 selopt: roundrobin ## Server role (master, worker) [default: any] xpd. role worker if lxb*. cern. ch xpd. role master if lxb 6046. cern. ch ## Master(s) allowed to connect. By default all connections are allowed. xpd. allow lxb 6046. cern. ch 44

proof. conf ## machine running the master lxb 6046. cern. ch ## machine(s) running Workers, dual CPU machines have to be listed twice worker lxb 6047. cern. ch worker lxb 6048. cern. ch. . . 45

Starting xrootd Service xrootd -b -l xrootd. log -R proofaccount -c xrd. cf -d Options: -b : background (skip for debugging) -l : log file -R <useraccount> : user account that runs xrootd service -c <configfile> : configuration file -d : debug flag Do not forget full paths to the files 46

Learning about Branches The ESD tree consists of several branches Switching off not needed branches increases speed of analysis significantly Looking at the available branches chain = new TChain("esd. Tree") chain->Add("root: //lxb 6046. cern. ch//pool/proofpool/ pdc 06/100/002/root_archive. zip#Ali. ESDs. root") chain->Print() Disable all branches (in Init) tree->Set. Branch. Status("*", 0) Enable a needed branch (in Init) tree->Set. Branch. Status("f. Tracks. f. Cp", 1) Try this! What is the increase in processing speed? 47