Ali En Tutorial Panos Christakoglou University of Athens

Ali. En Tutorial Panos Christakoglou University of Athens - CERN 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 1

OUTLINE (1) ● Installation of the Ali. En software. ● Authentication. ● General description of the shell: – Basic commands. – Basic functionalities. ● News on production. ● Working with the file catalog: – Copying files from/to the catalog. – File catalog structure. – Querying the file catalog. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 2

OUTLINE (2) ● Flow of the overall analysis procedure. ● Local analysis. – Creation of tag files. – Local analysis using the Event Tag System. ● Interactive analysis with Ali. En stored files. ● Batch analysis: – Flow of the procedure. – Description of the files needed. – Description of the jdl fields. – Practical examples. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 3

Installation of Ali. En software 10/10/06 Getting the installer Panos. Christakoglou@cern. ch - Ali. En Tutorial 4

Installation of Ali. En software 10/10/06 Selecting the version Panos. Christakoglou@cern. ch - Ali. En Tutorial 5

Installation of Ali. En software 10/10/06 Reuse of software Panos. Christakoglou@cern. ch - Ali. En Tutorial 6

Installation of Ali. En software 10/10/06 Workspace directory Panos. Christakoglou@cern. ch - Ali. En Tutorial 7

Installation of Ali. En software 10/10/06 Platform Panos. Christakoglou@cern. ch - Ali. En Tutorial 8

Installation of Ali. En software 10/10/06 Supported platforms Panos. Christakoglou@cern. ch - Ali. En Tutorial 9

Installation of Ali. En software 10/10/06 Installation directory Panos. Christakoglou@cern. ch - Ali. En Tutorial 10

Installation of Ali. En software 10/10/06 Installation directory Panos. Christakoglou@cern. ch - Ali. En Tutorial 11

Installation of Ali. En software 10/10/06 Overwrite files Panos. Christakoglou@cern. ch - Ali. En Tutorial 12

Installation of Ali. En software 10/10/06 Selecting packages Panos. Christakoglou@cern. ch - Ali. En Tutorial 13

Installation of Ali. En software 10/10/06 Progress bar Panos. Christakoglou@cern. ch - Ali. En Tutorial 14

Installation of Ali. En software Try it out ● Download the alien installer from http: //alien. cern. ch. ● Make the file executable. ● Run the installer. ● Select v 2. 11. ● Platform should be i 686. ● Select installation directory. ● Select the following packages: – Client – g. Shell – ROOT 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 15

Authentication Getting a grid proxy Password that you selected when making the request for the certificate 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 16

Authentication Getting an alien proxy alien-token-init <username> 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 17

Authentication ● Get a valid grid proxy. ● Get a valid alien token. ● Check the information of your proxy by typing: – ● Try it out grid-proxy-info Check the information of your token by typing: – 10/10/06 alien-token-info Panos. Christakoglou@cern. ch - Ali. En Tutorial 18

General description of the shell 10/10/06 Accessing the shell Panos. Christakoglou@cern. ch - Ali. En Tutorial 19

General description of the shell 10/10/06 Basic commands (1) Panos. Christakoglou@cern. ch - Ali. En Tutorial 20

General description of the shell 10/10/06 Basic commands (2) Panos. Christakoglou@cern. ch - Ali. En Tutorial 21

General description of the shell 10/10/06 Whereis command Panos. Christakoglou@cern. ch - Ali. En Tutorial 22

General description of the shell 10/10/06 Viewing files (1) Panos. Christakoglou@cern. ch - Ali. En Tutorial 23

General description of the shell 10/10/06 Viewing files (2) Panos. Christakoglou@cern. ch - Ali. En Tutorial 24

General description of the shell ● Check your user name by typing whoami. ● List the contents of your home directory. ● Check the working directory. ● Create the following directory structure: ● – $HOME/bin – $HOME/Tutorial/XML/jdl – $HOME/Tutorial/XML/par – $HOME/Tutorial/XML/output – $HOME/Tutorial/XML/selectors – $HOME/Tutorial/XML/macros Try it out Get the information of the file: /alice/cern. ch/user/p/pchrist/Tutorial/XML/selectors/esd. Pt. C 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 25

News on production ● p+p min bias @ 14 Te. V --- NEvents = 3579700: – ● /alice/cern. ch/user/a/aliprod/prod 2006_2/output_pp/ ● All RUNS have been tested (100 -111 & 200 -246). ● Merged tag files have been produced at the RUN level for all Run. Ids. p+p min bias @ 900 Ge. V --- NEvents = 190400: – ● File location /alice/sim/2006/pp_900 Ge. V/ ● All RUNS have been tested (600 -601). ● Merged tag files have been produced at the RUN level for all Run. Ids. Muon events --- NEvents = 61150: – /alice/sim/2006/muon/ 10/10/06 ● All RUNS have been tested (500 -502). ● Merged tag files have been produced at the RUN level for all Run. Ids. Panos. Christakoglou@cern. ch - Ali. En Tutorial 26

Working with the file catalog 10/10/06 Copying from/to the catalog Panos. Christakoglou@cern. ch - Ali. En Tutorial 27

Working with the file catalog File catalog structure • The path name will be: • for ‘real’ data: /data/<Year>/<Accelerator. Period>/<Run. Number>/ • for simulated data: /sim/<Year>/<Production. Type>/<Run. Number>/ • Subdirectories will be called: MARKUS OLDENBURG • Raw/ • cond/ • reco/<Pass. X>/ESD/ AN INTERNAL NOTE IS ON THE WAY • reco/<Pass. X>/AOD/ • … • File names will look like this: <xxxx>. Ali. ESD. root • For further information see http: //indico. cern. ch/conference. Display. py? conf. Id=3280 http: //cern. ch/Oldenburg/Meta. Data. doc 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 28

Working with the file catalog Querying the file catalog (1) $ find -x pp /alice/cern. ch/user/p/pchrista/production/pp/PDC 06/* Ali. ESDs. root > pp. xml Redirect the output to the xml collection. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 29

Working with the file catalog Querying the file catalog (2) $ find -x pp /alice/data/2008/LHC 08 a/*/reco/Pass 3/* Ali. ESDs. root Run: collision_system=”pp” and Run: stop<"2008 -03 -20 10: 20: 33" and Run: start>"2008 -03 -19" > pp. xml 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 30

Working with the file catalog ● ● Try it out (1) Create the following directory structure locally: – $HOME/Ali. En/PDC 06/001 – $HOME/Ali. En/PDC 06/002 – $HOME/Ali. En/Tags – $HOME/Ali. En/Local – $HOME/Ali. En/Interactive – $HOME/Ali. En/Batch Copy the following files to your local $HOME/Ali. En/Local: – /alice/cern. ch/user/p/pchrist/Tutorial/LOCAL/ESD. par – /alice/cern. ch/user/p/pchrist/Tutorial/LOCAL/esd. Pt. h – /alice/cern. ch/user/p/pchrist/Tutorial/LOCAL/esd. Pt. C – /alice/cern. ch/user/p/pchrist/Tutorial/LOCAL/demo. Local. C 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 31

Working with the file catalog ● ● Try it out (2) Copy the following files to your local $HOME/Ali. En/Tags: – /alice/cern. ch/user/p/pchrist/Tutorial/TAGS/ESD. par – /alice/cern. ch/user/p/pchrist/Tutorial/TAGS/Create. Tags. C Copy the following files to $HOME/Ali. En/PDC 06/001 and $HOME/Ali. En/PDC 06/002: – /alice/cern. ch/user/a/aliprod/prod 2006_2/output_pp/100/002/Ali. ESDs. root – /alice/cern. ch/user/a/aliprod/prod 2006_2/output_pp/100/004/Ali. ESDs. root Query the f. c. and get all the tag files (*. tag. root) under: /alice/cern. ch/user/a/aliprod/prod 2006_2/output_pp/100/* – Get the output on your terminal. – Count the files using the shell. – Redirect the results to the tag 100. xml collection. Repeat the previous exercise limiting the number of output files to 10 (find -l 10. . . ) and copy the xml to your local $HOME/Ali. En/Interactive. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 32

Flow of the analysis procedure Query the file catalog Tag xml collection User's macro Selection criteria Extract an esd xml collection Query the Event Tag System Local Ali. En Process chain with selector Create an esd. Tree chain PROOF 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 33

Local analysis 10/10/06 Event Tag System Panos. Christakoglou@cern. ch - Ali. En Tutorial 34

Local analysis Create tag files Setup par archive Load the needed libraries Ali. Tag. Creator *t = new Ali. Tag. Creator(); t->Set. Storage(0); t->Read. Local. Collection(“/home/pchrist/PDC 06/pp 14 Te. V/”); t->Merge. Tags(); 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 35

Local analysis with tags Setup par archive Load the needed libraries Ali. Event. Tag. Cuts *Ev. Cuts = new Ali. Event. Tag. Cuts(); Ev. Cuts->Set. Multiplicity. Range(0, 1500); Ali. Tag. Analysis *Tag. Ana = new Ali. Tag. Analysis(); Tag. Ana->Chain. Local. Tags(“. ”); analysischain = Tag. Ana->Query. Tags(Ev. Cuts); const char *selectorfile = "esd. Pt. C"; analysischain->Process(selectorfile); 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 36

Local analysis ● Try it out Open your local $HOME/Ali. En/Tags/Create. Tags. C file and modify it accordingly: – Change the line where you define where you have the locally stored ESDs. ● Run it to create the tag files. ● Delete the single tag files and stay with just the merged one. ● ● ● Go to your local $HOME/Ali. En/Local directory and open the demo. Local. C file. Change the line where you define the location of the tag files and run the macro with root. Impose some selection criteria and rerun the example. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 37

Interactive analysis with tags Setup par archive Load the needed libraries TGrid: : Connect("alien: //pcapiserv 01. cern. ch: 10000", "pchrist") Ali. Event. Tag. Cuts *Ev. Cuts = new Ali. Event. Tag. Cuts(); Ev. Cuts->Set. Multiplicity. Range(0, 1500); TAlien. Collection* coll = Talien. Collection: : Open("tag 100. xml"); TGrid. Result* Tag. Result = coll->Get. Grid. Result(""); Ali. Tag. Analysis *Tag. Ana = new Ali. Tag. Analysis(); Tag. Ana->Chain. Grid. Tags(Tag. Result); analysischain = Tag. Ana->Query. Tags(Ev. Cuts); const char *selectorfile = "esd. Pt. C"; analysischain->Process(selectorfile); 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 38

Interactive analysis ● ● Try it out Copy the following files to your local $HOME/Ali. En/Interactive: – /alice/cern. ch/user/p/pchrist/Tutorial/INTERACTIVE/ESD. par – /alice/cern. ch/user/p/pchrist/Tutorial/INTERACTIVE/esd. Pt. h – /alice/cern. ch/user/p/pchrist/Tutorial/INTERACTIVE/esd. Pt. C – /alice/cern. ch/user/p/pchrist/Tutorial/INTERACTIVE/demo. Interactive. C Go to your local $HOME/Ali. En/Interactive directory and open the demo. Interactive. C file. ● Change the line where you define the tag collection and put the name of the file you created by querying the f. c (it should be tag 100. xml). ● Run the macro. ● Impose some selection criteria and rerun the example. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 39

Analysis flow Batch analysis F. C. query JDL: : Input. Data. Collection <collection> <event>. . . guid=”asdf 1”. . . lfn=”/alice/cern. ch/. . . /file 1. root” <event>. . . guid=”asdf 2”. . . lfn=”/alice/cern. ch/. . . /file 2. root” <event>. . . guid=”asdf 3”. . . lfn=”/alice/cern. ch/. . . /file 3. root” </collection> JDL: : Input. Data <collection 1> <event> lfn=”/alice/. . . /file 1. root” </collection 1> JOB ANGENT ROOT JDL: : Input. Data <collection 1> <event> lfn=”/alice/. . . /file 2. root” </collection 1> Job Optimiser Splitting JDL: : Input. Data <collection 1> <event> lfn=”/alice/. . . /file 3. root” </collection 1> TAlien. Collection ROOT execution SITE A SITE B SITE C XROOTD MSS 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 40

Analysis flow with the tags Batch analysis Optimiser Splitting Global Chain C GUID-LFN Event. List Event list gets lost during splitting Input. Data in new JDL Job agents C 1 C 2 C 3 LFN-GUID Event. List 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial Cn AND C 41

Batch analysis ● Executable ● Par file ● Macro ● Selectors ● xml collection ● jdl 10/10/06 Files needed Panos. Christakoglou@cern. ch - Ali. En Tutorial 42

Batch analysis Executable #!/bin/bash echo echo ============== $PATH $ROOTSYS $LD_LIBRARY_PATH ============= root -b -x Analysis. C; IT SHOULD BE STORED UNDER $HOME/bin IN THE FILE CATALOG!!! 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 43

Batch analysis Macro • Setup the par file – compile and load the lib. ESD. so (or any necessary library that is needed for the analysis). • Get the xml collection. • Convert the collection to a list of files. • Process the chain with the selector. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 44

Batch analysis ● JDL fields (1) Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V 0/bin or $HOME/bin. ● Arguments: They will be passed to the executable. ● Packages: Type packages in the shell to see what kind of packages are installed. ● Input. File: The files that will be transported to the node where the job will run. ● ● ● Input. Data: It will require that the job will be executed in a site close to the files specified here. Input. Data. List: The filename in which the Job Agent will write the Input. Data list. Input. Data. List. Format: The format of the Input. Data list. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 45

Batch analysis ● JDL fields (2) Output. File: The files that will be registered in the catalog once the job finishes. ● Output. Archive: What files will be archived in a zip file. ● Validationcommand: Specifies the script to be used as a validation script. ● Email: Receive a mail when the job finishes, ● TTL: The maximum run time of your job. ● Split: Split the jobs in several sub jobs. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 46

Batch analysis ● ● Try it out (1) Copy the following files to $HOME/Ali. En/Batch: – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/ESD. par – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/esd. Pt. h – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/esd. Pt. C – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/demo. Batch. C – /alice/cern. ch/user/p/pchrist/bin/batch. sh – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/pp 10. xml – /alice/cern. ch/user/p/pchrist/Tutorial/BATCH/analysis. jdl Modify the Email, Input. File, Input. Data. Collection and Output. Dir fields of your analysis. jdl. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 47

Batch analysis Try it out (2) ● Copy the local Analysis. sh to your Ali. En $HOME/bin. ● Copy the local analysis. jdl to your Ali. En $HOME/Tutorial/XML/jdl. ● Copy the local pp 10. xml to your Ali. En $HOME/Tutorial/XML/xml. ● Copy the local ESD. par to your Ali. En $HOME/Tutorial/XML/par. ● Copy the local esd. Pt. * to your Ali. En $HOME/Tutorial/XML/selectors. ● Copy the local demo. Batch. C to your Ali. En $HOME/Tutorial/XML/macros. ● Go to your Ali. En $HOME/Tutorial/XML/jdl and submit the job by typing: “submit analysis. jdl”. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 48

Batch analysis Try it out (3) ● Check your job priority by typing: “queue priority jobs $username”. ● Display the jdl of your job by typing: “ps -jdl $jobid”. ● Trace the status of your job by typing: “ps -trace $jobid”. ● When the job 's status turns to RUNNING you can get the stdout and stderr of the job by typing: – “spy $jobid stdout”. – “spy $jobid stderr”. 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 49

References (1) ● Registration – Certificates: – ● Ali. En: – ● http: //alien. cern. ch Gshell: – ● http: //alien. cern. ch/twiki/bin/view/Alice/User. Registration http: //alien. cern. ch/twiki/bin/view/Ali. En/GAPI User's guide: – http: //project-arda-dev. web. cern. ch/project-ardadev/alice/apiservice/AA-User. Guide-0. 0 m. pdf 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 50

References (2) ● Previous tutorials: – http: //project-arda-dev. web. cern. ch/project-ardadev/alice/apiservice/Tutorial-Exercises. pdf – http: //project-arda-dev. web. cern. ch/project-ardadev/alice/apiservice/Tutorial-Exercises-Solutions. pdf – http: //project-arda-dev. web. cern. ch/project-ardadev/alice/apiservice/Tutorial 2 -Exercises-Solutions. pdf ● File catalog structure – Queries: – http: //pcaliweb 02. cern. ch/Offline/Analysis/Run. Event. Tag. System/Run. Tags. html#Run/File%20 metadata 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 51

References (3) ● Event Tag System: – http: //pcaliweb 02. cern. ch/Offline/Analysis/Run. Event. Tag. System/Event. Ta gs. html#Event%20 tag%20 system – ● ALICE internal note. Creation of tag files: – http: //pcaliweb 02. cern. ch/Offline/Analysis/Run. Event. Tag. System/Event. Ta gs. Creation. html#Create%20 tags%20 howto ● Analysis using the Event Tag System: – http: //pcaliweb 02. cern. ch/Offline/Analysis/Run. Event. Tag. System/Event. Ta gs. Analysis. html#Analysis%20 with%20 tags 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 52

Reference (4) ● File level metadata: – 10/10/06 http: //cern. ch/Oldenburg/Meta. Data. doc Panos. Christakoglou@cern. ch - Ali. En Tutorial 53

BACKUP 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 54

Installation ● Problems with gcc versions Ali. En software comes with a precompiled library with gcc 3. 2. 3 which is also shipped with Ali. En. ● If you are using a different version of gcc (check it with gcc –v) then do the following: – cd $ALIEN/api/src – . /recompile. gapi – cd $ALIEN/api/lib – Copy all the libgapi. UIXX files to the libgapi. UI. so. XX 10/10/06 Panos. Christakoglou@cern. ch - Ali. En Tutorial 55