Data Management Status of CVS repository Production databases

Data Management ØStatus of CVS repository ØProduction databases ØProduction tools M. Frank LHCb/CERN

CVS ØSeparate area for each sub-project ØInternal structure Øcmt Ø<package> Øsrc : requirements : public headers : source opt: src/<component> ØOpt: python, cmds, images, etc POOL M. Frank LHCb/CERN

Components Job. Configuration Job Submission App/ Pack. Mgr Data. Production Prod. Mgr Tool Physicist Book. keeping POOL Real dataflow Apparent dataflow M. Frank LHCb/CERN

Components ØJob configuration ØData production ØJob submission ØBookkeeping POOL Markus Andrei Sebastien/Francoise M. Frank LHCb/CERN

Job Configuration ØDatabase ok. ØSimple HTTP based Editor ØXML-RPC based interface ØBoost + python ØTool to import applications POOL M. Frank LHCb/CERN

Job Configuration: Editor POOL M. Frank LHCb/CERN

Data Production Pythia – v 2 1) Generation 2) Simulation Gauss - v 5 Brunel - v 12 3)Reconstruct Gauss - v 5 Brunel - v 12 Gen. Tag v 7 4) Mini Work Flow Execution Flow (Jobs) POOL M. Frank LHCb/CERN Gauss - v 5

Data Production - Details Workflow • Name • Step Seq. # • Nevt(Split) • Nevt(Merge) Step • Name • State • N(evt) to be processed Run • Name • Input file types • Output file types • Application Name Step <Rec> Step Job Input File • State • Center • Machine <Tags> Step Job Output File. Type File Type Workflow description POOL Job Configuration DB M. Frank LHCb/CERN

Data Production ØWork flows ØDatabase ok. ØSimple HTTP based Editor ØXML-RPC based interface ØBoost + python ØData Production ØDatabase ok. ØSimple HTTP based Editor ØXML-RPC based interface ØBoost + python POOL Instantiation of work flows to production runs Very primitive M. Frank LHCb/CERN

Work Flow Instantiation Pythia – v 2 1) Generation 2) Simulation Gauss - v 5 Brunel - v 12 3)Reconstruct Gauss - v 5 All processes in one job Gauss - v 5 Tools missing… Brunel - v 12 Gen. Tag v 7 4) Mini Work Flow Execution Flow (Jobs) POOL M. Frank LHCb/CERN

Work Flow Instantiation ØScript generation through servlet ØSelect work flow ØInstantiate work flow to production run ØAdd basic information Ø# of events ØCPU requirements etc. ØGenerate scripts ØPossibility to extend existing runs ØScripts are then picked up by Andrei’s client POOL M. Frank LHCb/CERN

Interacting Components ØMain strategy on component failure(s) ØThe system must stay operable ØIntermediately produced information items are cached locally until servers are up again ØProduction server (see next slide) ØProduction client (see Andrei’s talk) ØServers/servlets to publish bookkeeping information (see Sebastiens talk) POOL M. Frank LHCb/CERN

Component Collaboration Production Editor Work flow Editor Production data Production DB Scripts Edit Instantiate Workflow Production Server • Job request • Status updates Prod. Mgr Production Client Bookkeeping info XML Bookkeeping POOL M. Frank LHCb/CERN

Script Generation “Instantiated” Workflow POOL setenv PRODUCTION_IDENTIFIER 400000067 setenv JOB_NAME 400000369 # Target operating system: linux # Parameters of step Min. Bias. Dec. 2002 setenv MBIAS_QQ_USER_FILE None setenv MBIAS_EVTYPE 61 # Parameters of step B 2 pipi. Dec. 2002 setenv B 2 PIPI_QQ_USER_FILE B 2 pipi. dec setenv B 2 PIPI_EVTYPE 411300 # Parameters of work flow B 2 pipi. Dec. 2002 setenv SIMULATION_APP SICBMC setenv SIMULATION_VSN v 254 r 2 setenv RECONSTRUCTION_APP Brunel setenv RECONSTRUCTION_VSN v 13 r 1 setenv DBASE_PKG dbase setenv DBASE_VSN v 248 r 4 setenv XMLDDDB_PKG Xml. DDDB setenv XMLDDDB_VSN v 8 r 1 setenv QQ_USER_FILE ${MBIAS_QQ_USER_FILE}, ${B 2 PIPI_QQ_USER_FILE} setenv mcevtype ${MBIAS_EVTYPE}, ${B 2 PIPI_EVTYPE} setenv SPILLOVER on setenv mcmonihisto on M. Frank LHCb/CERN

Production Editor POOL M. Frank LHCb/CERN

Work Flow Editor POOL M. Frank LHCb/CERN

Production DBs - Summary ØPrimitive production tools exist ØWWW based ØBasic interface for production clients exists ØNeeds testing ØThe environment to start developing production tools is in place ØProblems with My. SQL ØMy. SQL does not support nested SELECT ØMy. SQL will not come in the very near future POOL M. Frank LHCb/CERN

Job List ØImport Brunel versions used in production run to job configuration database ØConstruct “reasonable” work flow ØTest generated production script(s) ØTest production server-client collaboration POOL M. Frank LHCb/CERN