Status and plans for bookkeeping system and production

Status and plans for bookkeeping system and production tools Eric van Herwijnen Thursday, 19 september 2002

Contents u u u u Status remote centres & datachallenge plans Data. Grid status and plans Data management and production tools Job submission Bookkeeping Configuration Data production and monitoring Roadmap and conclusions

Status remote centres Center No. of CPU’s (1 GHz) Production tools CERN ~ 400 new Lyon 60 + new Liverpool ~ 120 new Imperial College ~ 100 new Data. Grid ~ 20 new RAL ~ 300 old Bologna ~ 200 old Nikhef ~ 20 old Bristol ~ 20 old Edinburgh ~ 120 old Cambridge ~ 15 old Oxford ~ 10 old Moscow ~ 40 old Rio ~ 20 old Total ~ 1000 (outside CERN)

Physics Data Challenge plans u u u MC production = Physics Data Challenge Available capacity seems to match requirements Planning: Preproduction: mid Dec 2002 – mid Jan 2003 Production: Feb – May 2003

Data. Grid status and plans u u u Installation 1. 2. 2 operational Long job problem fixed Long file transfer problem (~ 1 Gb) New production tools being installed Test: n Run 500 event MC generation n Store on SE n Recover logs and histograms to CERN n Run reconstruction. Output to SE. Recover log files and histos. n Write recon output to mass store (Castor) n Read Castor data with an analysis job outside Grid

Data Management

Job submission u u u New tools designed and written by A. Tsaregorodtsev Site dependencies concentrated in 3 scripts Simple installation procedure: n u u u http: //lhcb-wdqa. web. cern. ch/lhcb-wdqa/distribution/ AFS independent Standard directory structure No (or very little) java More flexible updating of bookkeeping database Remote centers should migrate to new system

Job submission, future Job. opts Modify Data Production DB Bookkeeping DB Production done Create job(s) script Prod. Mgr • Build new configuration • Selection of Defaults Configuration DB Information Flow

Job submission, future u Workflow to be implemented (template scripts for each step, parameters added via web page) u Prototype exists (M. Frank)

Bookkeeping

Bookkeeping u u u Most components ready (S. Ponce, F. Loverre) Structure of database independent of tools, easy to add new datatypes, identify productions, replicas of datasets Database (Oracle) filled by a Java server via XML files API implemented in Java and Python Migration to new database transparent for user Final tests under way before production

Configuration u u Prototype of configuration database exists Need to integrate with job submission tools (5 line python script? ) Prototype of GUI to view the database developed by G. Klamke (summer student) Need to create tool to add configurations to database, and integrate with Ganga

Data Production and Monitoring

Data production and monitoring u Current PVSS system needs to be brought beyond the prototype stage n n n u u u Cosmetic changes, cleanup of dead entries, speed Adapt to new job submission tools Alarms should really be alarms, add corrective action Migrate to new version of PVSS Web interface Production database + interface need to be created Data quality check tools need to be reviewed and created Ongoing activity

Roadmap and conclusions u u u Installation of new job submission tools: octnov 2002 New bookkeeping db: dec 2002 Integration of job configuration db with job submission tools: feb 2003 Integration of job configuration db with Ganga: summer 2003 Creation of data production db, integration with monitoring system: summer-fall 2003