HAMBURG ZEUTHEN DESY Site Report Wolfgang Friebel DESY
HAMBURG • ZEUTHEN DESY Site Report Wolfgang Friebel DESY HEPi. X Meeting St. Louis Nov 6, 2007 DESY site report Nov 6, 2007
General remarks HAMBURG • ZEUTHEN § Rather complete overview over the IT landscape at DESY was given at previous meeting by Knut Woller • Many things mentioned there as being planned are reality now – 10 GBit interconnect between Hamburg and Zeuthen – Windows Homedirs on Net. App FAS 3050 – STK’s 8500 in Hamburg and Zeuthen in production – Backup at both locations using TSM (also for AFS) Nov 6, 2007 DESY site report 2
DESY – Physics news HAMBURG • ZEUTHEN § DESY is facing shift in user composition after HERA shutdown – Particle physics is continuing with HERA data – gearing up in CMS, ATLAS and LHCb – preparing for ILC § Photon physics is rebuilding parts of infrastructure to create PETRAIII – a new synchrotron radiation source § Continuing work on FLASH (Free Electron Laser in Hamburg) as preparation for XFEL, but also as a valuable new source in it's own right § XFEL construction will start early 2008 - 4. 3 kms of new tunnels and utility buildings Nov 6, 2007 DESY site report 3
XFEL (X-Ray Free Electron Laser) HAMBURG • ZEUTHEN § A light source producing ultrashort (100 femtoseconds) extremely brilliant and coherent light pulses • The tunable frequency (X-ray range) allows to study atomic details • The pulse length allows to study chemical reactions such as forming chemical compounds and study of the behaviour of biomolecules at the atomic level • The high brilliance guarantees „enough“ light even in the ultrashort pulses Nov 6, 2007 DESY site report 4
Photon Physics HAMBURG • ZEUTHEN § will pose new challenges – (n>1) PB of data p. a. – long term archival as required by funding agencies § Data produced in "short" experiments (hours) – bursty IO with several GB of data – (CCD with 8 k * 8 k pixel with 16 bits depth -> 128 MB/image, thousands of images per session – worst case are tomographs: 8 k slices, resulting in 1 TB raw data per probe run) – On 15 to 20 beamlines in parallel, 24 x 7 § So new demands on availability of IT infrastructure and services are formulated § on a side note: historically, photon physicists are used to taking their data home with them Nov 6, 2007 DESY site report 5
Grid HAMBURG • ZEUTHEN § New roles for DESY besides – Tier 0 for HERA – Tier 1 for AMANDA and Ice. Cube – International Lattice data grid (ILDG) § Tier 2 for ATLAS, CMS and LHCb § Tier 3 provided for Universities of Hamburg and Berlin § Tier 0 for ILC Nov 6, 2007 DESY site report 6
Physics at the Tera. Scale § Structured network between 17 universities and 3 institutes HAMBURG • ZEUTHEN § Experiments covered: Atlas, CMS, LHCb as well as ILC § Collaboration between experimentalists and theorists § Topics • fundamental questions of particle physics • distributed computing • novel detector development • accelerator science § Kickoff meeting in Dec 2007 § One key point: National Analysis Facility @ DESY Nov 6, 2007 DESY site report 7
National Analysis Facility HAMBURG • ZEUTHEN § Additional resources for the LHC experiments § Key points: • • New homogeneous distributed facility (Hamburg / Zeuthen) New design with distinct IP range, AFS cell, batch system. . . Fast and easy access to data A more personalized environment than Grid – Large home directories – Easier job submission and debugging § Build a flexible facility: Be able to react quickly to changes in needs when the first data arrives • “The NAF 2008/2009 will look different than a NAF 2012” Nov 6, 2007 DESY site report 8
NAF: Schematic basic layout NAF gsissh Interactive AFS/Kerberos Batch-Type (SGE) Parallel Cluster FS (Lustre) scp ? ? SRM ? ? qsub grid-submit Proof? ? grid-submit Nov 6, 2007 HAMBURG • ZEUTHEN ? Desy Tier 2 DESY site report SRM (d. Cache) Dedicated Space in DESY T 2 space Grid-ftp / srmcp 9
NAF planning HAMBURG • ZEUTHEN § Ambitious time table: first working services in Dec 07 § Different working groups: • OS (SL 4, SL 4+, no SL 3) • I/O file systems – AFS – Shared cluster file system for high-performance access • Batch (SGE grid tools + qsub) • Monitoring • Interactive (Login with proxy foreseen) • Software environment • Operation • Procurement Nov 6, 2007 DESY site report 10
Planned Hardware for the first setup HAMBURG • ZEUTHEN § Compute nodes: • 6 x 16 Dual. CPU-Quadcore Blades (HP-Proliant BL 460 c) • 2 GB RAM/core, 146 GB HD/Blade • Infiniband HCA § Storage nodes: • 7 x SUN Thumpers (17. 5 TB/box at raid 6) for d. Cache pools & Lustre • 4 x Sun X 4200 with 15 x 146 GB SAS Disks for AFS file services (planned to use virtual machines for AFS databases) § Other hardware: Racks, Infiniband Switch, Infrastructure servers, … Nov 6, 2007 DESY site report 11
Planned storage architecture HAMBURG • ZEUTHEN § Sun x 4500 (Thumpers) have performed well • Will use them for NAF d. Cache and Lustre (and maybe for AFS cell) • Problem: Best performance under Solaris / ZFS § ZFS and Solaris OK for d. Cache § Lustre not yet (but shortly? ) on Solaris • Linux in the meantime § IO connection via Infiniband § Reasons for Lustre • Users want a real /path/to/my/files access • Together with Infini. Band: Very high performance • Allows a very flexible reconfiguration of Computing Nodes (interactive, Batch & PROOF) without intervention on the storage Nov 6, 2007 DESY site report 12
Windows Vista HAMBURG • ZEUTHEN § Domain is ready for vista clients § No rollout plans yet: no real pressure • XP clients running very well, Vista would be very expensive, do vista benefits justify costs? § New application deployment strategy? – Current Netinstall 5. 72 is not vista ready, need update – Looking also at alternatives like SCCM (decision soon) – Good experiences with Netinstall (packaging, deploying, operating, maintenance) § Looking also at software virtualization as additional deployment tool (Thinstall, Softgrid) Nov 6, 2007 DESY site report 13
Windows system software HAMBURG • ZEUTHEN § Operating system deployment – WDS (Windows Deployment Services) will replace RIS – Supports in mixed mode both RIS and WDS config files § Windows Software Update Service 3. 0 – Launch at end of year (better reporting possibilities) § Microsoft Operations Manager 2005 – MOM used to monitor and collect server data – Upgrade to version 2007 is planned Nov 6, 2007 DESY site report 14
Windows software HAMBURG • ZEUTHEN § Office 2007 – Standard Office version is still Office. XP – No DESY wide rollout plans (high costs, no real requirements, few experiences of few users only) – Pros: better usability of Outlook 2007, extended capabilities of Excel and Power. Point – Cons: new user interface for XP users, licensing and training costs § MS Project 2007 Enterprise (server +clients) – Used for planning and control of XFEL construction – Overall bad experience: many bugs, bad response to bug reports, patches arrive late – Design flaws, e. g. caching algorithms – Waiting for SP 1, users feel like beta testers Nov 6, 2007 DESY site report 15
Collaborative tools HAMBURG • ZEUTHEN § Shutdown of HERA also means a reduction of collaboration members on site § Analysis however will continue past 2010 and close cooperation is still required § Investments in audio and video conferencing systems • Providing most lecture and meeting rooms with conferencing systems • Supporting classical videoconf systems as well as desktop solutions • IP Phones with video capability, personal computers with videoconf software Nov 6, 2007 DESY site report 16
Misc items HAMBURG • ZEUTHEN § d. Cache development (Chimera) – See separate talk by Peter van der Reest § Benchmarking new quad core machines – Tests with Simulation and Reconstruction part in ATLAS Athena frame work gave reasonable results, also good scalability with TForm (multi-threaded symbolic formula manipulation used for Feynman Integral calculation in QCD) – Less good results for parallel openmpi shared memory based Lattice QCD applications § Will start testing AFS-OSD extension soon – Installation, performance tests, development – Cooperation with developers and other interested sites Nov 6, 2007 DESY site report 17
HAMBURG • ZEUTHEN Thanks for your attention Questions? Remarks? Nov 6, 2007 DESY site report 18
- Slides: 18