The Virtual Data Toolkit OSG Software Alain Roy

The Virtual Data Toolkit & OSG Software Alain Roy OSG Software Coordinator Condor Team University of Wisconsin-Madison 1

Review: What is the VDT? n A collection of software q q q n An easy installation q q n Grid software (Condor, Globus and lots more) Virtual Data System (Origin of the name “VDT”) Utilities Goal: Push a button, everything just works Two methods: n Pacman: installs and configures it all n RPM: installs some of the software, no configuration A support infrastructure The State of the VDT 2

Who makes the VDT? n The VDT is a product of Open Science Grid (OSG) q n n VDT is used on all OSG grid sites OSG is new, but VDT has been around since 2002 Originally, VDT was a product of the Gri. Phy. N/i. VDGL q VDT was used on all Grid 2003 sites The State of the VDT 3

Who makes the VDT today? 1 Mastermind + 3 FTEs Expanding with new funding The State of the VDT Miron Livny Alain Roy Tim Cartwright Andy Pavlo 4

Why have the VDT? n n Everyone could download the software from the providers… But the VDT: q q q Figures out dependencies between software Works with providers for bug fixes Provides automatic configuration Packages it Tests everything on thirteen platforms (and growing) The State of the VDT 5

Example: VOMS can authorize people in a VO n VOMS has a web interface n We: n q q q q We pre-built binaries Install Tomcat Install Apache for each of these n Built with Globus SSL n Patched so GSI pass-through to Apache works Install VOMS Admin Install Perl modules needed by VOMS Admin Install My. SQL and set up database (with command-line tool) Configure all software Configure rotation of log files The State of the VDT 6

Example: Security Update n Last week, we got security update from Globus q q q We decided to patch VDT 1. 2. x, 1. 3. 9, 1. 3. 10, and 1. 3. 11 We need to build the updated binaries three times on about six platforms We need to coordinate creation of patch for Globus 2. 4. 3 We need to patch the Globus updates with our patches We need to take subset of Globus updates We need to package an update that can be reversed, if there are problems. The State of the VDT 7

How much software? The State of the VDT 8

Current Release: VDT 1. 3. 11 n Basis for OSG 0. 5. 0 n Adds some of requests for OSG 0. 6: q q q Globus 4. 0. 2 with Tera. Grid modifications SRM Clients Squid Wget New versions of CEMon and VOMS Support for Fedora Core 4 on x 86 -64 The State of the VDT 9

VDT 1. 3. 12: In Progress n Likely to include: q q q q n Gratia (for accounting) BDII (including Open. LDAP) glexec Edge Services Framework Tcl. Globus Improvements to upgrading the VDT A slew of bug fixes and smaller enhancements More at: http: //vdt. cs. wisc. edu/future. html Final decisions about content during this meeting The State of the VDT 10

VDT Stack => OSG Stack Input from stakeholders and OSG directors VDT Release OSG Integration Testbed Release OSG Production Release The State of the VDT 11

… Making a VDT release n n Requires at least one week of testing on VDT testbed Run nightly tests on each supported platform until they are clean The State of the VDT 12

… Making Int. Testbed Release Add software (mostly config) n Package subsets of VDT software n q q q Compute Element Storage Element Worker-node Client Submit-host Client … Deploy on testbed n Test services n Test middleware interoperability n Test functionality and scalability n Application validation n The State of the VDT 13

Challenges we face n How should we smoothly update a production service? q q q n In-place vs. on-the-side Preserve old configuration while making big changes. As easy as we try to make it, it still takes hours to fully install and set up from scratch How do we support more platforms? q q Fedora Core 6 It’s a struggle to keep up with the onslaught of Linux distributions Gentoo Fedora Core 5 AIX? Mac OS X? Solaris? The State of the VDT Fedora Core 4 RHEL 3 BCCD Debian Fedora Core 3 RHEL 4 RHEL 5 14

More challenges n Distributed creation of VDT q q n Improving testing q n Now we have funding for people to add support for new storage resource management systems (d-Cache, DRM, etc) These people are not at UW-Madison We care about interactions between the software: “When using a VOMS proxy with Condor-G, can we run a GT 4 job with Grid. FTP transfer, keeping the proxy in My. Proxy, while using PBS as the backend batch system…” Some people want native packaging formats q q RPM Deb The State of the VDT 15

Where do you learn more? n http: //vdt. cs. wisc. edu n Support: q q q Alain Roy: Miron Livny: Official Support: The State of the VDT roy@cs. wisc. edu miron@cs. wisc. edu vdt-support@ivdgl. org 16