From Compute Intensive to Data Intensive Grid Computing

From Compute Intensive to Data Intensive Grid Computing Miron Livny Computer Sciences Department University of Wisconsin-Madison miron@cs. wisc. edu http: //www. cs. wisc. edu/~miron

2 nd Welcome to the Annual Condor meeting!!! (Hope to see you all in the Euro. Globus+Condor meeting in southern Italy in mid June) www. cs. wisc. edu/condor

The Condor Project (Established ‘ 85) Distributed Computing research performed by a team of 26 faculty, full time staff and students who hface software engineering challenges in a UNIX/Linux/NT environment, hare involved in national and international collaborations, hactively interact with academic and commercial users, hmaintain and support a large distributed production environment, hand educate and train students. Funding - Do. D, Do. E, NASA, NSF, AT&T, IBM, INTEL, Microsoft and the UW Graduate School www. cs. wisc. edu/condor

State of the project › Extended the scope and size of the project › Extremely busy › Defended and preserved our unique approach to › › › distributed/Grid computing Enjoyed the impact we have on “real” users Maintained the reputation of our research and the quality of our technology Worked hard to make a commercial Condor fly alongside our project www. cs. wisc. edu/condor

The Physicists & Us › Some of us have our roots in physics (Derek, Doug, Miron, Raj) › Long history of joint work (NIKHF 91, INFN 96, PPDG 99, SMC, › › › WA 92 COROUS, PHINEX, CMS …) Grid Physics Network (Gri. Phy. N) project funded by the Information Technology Research program of NSF ($10. 9 M for five years $2 M to our group) Joint UW-HEP and Condor project funded by the UW Graduate School. 1 M hours of Condor CPUs were allocated by the National Resource Allocation Board (NREC) to the CMS group at Caltech Our team selected as the main CS group for the next phase of PPDG (DOE-Sci. DAC proposal pending) CMS group of UW is one of the four sites of the Grid Laboratory of Wisconsin (GLOW) (NSF-MRI proposal pending) www. cs. wisc. edu/condor

Physicists Have a LOT of data. (others have data problems too …) www. cs. wisc. edu/condor

The Data Grids & Us › Extend and enhance existing capabilities of Condor › Develop new storage and data transfer management › › technologies (key element in PPDG proposal) Interface “legacy” applications and the Grid “fabric” with new capabilities/technologies Work closely with the Globus team and follow their leadership on data transfer and replica management tools (joint Data Grid Sci. DAC proposal is pending) Close collaboration with the EU Data Grid effort Make Class. Ads the “Esperanto” of Grid management (joint Security Sci. DAC proposal with Globus is pending) www. cs. wisc. edu/condor

And what about … › Allocating Network Resources, › Scheduling of Storage Resources, › Co-allocation of Processing, Storage and Network › › › Resources, Resource Reservation, Data Shipping vs. Function Shipping, Data Security, Write Behind, Data Affinity, Or … ? www. cs. wisc. edu/condor

Challenges Ahead › Ride the “Grid Wave” without losing our balance › Leverage the talent and expertise of our new faculty › › (Distributed I/O, Distributed Scheduling, Networking, Security) Expend our UW-Flock to a state-wide system (Wisc. Net? ) Apply the Master Worker paradigm to domain decomposition problems (ongoing work with JPL) Scale our Master Worker framework to 10, 000 workers. Open Source vs. Public Domain binaries vs. a Commercial version of Condor www. cs. wisc. edu/condor

Don’t ask “what can the Grid do for me? ”ask “what can I do with the Grid? ” www. cs. wisc. edu/condor