Inte Grade ObjectOriented Grid Middleware Leveraging Idle Computing

Inte. Grade: Object-Oriented Grid Middleware Leveraging Idle Computing Power of Desktop Machines http: //gsd. ime. usp. br/Inte. Grade Andrei Goldchleger, Fabio Kon, Alfredo Goldman and Marcelo Finger {andgold, kon, gold, mfinger}@ime. usp. br Department of Computer Science IME/USP

Motivation need for computation • High demand for computationally-instensive applications – multimedia processing – scientific computing – finantial simulations and predictions – weather forecast – oil drilling – schedulling, planning, etc. 2

Motivation waste of resources • Corporations, universities, and government have hundreds or thousands of desktop computers for its employees and students. • Desktops are idle 99% of the time – idle at night (6 PM to 8 am) – idle during work hours – idle even when users are typing on the desktop keyboard • Dedicated clusters are idle most of the time generating heat and noise 3

Paradox 1. High demand for computatinal power 2. High level of idle resources • Third-world countries like Brazil cannot afford to waste resources like that. • Developed countries should also manage their resources better, at least for environmental reasons. Inte. Grade’s goal is to solve this paradox 4

Team Members o o Alfredo Goldman, Fabio Kon, Marcelo Finger e Siang W. Song (DCC – IME/USP) Markus Endler e Renato Cerqueira (DI – PUCRio) Edson Cáceres e Henrique Mongelli (DCT – UFMS) Approximately 10 graduate students 5

Inte. Grade: Description • Middleware to build a grid of commodity machines • Desktop users (Resource Providers) export their resources to the grid • Grid applications use only idle resources • Advantages over traditional dedicated clusters of commodity hardware

Inte. Grade: Key Features • Based on standard distributed object-oriented technology (CORBA) • Preserves resource provider’s Qo. S at all costs • Supports a wide range of parallel applications • Usage pattern collection and analysis

Inte. Grade: OO CORBA Middleware • Communication and architecture based on the CORBA industry standard – Object-orientation at all levels – Platform independent – Language independent • Leverages existing CORBA services (e. g. naming, trading, events, etc. ) • Export functionality as CORBA services • If desired, can also operate with other communication models – Sockets, MPI, BSP, etc. 8

Feature: Preserves Resource Provider’s Qo. S • User-level scheduler (DSRT) limits resource consumption of Grid applications • Lightweight CORBA ORB (O 2) • Configurable Resource Sharing (Optional)

Feature: Usage Pattern Collection and Analysis • Enhances scheduling by offering an approximate view of resource utilization • Usage Data is collected in short intervals (e. g. 5 min. ) and analysed • Data is grouped in larger intervals called periods • Clustering algorithms applied to data will derive behavioral categories (e. g. night, lunch-break, week-days, etc) • Each machine learns about the utilization of its resources and uses knowledge of past to predict the future

Feature: Support for a Wide Range of Parallel Applications • Often unsupported by other grid initiatives, especially ones that make opportunistic use of shared resources – In most Grid systems parallel applications must have little or no communication among application nodes • Inte. Grade research focuses on other kinds of parallel application (with communication) • Information about links interconnecting nodes must be collected and utilized for scheduling

Feature: Ensures Application Progress • Usage pattern collection and analysis provides hints, minimizing interruptions • Checkpointing for sequential applications – Must be implemented on a machine and OS independent way • Progress of parallel applications is more difficult to ensure, requiring global consistent checkpoints • Possible solution: use BSP as parallel application model

Architecture: Intra. Cluster LRM - Local Resource Manager GRM - Global Resource Manager

Architecture: Intra. Cluster LUPA - Local Usage Pattern Analyzer GUPA - Global Usage Pattern Analyzer

Architecture: Intra. Cluster NCC - Node Control Center ASCT - Application Submission and Control Tool

Architecture: Inter. Cluster 16

Related Work • Our work is influenced by 5 systems: – Globus, Legion, Condor, SETI@home, and 2 K • Condor (U. of Wisconsin-Madison) – Pioneer (started on late 80 s) – A “hunter” of idle workstations on local networks – Condor-G interfaces with Globus for integration with wide-area grids – Support for parallel applications is limited – We could not get its source-code • Globus (Argonne National Labs / U. of Chicago / USC) – Does not focus on Qo. S-preserving utilization of desktop machines – Not object-oriented – Inte. Grade uses CORBA and OO design

Related Work (continued) • Legion (U. of Virginia) – Proprietary distributed object model – Inte. Grade has deeper focus on idle resource management and desktop machines • SETI@Home (U. of California Berkeley) – Hard-coded application – No communication between application nodes • BOINC (U. of California Berkeley) – Limited support for parallel applications

Related Work (continued) • 2 K (U. of Illinois at Urbana-Champaign) – a CORBA-based distributed operating system – does not focus on grid computing or parallel applications – provided a proof-of-concept prototype for some of the protocols we are using in Inte. Grade

Implementation Status • Already Implemented: – Intra-Cluster Information Update Protocol – Intra-Cluster Execution Protocol • Sequential applications • Parametric applications • Software used: – GRM: Java using Jac. ORB – LRM: C++ using O 2

Implementation Status: Cluster. View

Ongoing sub-projects • Refinements and extensions to architecture and core software infrastructure • Initial support for parallel applications • Network discovery and monitoring • User usage pattern collection and analysis • Global, wide-area scheduling • Migration and mobile agents • lightweight middleware • autonomic computing – self-awareness, self-healing, self-adaptation • Security and privacy 22

Project Information • www. ime. usp. br/integrade • Source code available at FAPESP’s incubadora (anonymous CVS checkout & Web front end) • Increasing number of students working on the project • Initial beta version expected for the end of 2003 (alpha version already up and running)