NC Bio Grid Infrastructure Chuck Kesler Systems Architect

NC Bio. Grid Infrastructure Chuck Kesler Systems Architect MCNC / NCSC

Presentation Overview The NC Bio. Grid Infrastructure Project NC Bio. Grid Infrastructure Objectives Grid Infrastructure Platforms

The NC Bio. Grid Infrastructure Project

Project Approach Form Working Group(s) Perform High Level Requirements Study Identify Technology for the Test Bed Plan and Deploy the Test Bed Use Test Bed to Refine User and IT Requirements Plan and Deploy the Production Environment

Current State Working Groups Not Yet Formed No Technology Decisions Have Been Made High Level Characterization of Some Problems, Requirements and Solutions Identified Some Candidate Technologies Preliminary Conversations with Some Potential Vendors

NC Bio. Grid Infrastructure Objectives

“Blue Sky” Infrastructure Objectives Create Virtual Organizations Develop and Maintain Production Quality Support Heterogeneous Resources Single Sign-On Unified View of File Systems and Databases Unified View of Job Management Avoid Re-Inventing the Wheel

Create Virtual Organizations Definition: a Work or Project Group Made up of Participants from Different Institutions Present a Common View of Remote and Local Computing Resources Allow for Local Control of Local Resources / Autonomous Decision and Policy Making Shared Infrastructure for Complex and/or Expensive Services (e. g. HPC, Data Backups, Disaster Recovery, Storage Area Networks, Network Operations Center) Blends P 2 P (Peer-to-Peer) Computing and ASP (Application Service Provider) Models

Develop and Maintain Production Quality Standardize on Stable Hardware and Software Configurations Insure that Appropriate Usage, Operational and Security Policies are in Place Network and Systems Health Monitoring Intrusion Detection Troubleshooting and Incident Response Teams Scientific User Support

Support Heterogeneous Resources Major OS Platforms (e. g. Solaris, Linux, Windows 2000, AIX, etc…) File Systems (e. g. AFS, NFS, UFS, NTFS) Authentication and Authorization (e. g. UNIX login, Kerberos, Active Directory, X. 509 Certs, Smart Cards) Job Management (e. g. Load. Leveler, NQE, LSF, PBS)

Unified View of File Systems and Databases Bridge Different Types of Local File Systems Provide a Global Name Space Find the Optimal Resource for a User Policy-Based Replication and/or Local Caching Virtualize Databases

Unified View of Job Management Bridge Different Types of Local Job Scheduling Systems Locate Compute Servers Based on User or Policy Criteria (e. g. Load, CPU Type, OS) Provide Standard Job Management Functionality (e. g. Start / Stop Jobs, Check Status) Handle Remote “File Staging” Provide Usage-Based Accounting

Avoid Re-Inventing the Wheel Embrace Existing Technology Solutions Currently Agnostic with Respect to Open Source vs. Commercial Software Arguments From an IT Perspective, this is not primarily a Code Development Project; its Focus is Systems Integration

Grid Infrastructure Platforms

Types of Grid Platforms Point Solutions Job Schedulers Peer-to-Peer Web Services Toolkits Integrated Solutions

Point Solutions Example: LION SRS Bioinformatics Specific Unified View of and Access to Life Sciences Databases Model is Not Easily Applied to Other Kinds of Problems

Job Schedulers Examples: Condor, LSF, Sun Grid Engine, PBS, NQE, Load. Leveler High Performance Computing Oriented Works Well in Tightly Coupled Clusters Not Optimal for Data Management

Peer-to-Peer Examples: Entropia, Parabon, United Devices, Data Synapse Compute Oriented: Aggregates Cycles on Low-End PC’s Bioinformatics Application: Turbo. BLAST See Also: Folding@Home, Compute Against Cancer, Fight. AIDS@Home, SETI@Home

Web Services General Examples: XML, WSDL, SOAP, UDDI Bioinformatics Examples: GEML, BSML W 3 C Standards-Based Industry Convergence Not Here Yet, But May Be The Future…

Toolkits Example: The Globus Project Open Source, Promoting the Development of Standards. Based Grid Protocols Significant Momentum Flexibility: Allows for Local Development or Using Existing PD Code (e. g. SRB, Grid. Port, MPICH-G 2) But, Out-of-the-Box Functionality is Limited This is a Research Project in Progress Cost Associated with Developing Local Glueware (Currently) Limited Windows Support

Integrated Solutions Example: AVAKI (a. k. a. Legion from UVa) Commercial, Vendor-Supported Software, but Not Open Source Provides Significant Out-of-the-Box Functionality, Including a Distributed File System Global Name Space for Resources Good Support for UNIX and Windows Planned Evolution Towards Web Services (Some) Licensing Costs Not as Much Mind Share as Globus

Conclusions Understanding User Requirements is Critical This Must be a Collaborative Project Key Decisions Need to be Made Soon n Approach: Point Solution vs. General Solution Grid Infrastructure Platform NC Bio. Grid Test Bed Resources There is Much Work to be Done!