The Ibis eScience Software Framework Henri Bal Frank

The Ibis e-Science Software Framework Henri Bal, Frank J. Seinstra, Jason Maassen, Niels Drost High Performance Distributed Computing Group Department of Computer Science VU University, Amsterdam, The Netherlands

Introduction ● Distributed systems continue to change ● ● Distributed applications continue to change ● ● Clusters, grids, clouds, mobile devices e-Science, web, pervasive applications Distributed programming continues to be notoriously difficult

Distributed Systems: 1980 s Multiple PCs on a (local) network ● ● ● Networks of Workstations (NOWs) Collections of Workstations (COWs) Processor pools Condor pools Clusters

Distributed Systems: 1990 s Sharing wide-area resources ● ● ● Metacomputing (Smarr & Catlett, CACM) Flocking Condor (Epema) DAS (Distributed ASCI Supercomputer) Grid Blueprint (Foster & Kesselman) Desktop grids, SETI@home

Distributed Systems: 2000 s ● Cloud computing ● ● ● Pay-on-demand Virtualization Hardware diversity /heterogeneous computing Green IT The Networked World ● ● Sensor networks Smart phones

Our approach ● ● ● Study fundamental underlying problems … hand-in-hand with realistic applications … integrate solutions in one system: Ibis ! User Distributed Systems ● Funding from NWO (2002), VL-e (2003 -2009), EU (Java. GAT, Xtreem. OS, Contrail), VU, COMMIT

Outline ● ● ● ‘Problem Solving’ vs. ‘System Fighting’ Jungle Computing Example applications: ● ● The Ibis Software Framework The 3 Common Uses of Ibis ● ● Computational Astrophysics Multimedia Content Analysis ‘Master Key’ + ‘Glue’ + ‘HPC’ Some current work: Green Clouds

Ibis: ‘Problem Solving’ vs. ‘System Fighting’

A Random Example: Supernova Detection ● DACH 2008, Japan ● ● Distributed multi-cluster system ● Heterogeneous Distributed database (image pairs) ● Large vs small databases/images ● Partial replication Image-pair comparison given (in C) Find all supernova candidates ● ● Task 1: As fast as possible Task 2: Idem, under system crashes

‘Problem Solving’ vs. ‘System Fighting’ ● All participating teams struggled (1 month) ● ● Middleware instabilities… Connectivity problems… Load balancing… But not the Ibis team ● ● ● Winner (by far) in both categories Note: many Japanese teams with years of experience ● Hardware, middleware, network, C-code, image data… Focus on ‘problem solving’, not ‘system fighting’ ● incl. ‘opening’ of black-box C-code

Ibis Results: Awards & Prizes AAAI-VC 2007 Most Visionary Research Award 1 st Prize: DACH 2008 - BS 1 st Prize: DACH 2008 - FT Web. Pie: A Web-Scale Parallel Inference Engine J. Urbani, S. Kotoulas, J. Maassen, N. Drost, F. J. Seinstra, F. van Harmelen, and H. E. Bal 3 rd Prize: ISWC 2008 1 st Prize: SCALE 2010 ● Many domains; data/compute intensive, real-time. . . ● Winner Sustainability Award in the Enlighten Your Research (EYR) competition, 7 Dec. 2011 (Frank Seinstra)

Ibis Users… …and many more

Jungle Computing

Jungle Computing (Frank Seinstra) ● ‘Worst case’ computing as required by end-users ● ● ● Distributed Heterogeneous Hierarchical (incl. multi-/many-cores)

Why Jungle Computing? ● Scientists often forced to use a wide variety of resources simultaneously to solve computational problems, e. g. due to: ● ● ● ● Desire for scalability Distributed nature of (input) data Software heterogeneity (e. g. : mix of C/MPI and CUDA) Ad hoc hardware availability Energy consumption (use most energy-efficient resource) … Note: most users do not need ‘worst case’ jungle ● Ibis aims to apply to any subset

Example Application Domains ● Computational Astrophysics (Leiden) ● ● ● AMUSE: multi-model / multi-kernel simulations “Simulating the Universe on an Intercontinental Grid” - Portegies Zwart et al (IEEE Computer, Aug 2010) Climate Modeling (Utrecht) ● CPL: multi-model / multi-kernel simulations ● Atmosphere, ocean, source rock formation, … - hardware: (potentially) very diverse - high resolution => speed & scalability -…

Domain Example #1: Computational Astrophysics

Domain Example #1: Computational Astrophysics Demonstrated live at SC’ 11, Nov 12 -18, 2011, Seattle, USA (two week ago)

Domain Example #1: Computational Astrophysics ● The AMUSE system (Leiden University) ● Early Star Cluster Evolution, including gas gravitational dynamics stellar evolution AMUSE hydrodynamics radiative transport ● ● ● Gravitational dynamics (N-body): GPU / GPU-cluster Stellar evolution: Beowulf cluster / Cloud Hydro-dynamics, Radiative transport: Supercomputer

Domain Example #1: Computational Astrophysics Demonstrated live at SC’ 11, Nov 12 -18, 2011, Seattle, USA

Domain Example #2: Multimedia Content Analysis

Multimedia Content Analysis (MMCA) ● Aim: ● Automatic extraction of ‘semantic concepts’ from image sets and video streams ● Depending on specific problem & size of data set: ● May take hours, days, weeks, months, years…

Multimedia Content Analysis (MMCA) ● Applications in (a. o): ● ● ● ● Remote Sensing Security / Surveillance Medical Imaging Document Analysis Multimedia Systems Astronomy Application types: ● ● ● Real-time vs. off-line Fine-grained vs. coarse-grained Data-intensive / compute-intensive / information-intensive

Domain Example #2: Color-based Object Recognition by a Grid-connected Robot Dog Seinstra et al (IEEE Multimedia, Oct-Dec 2007) Seinstra et al (AAAI’ 07: Most Visionary Research Award)

Successful… ● …but many fundamental problems unsolved! ● ● ● ● Scaling up to very large systems Platform independence Middleware independence Connectivity (a. o. firewalls, …) Fault-tolerance … Software support tool(s) urgently needed! ● ● Jungle-aware + transparent + efficient No progress until ‘discovery’ of Ibis

The Ibis Software Framework

The Ibis Software Framework ● Offers all functionality to efficiently & transparently implement & run Jungle Computing applications ● Designed for dynamic / hostile environments ● Modular and flexible ● ● Allow replacement of Ibis components by external ones, including native code Open source ● Download: http: //www. cs. vu. nl/ibis/

Ibis Design ● Applications need functionality for ● ● Programming (as in programming languages) Deployment (as in operating systems) Programming Deployment Logical Practical Likes math Visual (GUI)

Ibis Software Stack

Java. GAT ● Java Grid Application Toolkit ● ● ● High-level API for developing (Grid) applications independently of the underlying (Grid) middleware Use (Grid) services; file cp, resource discovery, job submission, … Note: SAGA API standardized by OGF ● ● Simple API for Grid Applications (a. o. with LSU) SAGA on top of Java. GAT (and v. v. )

Zorilla ● A prototype P 2 P middleware ● ● ● A Zorilla system consists of a collection of nodes, connected by a P 2 P network Each node independent & implements all middleware functionality No central components Supports fault-tolerance and malleability Easily combines resources in multiple administrative domains

Ibis. Deploy

Ibis Portability Layer (IPL) ● Java-centric ‘run-anywhere’ communication library ● ● ● Supports fault-tolerance and malleability ● ● ● Sent along with your application “MPI for the Grid” Resource tracking (Join-Elect-Leave model) Open-world / Closed world Efficient ● ● Highly optimized object serialization Can use optimized native libraries (e. g. MPI, Infiniband)

Smart. Sockets ● Robust connection setup Problems: Firewalls Network Address Translation (NAT) Non-routed networks Multi-homing … ● Always connection in 30 different scenarios

Ibis Programming Models ● IPL-based programming models, a. o. : ● ● Satin: ● A divide-and-conquer model MPJ: ● The MPI binding for Java RMI: ● Object-Oriented remote Procedure Call Jorus: ● A ‘user transparent’ parallel model for multimedia applications

The 3 Common Uses of Ibis

Ibis as ‘Master Key’ (or ‘Passepartout’) ● Use Java. GAT to access ‘any’ system ● ● Develop/run applications independently of available middlewares Java. GAT ‘adaptors’ required for each middleware ‘Intelligent dispatching’ even allows for transparent use of multiple middlewares Example: file copy ● Java. GAT vs. Globus package tutorial; import org. gridlab. gat. GATContext; import org. gridlab. gat. URI; import org. gridlab. gat. io. File; public class Remote. Copy { public static void main(String[] args) throws Exception { GATContext context = new GATContext(); URI src = new URI(args[0]); URI dest = new URI(args[1]); File file = GAT. create. File(context, src); file. copy(dest); GAT. end(); } } ● ● Simple, portable, … SAGA API standardized package org. gridlab. gat. io. cpi. rftgt 4; import java. net. Malformed. URLException; import java. net. URL; import java. rmi. Remote. Exception; import java. security. cert. X 509 Certificate; import java. util. Calendar; import java. util. Hash. Map; import java. util. Linked. List; import java. util. Map; import java. util. Vector; import javax. xml. namespace. QName; import javax. xml. rpc. Service. Exception; import javax. xml. rpc. Stub; import javax. xml. soap. SOAPElement; import org. apache. axis. message. addressing. Endpoint. Reference. Type; import org. apache. axis. types. URI. Malformed. URIException; import org. globus. axis. util. Util; import org. globus. delegation. Delegation. Constants; import org. globus. delegation. Delegation. Exception; import org. globus. delegation. Delegation. Util; import org. globus. gsi. Globus. Credential. Exception; import org. globus. gsi. gssapi. Globus. GSSCredential. Impl; import org. globus. gsi. jaas. Jaas. Gss. Util; import org. globus. rft. generated. Base. Request. Type; import org. globus. rft. generated. Create. Reliable. File. Transfer. Input. Type; import org. globus. rft. generated. Create. Reliable. File. Transfer. Output. Type; import org. globus. rft. generated. Delete. Request. Type; import org. globus. rft. generated. Delete. Type; import org. globus. rft. generated. Overall. Status; import org. globus. rft. generated. RFTFault. Resource. Property. Type; import org. globus. rft. generated. RFTOptions. Type; import org. globus. rft. generated. Reliable. File. Transfer. Factory. Port. Type; import org. globus. rft. generated. Reliable. File. Transfer. Port. Type; import org. globus. rft. generated. Start; import org. globus. rft. generated. Transfer. Request. Type; import org. globus. rft. generated. Transfer. Type; import org. globus. transfer. reliable. client. Base. RFTClient; import org. globus. transfer. reliable. service. RFTConstants; import org. globus. wsrf. Notification. Consumer. Manager; import org. globus. wsrf. Notify. Callback; import org. globus. wsrf. Resource. Exception; import org. globus. wsrf. WSNConstants; import org. globus. wsrf. container. Container. Exception; import org. globus. wsrf. container. Service. Container; import org. globus. wsrf. core. notification. Resource. Property. Value. Change. Notification. Element. Type; import org. globus. wsrf. encoding. Deserialization. Exception; import org. globus. wsrf. encoding. Object. Deserializer; import org. globus. wsrf. impl. security. authentication. Constants; import org. globus. wsrf. impl. security. authorization. Authorization; import org. globus. wsrf. impl. security. authorization. Host. Authorization; import org. globus. wsrf. impl. security. authorization. Identity. Authorization; import org. globus. wsrf. impl. security. authorization. Self. Authorization; import org. globus. wsrf. impl. security. descriptor. Client. Security. Descriptor; import org. globus. wsrf. impl. security. descriptor. Container. Security. Descriptor; import org. globus. wsrf. impl. security. descriptor. GSISecure. Msg. Auth. Method; import org. globus. wsrf. impl. security. descriptor. GSITransport. Auth. Method; import org. globus. wsrf. impl. security. descriptor. Resource. Security. Descriptor; import org. globus. wsrf. impl. security. descriptor. Security. Descriptor. Exception; import org. globus. wsrf. security. Security. Manager; import org. gridlab. gat. Could. Not. Initialize. Credential. Exception; import org. gridlab. gat. Credential. Expired. Exception; import org. gridlab. gat. GATContext; import org. gridlab. gat. GATInvocation. Exception; import org. gridlab. gat. GATObject. Creation. Exception; import org. gridlab. gat. Preferences; import org. gridlab. gat. URI; import org. gridlab. gat. io. cpi. File. Cpi; import org. gridlab. gat. security. globus. Globus. Security. Utils; import org. ietf. jgss. GSSCredential; import org. ietf. jgss. GSSException; import org. oasis. wsn. Subscribe; import org. oasis. wsn. Topic. Expression. Type; import org. oasis. wsrf. faults. Base. Fault. Type; import org. oasis. wsrf. lifetime. Set. Termination. Time; import org. oasis. wsrf. properties. Get. Multiple. Resource. Properties. Response; import org. oasis. wsrf. properties. Get. Multiple. Resource. Properties_Element; import org. oasis. wsrf. properties. Resource. Property. Value. Change. Notification. Type; class RFTGT 4 Notify. Callback implements Notify. Callback { RFTGT 4 File. Adaptor transfer; Overall. Status status; public RFTGT 4 Notify. Callback(RFTGT 4 File. Adaptor transfer) { super(); this. transfer = transfer; this. status = null; } @Suppress. Warnings("unchecked") public void deliver(List topic. Path, Endpoint. Reference. Type producer, Object message. Wrapper) { try { Resource. Property. Value. Change. Notification. Type message = ((Resource. Property. Value. Change. Notification. Element. Type) message. Wrapper) . get. Resource. Property. Value. Change. Notification(); this. status = (Overall. Status) message. get. New. Value(). get_any()[0] . get. Value. As. Type(RFTConstants. OVERALL_STATUS_RESOURCE, Overall. Status. class); if (status. get. Fault() != null) { transfer. set. Fault(get. Fault. From. RP(status. get. Fault())); } // Run. Queue. get. Instance(). add(this. resource. Key); } catch (Exception e) { } transfer. set. Status(status); } private Base. Fault. Type get. Fault. From. RP(RFTFault. Resource. Property. Type fault) { if (fault == null) { return null; } if (fault. get. Delegation. EPRMissing. Fault. Type() != null) { return fault. get. Delegation. EPRMissing. Fault. Type(); } else if (fault. get. Rft. Authentication. Fault. Type() != null) { return fault. get. Rft. Authentication. Fault. Type(); } else if (fault. get. Rft. Authorization. Fault. Type() != null) { return fault. get. Rft. Authorization. Fault. Type(); } else if (fault. get. Rft. Database. Fault. Type() != null) { return fault. get. Rft. Database. Fault. Type(); } else if (fault. get. Rft. Repeatedly. Started. Fault. Type() != null) { return fault. get. Rft. Repeatedly. Started. Fault. Type(); } else if (fault. get. Transfer. Transient. Fault. Type() != null) { return fault. get. Transfer. Transient. Fault. Type(); } else if (fault. get. Rft. Transfer. Fault. Type() != null) { return fault. get. Rft. Transfer. Fault. Type(); } else { return null; } } } @Suppress. Warnings("serial") public class RFTGT 4 File. Adaptor extends File. Cpi { public static final Authorization DEFAULT_AUTHZ = Host. Authorization . get. Instance(); Integer msg. Protection. Type = Constants. SIGNATURE; static final int TERM_TIME = 20; static final String PROTOCOL = "https"; private static final String BASE_SERVICE_PATH = "/wsrf/services/"; public static final int DEFAULT_DURATION_HOURS = 24; public static final Integer DEFAULT_MSG_PROTECTION = Constants. SIGNATURE; public static final String DEFAULT_FACTORY_PORT = "8443"; private static final int DEFAULT_GRIDFTP_PORT = 2811; Notification. Consumer. Manager notification. Consumer. Manager; Endpoint. Reference. Type notification. Consumer. EPR; Endpoint. Reference. Type notification. Producer. EPR; String security. Type; String factory. Url; GSSCredential proxy; Authorization authorization; String host; Overall. Status status; Base. Fault. Type fault; String location. Str; Reliable. File. Transfer. Factory. Port. Type factory. Port; public RFTGT 4 File. Adaptor(GATContext gat. Context, Preferences preferences, URI location) throws GATObject. Creation. Exception { super(gat. Context, preferences, location); if (!location. is. Compatible("gsiftp") && !location. is. Compatible("gridftp")) { throw new GATObject. Creation. Exception("cannot handle this URI"); } String globus. Location = System. getenv("GLOBUS_LOCATION"); if (globus. Location == null) { throw new GATObject. Creation. Exception("$GLOBUS_LOCATION is not set"); } System. set. Property("GLOBUS_LOCATION", globus. Location); System. set. Property("axis. Client. Config. File", globus. Location + "/client-config. wsdd"); this. host = location. get. Host(); this. security. Type = Constants. GSI_SEC_MSG; this. authorization = null; this. proxy = null; try { proxy = Globus. Security. Utils. get. Globus. Credential(gat. Context, preferences, "globus", location, DEFAULT_GRIDFTP_PORT); } catch (Could. Not. Initialize. Credential. Exception e) { // TODO Auto-generated catch block e. print. Stack. Trace(); } catch (Credential. Expired. Exception e) { // TODO Auto-generated catch block e. print. Stack. Trace(); } this. notification. Consumer. Manager = null; this. notification. Consumer. EPR = null; this. notification. Producer. EPR = null; this. status = null; this. fault = null; factory. Port = null; this. factory. Url = PROTOCOL + ": //" + host + ": " + DEFAULT_FACTORY_PORT + BASE_SERVICE_PATH + RFTConstants. FACTORY_NAME; location. Str = set. Location. Str(location); } String set. Location. Str(URI location) { if (location. get. Scheme(). equals("any")) { return "gsiftp: //" + location. get. Host() + ": " + location. get. Port() + "/" + location. get. Path(); } else { return location. to. String(); } } protected boolean copy 2(String dest. Str) throws GATInvocation. Exception { Endpoint. Reference. Type credential. Endpoint = get. Credential. EPR(); Transfer. Type[] transfer. Array = new Transfer. Type[1]; transfer. Array[0] = new Transfer. Type(); transfer. Array[0]. set. Source. Url(location. Str); transfer. Array[0]. set. Destination. Url(dest. Str); RFTOptions. Type rft. Options = new RFTOptions. Type(); rft. Options. set. Binary(Boolean. TRUE); // rft. Options. set. Ignore. File. Perm. Err(false); Transfer. Request. Type request = new Transfer. Request. Type(); request. set. Rft. Options(rft. Options); request. set. Transfer(transfer. Array); request. set. Transfer. Credential. Endpoint(credential. Endpoint); set. Request(request); while (!transfers. Done()) { try { Thread. sleep(1000); } catch (Interrupted. Exception e) { throw new GATInvocation. Exception("RFTGT 4 File. Adaptor: " + e); } } return transfers. Succ(); } public void copy(URI dest) throws GATInvocation. Exception { String dest. Url = set. Location. Str(dest); if (!copy 2(dest. Url)) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: file copy failed"); } } public void subscribe(Reliable. File. Transfer. Port. Type rft) throws GATInvocation. Exception { Map<Object, Object> properties = new Hash. Map<Object, Object>(); properties. put(Service. Container. CLASS, "org. globus. wsrf. container. GSIService. Container"); if (this. proxy != null) { Container. Security. Descriptor container. Sec. Desc = new Container. Security. Descriptor(); Security. Manager. get. Manager(); try { container. Sec. Desc. set. Subject(Jaas. Gss. Util . create. Subject(this. proxy)); } catch (GSSException e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: Container. Security. Descriptor failed, " + e); } properties. put(Service. Container. CONTAINER_DESCRIPTOR, container. Sec. Desc); } this. notification. Consumer. Manager = Notification. Consumer. Manager . get. Instance(properties); try { this. notification. Consumer. Manager. start. Listening(); } catch (Container. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: Notification. Consumer. Manager failed, " + e); } List<Object> topic. Path = new Linked. List<Object>(); topic. Path. add(RFTConstants. OVERALL_STATUS_RESOURCE); Resource. Security. Descriptor security. Descriptor = new Resource. Security. Descriptor(); String authz = null; if (authorization == null) { authz = Authorization. AUTHZ_NONE; } else if (authorization instanceof Host. Authorization) { authz = Authorization. AUTHZ_NONE; } else if (authorization instanceof Self. Authorization) { authz = Authorization. AUTHZ_SELF; } else if (authorization instanceof Identity. Authorization) { // not supported throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: identity authorization not supported"); } else { // throw an sg throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set authorization failed"); } security. Descriptor. set. Authz(authz); Vector<Object> auth. Method = new Vector<Object>(); if (this. security. Type. equals(Constants. GSI_SEC_MSG)) { auth. Method. add(GSISecure. Msg. Auth. Method. BOTH); } else { auth. Method. add(GSITransport. Auth. Method. BOTH); } try { security. Descriptor. set. Auth. Methods(auth. Method); } catch (Security. Descriptor. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set. Auth. Methods failed, " + e); } RFTGT 4 Notify. Callback notify. Callback = new RFTGT 4 Notify. Callback(this); try { notification. Consumer. EPR = notification. Consumer. Manager . create. Notification. Consumer(topic. Path, notify. Callback, security. Descriptor); } catch (Resource. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: create. Notification. Consumer failed, " + e); } Subscribe subscription. Request = new Subscribe(); subscription. Request. set. Consumer. Reference(notification. Consumer. EPR); Topic. Expression. Type topic. Expression = null; try { topic. Expression = new Topic. Expression. Type( WSNConstants. SIMPLE_TOPIC_DIALECT, RFTConstants. OVERALL_STATUS_RESOURCE); } catch (Malformed. URIException e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: create Topic. Expression. Type failed, " + e); } subscription. Request. set. Topic. Expression(topic. Expression); try { rft. subscribe(subscription. Request); } catch (Remote. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: subscription failed, " + e); } } protected Endpoint. Reference. Type get. Credential. EPR() throws GATInvocation. Exception { this. status = null; URL factory. URL = null; try { factory. URL = new URL(factory. Url); } catch (Malformed. URLException e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set factory. URL failed, " + e); } try { factory. Port = Base. RFTClient. rft. Factory. Locator . get. Reliable. File. Transfer. Factory. Port. Type. Port(factory. URL); } catch (Service. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set factory. Port failed, " + e); } set. Security. Type. From. URL(factory. URL); return populate. RFTEndpoints(factory. Port); } protected void set. Request(Base. Request. Type request) throws GATInvocation. Exception { Create. Reliable. File. Transfer. Input. Type input = new Create. Reliable. File. Transfer. Input. Type(); if (request instanceof Transfer. Request. Type) { input. set. Transfer. Request((Transfer. Request. Type) request); } else { input. set. Delete. Request((Delete. Request. Type) request); } Calendar term. Time. Del = Calendar. get. Instance(); term. Time. Del. add(Calendar. MINUTE, TERM_TIME); input. set. Initial. Termination. Time(term. Time. Del); Create. Reliable. File. Transfer. Output. Type response = null; try { response = factory. Port. create. Reliable. File. Transfer(input); } catch (Remote. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set create. Reliable. File. Transfer failed, " + e); } Endpoint. Reference. Type reliable. RFTEndpoint = response . get. Reliable. Transfer. EPR(); Reliable. File. Transfer. Port. Type rft = null; try { rft = Base. RFTClient. rft. Locator . get. Reliable. File. Transfer. Port. Type. Port(reliable. RFTEndpoint); } catch (Service. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: get. Reliable. File. Transfer. Port. Type. Port failed, " + e); } set. Stub. Security. Properties((Stub) rft); subscribe(rft); Calendar term. Time = Calendar. get. Instance(); term. Time. add(Calendar. MINUTE, TERM_TIME); Set. Termination. Time req. Term. Time = new Set. Termination. Time(); req. Term. Time. set. Requested. Termination. Time(term. Time); try { rft. set. Termination. Time(req. Term. Time); } catch (Remote. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: set. Termination. Time failed, " + e); } try { rft. start(new Start()); } catch (Remote. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: start failed, " + e); } } private void set. Security. Type. From. URL(URL url) { if (url. get. Protocol(). equals("http")) { security. Type = Constants. GSI_SEC_MSG; } else { Util. register. Transport(); security. Type = Constants. GSI_TRANSPORT; } } private void set. Stub. Security. Properties(Stub stub) { Client. Security. Descriptor sec. Desc = new Client. Security. Descriptor(); if (this. security. Type. equals(Constants. GSI_SEC_MSG)) { sec. Desc. set. GSISecure. Msg(this. get. Message. Protection. Type()); } else { sec. Desc. set. GSITransport(this. get. Message. Protection. Type()); } sec. Desc. set. Authz(get. Authorization()); if (this. proxy != null) { // set proxy credential sec. Desc. set. GSSCredential(this. proxy); } stub. _set. Property(Constants. CLIENT_DESCRIPTOR, sec. Desc); } public Integer get. Message. Protection. Type() { return (this. msg. Protection. Type == null) ? RFTGT 4 File. Adaptor. DEFAULT_MSG_PROTECTION : this. msg. Protection. Type; } public Authorization get. Authorization() { return (authorization == null) ? DEFAULT_AUTHZ : this. authorization; } private Endpoint. Reference. Type populate. RFTEndpoints( Reliable. File. Transfer. Factory. Port. Type factory. Port) throws GATInvocation. Exception { Endpoint. Reference. Type[] delegation. Factory. Endpoints = fetch. Delegation. Factory. Endpoints(factory. Port); Endpoint. Reference. Type delegation. Endpoint = delegate(delegation. Factory. Endpoints[0]); return delegation. Endpoint; } private Endpoint. Reference. Type delegate( Endpoint. Reference. Type delegation. Factory. Endpoint) throws GATInvocation. Exception { Globus. Credential credential = null; if (this. proxy != null) { credential = ((Globus. GSSCredential. Impl) this. proxy) . get. Globus. Credential(); } else { try { credential = Globus. Credential. get. Default. Credential(); } catch (Globus. Credential. Exception e) { throw new GATInvocation. Exception("RFTGT 4 File. Adaptor: " + e); } } int lifetime = DEFAULT_DURATION_HOURS * 60; Client. Security. Descriptor sec. Desc = new Client. Security. Descriptor(); if (this. security. Type. equals(Constants. GSI_SEC_MSG)) { sec. Desc. set. GSISecure. Msg(this. get. Message. Protection. Type()); } else { sec. Desc. set. GSITransport(this. get. Message. Protection. Type()); } sec. Desc. set. Authz(get. Authorization()); if (this. proxy != null) { sec. Desc. set. GSSCredential(this. proxy); } // Get the public key to delegate on. X 509 Certificate[] certs. To. Delegate. On = null; try { certs. To. Delegate. On = Delegation. Util. get. Certificate. Chain. RP( delegation. Factory. Endpoint, sec. Desc); } catch (Delegation. Exception e) { throw new GATInvocation. Exception("RFTGT 4 File. Adaptor: " + e); } X 509 Certificate cert. To. Sign = certs. To. Delegate. On[0]; // FIXME remove when there is a Delegation. Util. delegate(EPR, . . . ) String protocol = delegation. Factory. Endpoint. get. Address(). get. Scheme(); String host = delegation. Factory. Endpoint. get. Address(). get. Host(); int port = delegation. Factory. Endpoint. get. Address(). get. Port(); String factory. Url = protocol + ": //" + host + ": " + port + BASE_SERVICE_PATH + Delegation. Constants. FACTORY_PATH; // send to delegation service and get epr. Endpoint. Reference. Type credential. Endpoint = null; try { credential. Endpoint = Delegation. Util. delegate(factory. Url, credential, cert. To. Sign, lifetime, false, sec. Desc); } catch (Delegation. Exception e) { throw new GATInvocation. Exception("RFTGT 4 File. Adaptor: " + e); } return credential. Endpoint; } public Endpoint. Reference. Type[] fetch. Delegation. Factory. Endpoints( Reliable. File. Transfer. Factory. Port. Type factory. Port) throws GATInvocation. Exception { Get. Multiple. Resource. Properties_Element request = new Get. Multiple. Resource. Properties_Element(); request . set. Resource. Property(new QName[] { RFTConstants. DELEGATION_ENDPOINT_FACTORY }); Get. Multiple. Resource. Properties. Response response; try { response = factory. Port. get. Multiple. Resource. Properties(request); } catch (Remote. Exception e) { e. print. Stack. Trace(); throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: get. Multiple. Resource. Properties, " + e); } SOAPElement[] any = response. get_any(); Endpoint. Reference. Type epr 1 = null; try { epr 1 = (Endpoint. Reference. Type) Object. Deserializer. to. Object(any[0], Endpoint. Reference. Type. class); } catch (Deserialization. Exception e) { throw new GATInvocation. Exception( "RFTGT 4 File. Adaptor: Object. Deserializer, " + e); } Endpoint. Reference. Type[] endpoints = new Endpoint. Reference. Type[] { epr 1 }; return endpoints; } synchronized void set. Status(Overall. Status status) { this. status = status; } public int transfers. Active() { if (status == null) { return 1; } return status. get. Transfers. Active(); } public int transfers. Finished() { if (status == null) { return 0; } return status. get. Transfers. Finished(); } public int transfers. Cancelled() { if (status == null) { return 0; } return status. get. Transfers. Cancelled(); } public int transfers. Failed() { if (status == null) { return 0; } return status. get. Transfers. Failed(); } public int transfers. Pending() { if (status == null) { return 1; } return status. get. Transfers. Pending(); } public int transfers. Restarted() { if (status == null) { return 0; } return status. get. Transfers. Restarted(); } public boolean transfers. Done() { return (transfers. Active() == 0 && transfers. Pending() == 0 && transfers. Restarted() == 0); } public boolean transfers. Succ() { return (transfers. Done() && transfers. Failed() == 0 && transfers. Cancelled() == 0); } /* * private Base. Fault. Type get. Fault. From. RP(RFTFault. Resource. Property. Type fault) { * if (fault == null) { return null; } * * if (fault. get. Rft. Transfer. Fault. Type() != null) { return * fault. get. Rft. Transfer. Fault. Type(); } else if * (fault. get. Delegation. EPRMissing. Fault. Type() != null) { return * fault. get. Delegation. EPRMissing. Fault. Type(); } else if * (fault. get. Rft. Authentication. Fault. Type() != null) { return * fault. get. Rft. Authentication. Fault. Type(); } else if * (fault. get. Rft. Authorization. Fault. Type() != null) { return * fault. get. Rft. Authorization. Fault. Type(); } else if * (fault. get. Rft. Database. Fault. Type() != null) { return * fault. get. Rft. Database. Fault. Type(); } else if * (fault. get. Rft. Repeatedly. Started. Fault. Type() != null) { return * fault. get. Rft. Repeatedly. Started. Fault. Type(); } else if * (fault. get. Transfer. Transient. Fault. Type() != null) { return * fault. get. Transfer. Transient. Fault. Type(); } else { return null; } } */ /* * private Base. Fault. Type deserialize. Fault. RP(SOAPElement any) throws * Exception { return get. Fault. From. RP((RFTFault. Resource. Property. Type) * Object. Deserializer. to. Object(any, RFTFault. Resource. Property. Type. class)); } */ void set. Fault(Base. Fault. Type fault) { this. fault = fault; } }

Ibis as ‘Glue’ ● Use IPL + Smart. Sockets, generally for wide-area communication ● ● Linking up separate ‘activities’ of an application ● Activities: often largely ‘independent’ tasks implemented in any popular language or model (e. g. C/MPI, CUDA, Fortran, Java…) ● Each typically running on a single GPU/node/Cluster/Cloud/… Automatically circumvent connectivity problems ● Example: With Smart. Sockets: No Smart. Sockets:

Ibis as ‘HPC Solution’ ● Use Ibis as replacement for e. g. C++/MPI code ● Benefits: (better) portability ● malleability (open world) ● fault-tolerance ● (run-time) task migration Downside: ● requires recoding ● ● ● Comparable speedups:

MMCA: Situation in 2004/2005 SSH Parallel Horus Client Sockets + SSH Tunneling Server C++/MPI ● ● Code pre-installed at each cluster site Instable / faulty communication Connectivity problems Execution on each cluster ‘by hand’ Parallel Horus Client

Phase 1: Ibis as ‘Master Key’ (2006) Java. GAT + Ibis. Deploy Parallel Horus Client Sockets + SSH Tunneling Server C++/MPI ● ● Code pre-installed at each cluster site Instable / faulty communication Connectivity problems Execution on each cluster ‘by hand’ Parallel Horus Client

Phase 2: Ibis as ‘Glue’ (2006/2007) Java. GAT + Ibis. Deploy Parallel Horus Client IPL + Smart. Sockets Server C++/MPI ● ● Code pre-installed at each cluster site Instable / faulty communication Connectivity problems Execution on each cluster ‘by hand’ Parallel Horus Client

Phase 3: Ibis as ‘HPC Solution’ (2008) Java. GAT + Ibis. Deploy Parallel Jorus Client IPL + Smart. Sockets Server Ibis/Java ● ● Code pre-installed at each cluster site Instable / faulty communication Connectivity problems Execution on each cluster ‘by hand’ Parallel Jorus Client

‘Master Key’ + ‘Glue’ + ‘HPC’ ● Step-wise conversion to 100% Ibis / Java ● ● ● Phase 1: Java. GAT as ‘Master Key’ Phase 2: IPL + Smart. Sockets as ‘Glue’ Phase 3: Ibis as ‘HPC Solution’ After each phase a fully functional, working solution was available! Eventual result: ● ● ● ‘wall-socket computing from a memory stick’ Remember: the ‘Promise of the Grid’? Awards at AAAI 2007 and CCGrid 2008

100% Ibis Implementation (2008++) Seinstra, Maassen, Drost et al. (SCALE 2008 @ CCGrid 2008: First Prize Winner) Bal, Maassen, Drost, Seinstra et al. (IEEE Computer, Aug 2010)

Some current work

Green Clouds ● ● ● NWO Smart Energy Systems project with Univ. of Amsterdam (Cees de Laat) & SARA How to map high-performance applications onto hybrid distributed computing system, taking both performance & energy consumption into account System-level approach to reduce HPC energy consumption

DAS-4: infrastructure for Green IT Uv. A/Multimedia. N (16/36) Dual quad-core Xeon E 5620 Various accelerators (GPUs, multicores, …. ) Scientific Linux Built by Cluster. Vision VU (74) SURFnet 6 ASTRON (23) 10 Gb/s lambdas TU Delft (32) Leiden (16)

Main ideas ● Adapt resources to application needs dynamically, accounting for computational & energy efficiency ● ● Exploit hardware diversity ● ● ● Using Ibis malleability support Graphics Processing Units (GPUs) have much higher FLOPS/Watt for many applications Use optical and photonic networks Build a knowledge base & semantic infrastructure description

Other current Ph. D projects using Ibis ● Distributed reasoning over semantic web data ● ● ● Graph applications (Hi. PG) ● ● ● E. g. for bioinformatics applications http: //www. graph 500. org/ Games & distributed model checking ● ● Web. PIE: Parallel reasoner on Web scale Written in Java, uses Hadoop (Map. Reduce) Deal with large state space Distributed smart phone applications ● Computation & communication offloading to a cloud

Conclusions ● Ibis enables problem solving (avoids system fighting) ● Successfully applied in many domains ● ● ● Astronomy, multimedia analysis, climate modeling, remote sensing, semantic web, medical imaging, … Data intensive, compute intensive, real-time… Open source, download: ● www. cs. vu. nl/ibis/

Conclusions (2) ● Jungle Computing is hard ● High-Performance Jungle Computing even harder ● While research into efficient & green Jungle-aware programming models has only just begun… ● …Ibis provides the basic functionality to efficiently & transparently overcome most Jungle Computing complexities

Omitted

General Requirements ● ● Resource independence Transparent / easy deployment ● ● ● Jungle-aware communication ● ● ● Middleware independence & interoperability Jungle-aware middleware Robust connectivity System-support for malleability and fault-tolerance Globally unique naming Transparent parallelism & application-level fault-tolerance Easy integration with external software (legacy codes) ● MPI, CUDA, C++, Python, Java, Fortran, …

Multimedia Content Analysis (MMCA) ● Need for user-friendly programming tools ● Shield domain-experts from all complexities of parallel, distributed, heterogeneous, and hierarchical computing ● Familiar (sequential) programming model(s) Solution: tool to make parallel & distributed computing transparent to user User - familiar programming - easy execution Jungle Computing Systems

“The Future of Ibis” ● Ibis/Constellation: ● ● Generalized programming framework for ‘all’ Jungle Computing applications Automatically maps any application activity (task) onto any appropriate executor (HW) By way of ‘contexts’, for example: ● Activity's context: “I need a GPU” ● Executor’s context: “I represent a GPU” Note: ● Activities may represent any type of task: ● Also legacy codes, scripts, 3 rd party software, …