Building a KEPLER Extension using Ptolemy II KEPLER
Building a KEPLER Extension using Ptolemy II KEPLER Collaboration Staff Presented by: Ilkay Altintas and Efrat Jeager @ San Diego Supercomputer Center, UCSD Building a KEPLER Extension using Ptolemy II
Outline n n Overview of Ptolemy II and Kepler collaboration Ptolemy II n n n n Ptolemy II Architecture Download, Make and Install The Vergil user interface and application Directors and Actors Ports and Parameters Manager Kepler n n n Overview of pre-built Kepler actors and demos Ingredients to build a Kepler extension Building you first actor n 2/20/2004 Hello. World revisited… Building a KEPLER Extension Using Ptolemy II 2
The KEPLER System for Scientific Workflows … A framework for design, execution and deployment of scientific workflows n Caters specifically to the domain scientist n Builds on Ptolemy II (next slide. . . : -) n Application pull from various projects n http: //kepler. ecoinformatics. org 2/20/2004 Building a KEPLER Extension Using Ptolemy II 3
… based on Ptolemy II n n A set of Java packages for heterogeneous, concurrent modeling, design and execution. Strengths include: n Precisely defined models of computation and component interaction n n An intuitive GUI that lets rapid workflow composition A modular, reusable and extendable object-oriented environment An XML based workflow definition – Mo. ML n n 2/20/2004 e. g. Process Networks (PN) – data-flow oriented Workflows defined in Ptolemy II Mo. ML XML schema Easily exchangeable Building a KEPLER Extension Using Ptolemy II 4
Actor-Oriented Design Reference: Ptolemy. II Design Document-1, Chapter 1 2/20/2004 Building a KEPLER Extension Using Ptolemy II 5
Focus on Actor-Oriented Design n Object orientation: What flows through an object is sequential control class name data methods call n return Actor orientation: What flows through an object is streams of data actor name data (state) Input data 2/20/2004 parameters ports Output data Building a KEPLER Extension Using Ptolemy II Adapted from the *. ppt slides by Edward A. Lee (See References) 6
Layered Software Architecture Ptolemy II packages have carefully constructed dependencies and interfaces CT Data Kernel PN M Building a KEPLER Extension Using Ptolemy II Math FS SD F Actor DE 2/20/2004 Graph CSP Adapted from the *. ppt slides by Edward A. Lee (See References) 7
Ptolemy II Architecture n Core packages n n n Domain polymorphic actors Domain packages n 2/20/2004 Mo. ML and visual interface support Library packages n n The data model(--abstract syntax) of the models Abstract semantics User Interface (UI) packages n n (actor, data, kernel, math, util) Implementations of different models of computation Building a KEPLER Extension Using Ptolemy II 8
Overview of the Key Classes 2/20/2004 Building a KEPLER Extension Using Ptolemy II 9
Installing Ptolemy II http: //ptolemy. eecs. berkeley. edu/ptolemy. II/ n User version n n Version Ptolemy II 3. 0. 2 – Web Start Installer Source download n n Precompiled (…downloaded before…) Source-only code – 4 steps to make n n 2/20/2004 cd “PTII” rm –f config. *. /configure make fast install Building a KEPLER Extension Using Ptolemy II 10
Abstract Syntax of PTII Models • Hierarchical Entities, Ports, Connections and Attributes Abstract syntax choices: • • Abstract syntax defines the structure of a model, but says little about what it means. 2/20/2004 Building a KEPLER Extension Using Ptolemy II • Hierarchy is tree structured (like XML). A relation mediates connections. Ports can link multiple relations and relations can link multiple ports. Ports mediate connections across levels of the hierarchy (no statecharts-style levelcrossing links) … Adapted from the *. ppt slides by Edward A. Lee (See References) 11
Models of Computation n Semantic interpretations of the abstract syntax Different models <=> Different semantics <=> Different execution One Class of Semantic Models: Producer / Consumer process { … write(); … } process { … channel read(); port … } receiver Are actors active? passive? reactive? l Are communications timed? synchronized? buffered? l 2/20/2004 Building a KEPLER Extension Using Ptolemy II Models of Computation: • continuous-time • dataflow • rendezvous • discrete events • synchronous • time-driven • publish/subscribe • … Adapted from the *. ppt slides by Edward A. Lee (See References) 12
Director n Governs the execution of a composite entity. n n Scheduling, dispatching threads, generate code, etc. A composite entity is called opaque if it doesn’t have a local director. n 2/20/2004 An opaque composite entity inherits the director of its container as its executive director. Building a KEPLER Extension Using Ptolemy II 13
Vergil User Interface n n n n The graph editor Actor library Director library Utilities Ports Parameters Quick tour 2/20/2004 Building a KEPLER Extension Using Ptolemy II 14
Using Vergil n n Executing a pre-built model from the graph editor Building a model n n n Choosing actors Making connections Choosing a director Building composite actors Saving and running the model Start Exercise-0 2/20/2004 Building a KEPLER Extension Using Ptolemy II 15
Mo. ML n n Modeling Markup Language A primary persistent XML file format for Ptolemy II. n n 2/20/2004 ptolemy filename. xml ptexecute filename. xml vergil filename. xml moml configuration. xml filename. xml Building a KEPLER Extension Using Ptolemy II 16
An Example Mo. ML File <? xml version="1. 0" standalone="no"? > <!DOCTYPE entity PUBLIC "-//UC Berkeley//DTD Mo. ML 1//EN" "http: //ptolemy. eecs. berkeley. edu/xml/dtd/Mo. ML_1. dtd"> <entity name="test" class="ptolemy. actor. Typed. Composite. Actor"> <property name="director" class="ptolemy. domains. sdf. kernel. SDFDirector"/> <entity name="ramp" class="ptolemy. actor. lib. Ramp"/> <entity name="plot" class="ptolemy. actor. lib. gui. Sequence. Plotter"/> <relation name="r" class="ptolemy. actor. Typed. IORelation"/> <link port="ramp. output" relation="r"/> <link port="plot. input" relation="r"/> </entity> 2/20/2004 Building a KEPLER Extension Using Ptolemy II Top-level entity SDF domain Reference: Ptolemy. II Design Document-1, Chapter 6 17
Type System 2/20/2004 Building a KEPLER Extension Using Ptolemy II 18
Building Kepler Extensions n n FOCUS: How to build actors? Exercises are under: n n http: //www. sdsc. edu/~altintas/Kepler. Tutorial/Kepler. Hands. On. txt Please complete the preparation for the programming exercises… 10 minute break… 2/20/2004 Building a KEPLER Extension Using Ptolemy II 19
Ingredients We have: n n n Java 1. 4. 2 Cygwin for Developers (for Windows users) Ant 1. 5 or higher Downloaded: n Compiled Ptolemy II source code http: //www. sdsc. edu/~altintas/Kepler. Tutorial/pt. II 3. 0. 2. tar. gz n Kepler extensions (a version from CVS) http: //www. sdsc. edu/~altintas/Kepler. Tutorial/kepler. zip Completed the preparation for the hands-on programming exercises. n Coding-style guidelines are at: http: //users. sdsc. edu/~altintas/coding. Style. pdf Please use them! 2/20/2004 Building a KEPLER Extension Using Ptolemy II 20
Adding Actors n n Domain and/or data polymorphic actors Use object-oriented inheritance to avoid code duplication n n To use the actors in Vergil n n Add them to one of the actor libraries Most libraries are under $PTII/ptolemy/actor/lib Libraries are XML files The basic structure of an actor: n 2/20/2004 3 base classes: Source, Sink, Transformer See http: //www. sdsc. edu/~altintas/Kepler. Tutorial/Actor. Structure. txt Building a KEPLER Extension Using Ptolemy II 21
Actor Interfaces: input ports Ports & Parameters parameters: a 1 = value a 2 = value Example: output port p 1 p 3 p 2 input/output port 2/20/2004 port Building a KEPLER Extension Using Ptolemy II 22
Anatomy of an Actor: Ports n n n Used for “message transport”, can be an input, an output, or both. Key class: IOPort (Can be connected to other IOPort instances via IORelations. ) Use Typed. IOPort in order to benefit from the type system! (Domain specific: DEIOPort) Receiver and Sender interfaces depending on the usage of the port. Public members of the actors!!! 2/20/2004 Building a KEPLER Extension Using Ptolemy II 23
Ports: Introduction to the API public Typed. IOPort port. Name; //Definition //Create the port. Name = new Typed. IOPort(this, “port. Name”, true, false); port. Name. set. Multiport(true); //Can support more than one link int width = port. Name. get. Width(); //0 or 1 if single port //Reading and Writing port. Name. send(channel. Number, token); Token token = port. Name. get(channelnumber); //Setting the type of the port. Name. set. Type. Equals(Base. Type…a type in the type system…); port. Name. set. Type. At. Least(…must be a port or parameter…); 2/20/2004 Building a KEPLER Extension Using Ptolemy II 24
Anatomy of an Actor: Parameters n n Public members of the actors! Similar API with ports… public Parameter parameter. Name; //Definition //Creation and setting the initial value: 2 -ways parameter. Name = new Parameter (this, “parameter. Name”, new Token(…token value…)); OR parameter. Name = new Parameter (this, “parameter. Name”); parameter. Name. set. Expression(…token. Value…); 2/20/2004 Building a KEPLER Extension Using Ptolemy II 25
Anatomy of an Actor: Constructors n The major task: n n super(container, name); n n To create and configure ports and parameters Carries the Name. Duplication and Illegal. Action exceptions from the super class. The icon for the actor can be set here. 2/20/2004 Building a KEPLER Extension Using Ptolemy II 26
Exercise-1 n Create an actor with n n n An output port A parameter Introduction to the fire method. Adding your actors into the actor library Using your actors in a workflow Break for 10 minutes when finished… 2/20/2004 Building a KEPLER Extension Using Ptolemy II 27
Execution of an actor initialize() prefire Iterations wrap. Up() 2/20/2004 fire() postfire() Building a KEPLER Extension Using Ptolemy II 28
Action Methods n initialize(): initialize() Initialize the state variables of an actor. n prefire(): prefire Returns a boolean which indicates if the actor wants to fire. n n Can also be used to perform an operation that will happen exactly once per iteration. fire(): fire The main point of execution. n 2/20/2004 For reading inputs, producing outputs, read the current parameter values. Building a KEPLER Extension Using Ptolemy II 29
Action Methods (cont. ) n postfire(): postfire Has two tasks: n n n 2/20/2004 Updating persistent state Determining whether the execution of an actor is complete. wrap. Up(): wrap. Up For displaying final results. Building a KEPLER Extension Using Ptolemy II 30
Things to remember when implementing a fire() method n n Use the methods of the Token class for arithmetic whenever possible (to get data polymorphism) If data-polymorphism is not necessary, set the type to a base type then cast the token to that type. Cannot assume that there is be data available at all the input ports (for domain-polymorphism) Do not update the persistent state in fire() (use postfire()) 2/20/2004 Building a KEPLER Extension Using Ptolemy II 31
Implementing Polymorphism n n n Class: Port. Parameter. Function A Port. Parameter. Function object will be returned as a function of two objects. Set the type of the output equal to the type of this object. Type system will compute the type of the Port. Parameter. Function object and use it as the type of the output when necessary. We’ll see this in the example 2! 2/20/2004 Building a KEPLER Extension Using Ptolemy II 32
Exercise-2 n n Getting data through an input port. Understanding the action methods. Designing a polymorphic actor. Using inner classes Break for 10 minutes when finished… 2/20/2004 Building a KEPLER Extension Using Ptolemy II 33
The manager n n Controls the overall execution of a model. Interacts only with the “top-level composite actor” start. Run() -> run() -> execute() Execution. Listener interface provides the manager with info on the events generated during execution. Hand-outs. . . show Ptolemy II handles mixing models of computations hierarchically. --Using opaque composite actors! 2/20/2004 Building a KEPLER Extension Using Ptolemy II 34
Exceptions n n n A uniform mechanism for reporting errors Base class: Kernel. Exception chaining re-implemented since Java versions < 1. 4 doesn’t support it. n n n The detail message includes the detail message from the cause argument. A protected _set. Cause() method is implemented, but not the public init. Cause() method that JDK 1. 4 has. Non-severe exceptions: Illegal. Action. Exception, Name. Duplication. Exception, Name. Duplication. Exception. n Severe-exceptions: Kernel. Runtime. Exception, Invalid. State. Exception, Internal. Error. Exception. 2/20/2004 Building a KEPLER Extension Using Ptolemy II 35
Exercises-3&4 n n n Stopping a model using an actor Utilizing the manager functions Handling exceptions Using the postfire method Creating a customized icon for an actor Using an existing code/application in Kepler. Break for 10 minutes when finished… 2/20/2004 Building a KEPLER Extension Using Ptolemy II 36
Existing Kepler Actors n n n n Actor prototyping tool Generic Web and Grid Service RDBMS Connection and Querying Generic User Interface and Transformation Biological Service and Data Access Rock Classification EML Data Ingestion GARP Native Species Pipeline (Using JNI to utilize C++ code) 2/20/2004 Building a KEPLER Extension Using Ptolemy II 37
Actor Prototyping Tool n n n Allows “place-holder” actors to be defined on the fly by non-programmers during workflow creation Scientists can thereby create workflows without programming knowledge Workflows created with these actors can be executed once their functionality is implemented by a programmer 2/20/2004 Building a KEPLER Extension Using Ptolemy II 38
Distributed SWFs in KEPLER n Web and Grid Service plug-ins n n n WS Harvester n n n 2/20/2004 WSDL, GWSDL Proxy. Init, Globus. Grid. Job, Grid. FTP, Data. Access. Wizard Imports all the operations of a specific WS (or of all the WSs in a UDDI repository) as Kepler actors WS-deployment interface (…ongoing work…) XSLT and XQuery transformers to link non-fitting services together Building a KEPLER Extension Using Ptolemy II 39
A Generic Web Service Actor Given a WSDL and the name of an operation of a web service, dynamically customizes itself to implement and execute that method. Configure - select service operation n 2/20/2004 Building a KEPLER Extension Using Ptolemy II 40
Set Parameters and Commit Set parameters and commit 2/20/2004 Building a KEPLER Extension Using Ptolemy II 41
WS Actor after Instantiation 2/20/2004 Building a KEPLER Extension Using Ptolemy II 42
Web Service Harvester • Imports the web services in a repository into the actor library. • Has the capability to search for web services based on a keyword. 2/20/2004 Building a KEPLER Extension Using Ptolemy II 43
Composing 3 rd-Party WSs Output of previous web service User interaction & Transformations 2/20/2004 Building a KEPLER Extension Using Ptolemy II Input of next web service 44
Current Grid Actors 2/20/2004 Building a KEPLER Extension Using Ptolemy II 45
Providing RDBMS Access n Database connection. n n Database Querying. n n Opening a database connection and passing it to all actors accessing this database. A generic actor that queries a database and provides its result. DBConnection type and DBConnection. Token. n 2/20/2004 A new IOPort type and a token to distinguish a database connection from any general type. Building a KEPLER Extension Using Ptolemy II 46
Database Connection n Open. DBConnection actor: n n 2/20/2004 Input: database connection information Output: A DBConnection. Token, a reference to a database connection instance, through a DBConnection output port. Building a KEPLER Extension Using Ptolemy II 47
Querying a Database n Database. Query actor: n n n 2/20/2004 Input: A query string (SQL) and a database connection reference. Parameters: output type – XML, Record or String. Process: Execute query. Produce results according to parameters. Building a KEPLER Extension Using Ptolemy II 48
Using Bio-Services Actors: PIW 2/20/2004 Building a KEPLER Extension Using Ptolemy II 49
Sub-workflow: Gene Sequence Processing Data transformations between the tasks 2/20/2004 Building a KEPLER Extension Using Ptolemy II 50
Sub-workflow: 2/20/2004 Transfac Detail Building a KEPLER Extension Using Ptolemy II 51
Classifying Igneous Rocks 2/20/2004 Building a KEPLER Extension Using Ptolemy II 52
Actors for Modeling The Classifier Finer granularity Extracted from the mineral composition and this level’s diagram coordinates. Diagrams information and transitions between them. Classifier: Locates the point’s region. SVG to polygons. Displays the point in the diagram for this level. 2/20/2004 Building a KEPLER Extension Using Ptolemy II 53
EML Data Ingestion Actor n Ingests any data format described by EML metadata n Converts raw data to Ptolemy format n Data can then be operated on with other actors 2/20/2004 Building a KEPLER Extension Using Ptolemy II 54
Using EML ingestion actor 2/20/2004 Building a KEPLER Extension Using Ptolemy II 55
Third-Party Local Applications 2/20/2004 Building a KEPLER Extension Using Ptolemy II 56
What did we learn? n n n Basic concepts of Ptolemy II Creating actors within the Ptolemy II framework The existing Kepler extensions What we didn’t learn: n Extending the type system n n Extending the user interface n n Need to define a class for the new type and add it to the type lattice Effigy, Tableaux (Similar to Model-View-Controller architecture) Creating our own director n Developing a director is harder! n 2/20/2004 Need to inherit the Director class, and implement the semantics of how the actors will behave. (Semantics==Model of computation) Building a KEPLER Extension Using Ptolemy II 57
Ptolemy II Design Document Volume 1: User-Oriented 2/20/2004 Volume 2: Developer-Oriented Building a KEPLER Extension Using Ptolemy II Volume 3: Researcher-Oriented Adapted from the *. ppt slides by Edward A. Lee (See References) 58
Becoming a Kepler member? n http: //kepler. ecoinformatics. org/members. html Email me: altintas@sdsc. edu to get a read-only account. Will upgrade to become a contributing member depending in time. 2/20/2004 Building a KEPLER Extension Using Ptolemy II 59
References n n n n Ptolemy II Design Documentation and Source Code http: //ptolemy. eecs. berkeley. edu/papers/03/pt. IIDesign. Intro/ Kepler website: http: //kepler. ecoinformatics. org Overview of Ptolemy II (Powerpoint presentation), Edward A. Lee, October 9, 2003, presented in course EE 249 - Design of Embedded Systems: Models, Validation and Synthesis. An Overview of the Ptolemy Project a nd Actor-Oriented Design (powerpoint presentation), Edward A. Lee, February 4, 2004, OMG Technical Meeting, Anaheim, CA. Presentations from the Ptolemy website: http: //ptolemy. eecs. berkeley. edu/ptolemy. II/ Kepler documentation, javadocs and this tutorial will be made available at the website soon. 2/20/2004 60
- Slides: 60