INNOV10 Progress Event Engine Technical Overview Prashant Thumma

INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer

Agenda Slide n n n Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration 2 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Event Stream Processing (CEP) What is ESP? n ESP is about Operational Business Data – Capture – Query – Analysis n Characteristics – – n High Volume Streaming Data Time Critical Analysis Consistent and Reproducible Results Highly Available Simple Structured Data – Data is not Relational or complex Object model 3 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Event Stream Processing (ESP) ESP Platform Requirements n Event Processing Language – Data Driven – Flexible – Complete n Event Persistence Engine – High Performance n Analytical Capabilities – Temporal Data Analysis – Inbound and Outbound Event Processors n n High Availability Modeling, Deployment and Development Tools 4 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Event Stream Processing (CEP) ESP Applications n n n Financial Markets – Program Trading – Testing Trading Strategies RFID – Inventory Logistics – Asset Tracking – Theft Detection Manufacturing – Capture and analysis of data from PLCs Telecommunications – Cell Phone Call Tracking – Multimedia Services Billing – Network Management Scientific – Sensor Data 5 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Agenda Slide n n n Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration 6 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress ESP Products Event Cache for JMS Apama Adapter Java API -<!– simple. xml --> - <event> -<!– simple. xml --> - +<event> <info 1> + <info 2> + <info 1> + <info 2> </event> XML Configuration Files Event Server C++ API -<!– simple. xml --> - <event> -<!– simple. xml --> - +<event> <info 1> + <info 2> + <info 1> + <info 2> </event> XML Configuration Files Progress® Event Engine™ (PSEE) 7 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Agenda Slide n n n Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration 8 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine What is it for? n It’s all about Event Data – Collection, storage, management and query n What is Event Data? – Streaming, constantly flowing information n Stock Trades, RFID, IP Packets, Mfg PLC’s – Categorized n Stock tick symbol, RFID tag ID, IP address – Typically Time Ordered n Trade time, RFID scan time 9 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Data Flow Event Data Feed client 1. Collect store Convert external events to PSEE events Service 3. Query Event Cache sort/merge Persistent Storage disk query disk 2. Organize Structures data for fast retrieval 10 © 2005 Progress Software Corporation disk INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Why PSEE? n High Performance, Scalable – Collection Engine n n n Store tens of thousands events per second 20 TB+ database size Index data is ~1% of database size – Query Engine n n n High Performance Pipelined Architecture Extensible Functionality – Simultaneous Query and Collection n Writes do not block reads! 11 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Why PSEE? n Transparent Data Management – Event Persistence n Requires no coding to persist events – Transactional n n Consistent Data Hidden and Managed by PSEE – Tunable Latency n “Real-time” defined by you! 12 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Why PSEE? n n Simple API Supports Multiple Languages – Java and C++ language support n Multiple Platform Support – Window, Solaris, Linux – Heterogeneous deployment 13 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Components n Functor: A unit of work in PSEE – 3 Main types (Source, Process, Sink) – Built in library of Functors – Customer coded n Pipeline: A sequence of Functors – Collect, process, transform event streams n Task Manager – Manages pipeline execution n Events: Input and Output of Pipelines 14 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Event Definition n PSEE Event – XML schema – Supports All C++ Primitive Types <Event. Definitions bundle. Name=“TABundle” …> <Event name=“Trade” type=“Timed. Categorized. Event”> <Member name=“price” type=“Float 32”/> <Member name=“volume” type=“UInt 32”/> </Event. Definitions> 15 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Functor n n Implements Event Stream Processing Logic Interface For Accessing Events class My. Collector. Functor : public Functor { public: void initialize(Functor. Context& ctx); void handle(Functor. Context& ctx); void finalize(Functor. Context& ctx); private: // functor data variables } 16 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Collector Pipeline Real-Time Data Feed Event My. Collector Functor Store. Realtime Stream Functor Series (PSEE Database) Event-Data File for Bulk Loading 17 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Query Pipeline Event Retrieve Functor Event Filter Stream Functor Event Sort Stream Functor My. XMLwriter Stream Functor Series (PSEE Database) 18 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Predefined Functors n Collector Functors – Store. Realtime – Store. Bulkload 19 © 2005 Progress Software Corporation n Query Functors – Retrieve – Filter – Sort – Count – Group Count INNOV-10 Progress Event Engine Technical Overview

Progress Event Engine Scalable Deployment PSEE OSES Query Adapter Collection Adapter 1 OSES Raw Series PSEE Collection Adapter 2 Optimized db’s for query Series database VMMA & CFA Scalability 20 © 2005 Progress Software Corporation Query Adapter Messaging Scalability INNOV-10 Progress Event Engine Technical Overview

Agenda Slide n n n Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration 21 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

PSEE Performance Metrics n Event storage (events/sec) – Real-time collector – pipeline using Store. Realtime functor – Bulk load collector – pipeline using Store. Bulkload functor n Event retrieval (events/sec) – Threaded execution of query pipeline n n Random query Query on several (randomly chosen) categories 22 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

PSEE Performance Testing Environment n Database Characteristics – 160 Million Events – 20 Byte Event – 10000 Categories n System Configuration – – Sun Fire-V 440 Solaris 8 4 CPUs (1 GHz) 8 GB Memory 23 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

PSEE Performance Testing Results n Real-time collector – 94, 062 events / second n Bulk load collector – 191, 152 events / second n Query – 213 Queries / second – 10000 events / query – 2, 134, 560 events / sec 24 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Agenda Slide n n n Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration 25 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Under Development n n D I S C L A I M E R This talk includes information about potential future products and/or product enhancements. What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here. D I S C 26 © 2005 Progress Software Corporation L A I M E R INNOV-10 Progress Event Engine Technical Overview

PSEE Integration with Apama n n Provides Event Persistence for Apama Initial Implementation for Customer – Tick Database for Back-Testing Trading Strategies – Query Interface Using Apama Event Language n Future – Extend Apama Event Language to support Persistence – Seamless Configuration via Apama Management Interfaces 27 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

PSEE Integration with Sonic n n PSEE on the bus using Event Cache Features – Native JMS Message Based Query – Results are Java Objects – Batched for Optimal Performance 28 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

PSEE Integration with 4 GL n n PSEE based RFID Accelerator provides 4 GL Interface Web Services based Implementation RFID Accelerator as a reference implementation Further 4 GL Integration 29 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

In Summary n n n Exponential growth of event data requires ESP solutions PSEE + Apama is an ESP solution Based on proven technologies Simple! Infrastructure problems are handled by PSEE It can be integrated into existing business systems 30 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Questions? 31 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

Thank you for your time! 32 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview

33 © 2005 Progress Software Corporation INNOV-10 Progress Event Engine Technical Overview