Systembergreifende Prozessberwachung mit Event Stream Processing Thomas Dcker
Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz Juni, 2013
Agenda Herausforderung und Trends Übersicht Operational Process Intelligence Übersicht Beispiele Szenario Mehrwert für Sie Weitere Informationen © 2013 SAP AG or an SAP affiliate company. All rights reserved. 2
Event Stream Processing Capture Disseminate & Decide Response & Action Master. Data Alarm Market Data ESP Web Data Adjust Sensor Data © 2013 SAP AG or an SAP affiliate company. All rights reserved. Decision Support 3
Traditional Approaches to Event Stream Processing Database Application Custom Application “Black-box” Application or Dashboard High-Speed Data Queries / Results High-Speed Data Alerts / Actions • DB application (OLTP) — Pull not push (i. e. not event driven) or — Requires triggers – performance degradation / maintenance nightmare — latency in seconds • Custom application (C/C++ / Java) — Specialized, high development / maintenance cost — Slow to change, unresponsive to the business © 2013 SAP AG or an SAP affiliate company. All rights reserved. 4
The SAP Approach – Sybase ESP Input Streams PUBLISH Visual / SQL Authoring Output Streams SUBSCRIBE Market Data Orders, Trades Console Input Reporting Tools ? Dashboards Trading Systems Message Bus Reference Data SAP HANA / Sybase IQ • Combines advantages of customized application with ease of visual data flow diagramming or SQL • Incoming data is processed as it arrives, according to the model rules and operations © 2013 SAP AG or an SAP affiliate company. All rights reserved. 5
CEP Product Convergence Coral 8 v 5. 6. 5 Sybase CEP R 4 Aleri 3. 2 Sept 2011 Sybase ESP 5. 0 Sept 2012 Sybase ESP 5. 1 Dec 2012 Sybase ESP 5. 1 SP 1 Optional (maintenance only) © 2013 SAP AG or an SAP affiliate company. All rights reserved. 6
Advantages of Sybase ESP Approach Analyze Events as they occur • Continuous Insight • Respond Immediately Rapid Application Development • Reduce/eliminate dependence on specialist programming skills for process • Cut implementation/deployment time • Broad out of the box connectivity Non-intrusive Deployment • Event-driven integration with existing systems • Unify existing disparate data models © 2013 SAP AG or an SAP affiliate company. All rights reserved. 7
Event Stream Processing Scenarios Capital Markets React to incoming ticks to determine trading strategies, monitor the success of the strategies by calculating Pn. L in real-time; monitor risk in real -time. © 2013 SAP AG or an SAP affiliate company. All rights reserved. Retail Provide clickstream analysis for online customer assistance / alternatives, use web scraping to react to competitor price changes 8
Event Stream Processing Scenarios Energy Monitor and optimize energy usage using Smart Metering to ensure Qo. S © 2013 SAP AG or an SAP affiliate company. All rights reserved. Utilities / Manufacturing Service Status Monitoring and Process Control using SCADA / OSISoft 9
Event Stream Processing Scenarios Telecoms Monitor events for Qo. S, use DPI to detect fraudulent use © 2013 SAP AG or an SAP affiliate company. All rights reserved. Police / Homeland Security ANPR event correlation for traffic monitoring, security 10
ESP Architektur
ESP Project Architecture Authoring Tools Operational Console Log Store C+C Command & Control (XML-RPC) Studio Security (PAM, SSL, RSA, Kerberos) Publish © 2013 SAP AG or an SAP affiliate company. All rights reserved. Data Streams Adapters Mem Store I/O Gateway Stream Processor + Multi-threaded + 64 bit - Suse 11 / Redhat 6 / Solaris 10 (Sparc / i 86) Win x 64 + Low latency + Optional persistence Security Adapters Data Streams I/O Gateway Data Streams Subscribe Data Streams Log Store 12
Out of the Box Connectivity In Process JMS, MQ, TIBCO ATOM (RSS) TCP Sockets Databases via JDBC / ODBC • Files • MS Excel RT • SMTP (out) • • © 2013 SAP AG or an SAP affiliate company. All rights reserved. External • Market Data (Reuters, Bloomberg, IDC, ACTIV, NYSE Tech) • FIX • Open. Adaptor Java Framework • Replication Server • IQ / HANA Pub. Sub SDK • C/C++ • Java • . NET 13
ESP Cluster Architecture I Manager Nodes Cache Manager Controller Project Server © 2013 SAP AG or an SAP affiliate company. All rights reserved. Controller Project Server 14
ESP Cluster Architecture II • Manager Node − Deploys project(s) to server(s) − Maintains heartbeat with project(s) to detect failure − Manages failover − In multi node, managers are run in a cache so can failover to manage other controller nodes • Controller Node − Basically acts as a launch daemon for Manager to start project(s) in a Container © 2013 SAP AG or an SAP affiliate company. All rights reserved. 15
ESP Cluster Architecture III • Single node cluster – − Single manager/controller node runs multiple projects on a single machine − Detects when a project fails and tries to restart it − This is the default for project development • Multi node cluster − Multiple equi-peer managers and controllers run multiple projects on multiple machines − Used to scale out projects across multiple machines − Allows for server failure – promotes failover recovery and data redundancy − Allows for Highly Available “active: active” deployment of projects within a Data Centre © 2013 SAP AG or an SAP affiliate company. All rights reserved. 16
ESP Installed Clusters • Three clusters created by install script: - • Studio Cluster (single node) on 9786 : – − Used for Studio Development − Can run multiple projects − Located in $ESP_HOME/studio/clustercfg • User-defined Cluster (single node) on 19011 − Located in $ESP_HOME/cluster/nodes/node 1 • Example multi-node Cluster on 19011 - 19014 − Located in $ESP_HOME/cluster/examples © 2013 SAP AG or an SAP affiliate company. All rights reserved. 17
ESP Key Concepts
Authoring Options – Eclipse Plugin Visual Dataflow Authoring CCL Authoring • Analyst-level skills (Excel, VBA) • Rapid programming • Easy to understand complex models • Easy to use language (CCL – derived from SQL) • No need to learn language syntax • Modular, project based approach © 2013 SAP AG or an SAP affiliate company. All rights reserved. 19
Visual Dataflow Authoring © 2013 SAP AG or an SAP affiliate company. All rights reserved. 20
CCL Authoring © 2013 SAP AG or an SAP affiliate company. All rights reserved. 21
Procedural Logic - SPLASH © 2013 SAP AG or an SAP affiliate company. All rights reserved. 22
ESP Modularity IMPORT 'Create. Complete. Channel. ccl'; -----CREATE MODULE Create. Complete. Channel IN Message. In, Cache. In OUT Message. Out BEGIN // Module Code END; LOAD MODULE Create. Complete. Channel AS Channel_A IN Message. In = Message. A, Cache. In = Cache. A OUT Message. Out = Message. Out. A; LOAD MODULE Create. Complete. Channel AS Channel_B IN Message. In = Message. B, Cache. In = Cache. B OUT Message. Out = Message. Out. B; © 2013 SAP AG or an SAP affiliate company. All rights reserved. 23
Project Code, Compiled Code & Project Resources • Project Source Code. CCL (Continuous Computation Language) − Used for schema, stream, window, method definitions compiled into • Project Executable. CCX (Continuous Computation e. Xecutable) • Project Resource File. CCR (Continuous Computation Resource) − An xml file used for in process adaptor, cluster, bindings, failover, HA definitions – allows for easy migration between DEV, TEST and PROD © 2013 SAP AG or an SAP affiliate company. All rights reserved. 24
Constructing ESP Projects • Build model • Connect to data feeds using publish adaptor • Connect to output devices via subscribe adaptor Tick data feed Trades ESP Project Container DB Replicated Data Console Positions © 2013 SAP AG or an SAP affiliate company. All rights reserved. 25
Stateful Events • Events are retained as records in memory with a primary key so that further events may modify the state of the record - either update or delete it. • A stateful event must have PK attribute(s) set. • A stateful event MUST have an associated operation code (opcode), one of INSERT, UPDATE, UPSERT, DELETE. If no opcode is provided, ESP defaults it to UPSERT. • When an event occurs it applies the opcode through the data flow model depending on the opcode. An insert in a source stream may become an update to an aggregate or join stream. • The full set of unique (insert) events to be retained MUST fit in memory. • Records in memory can be removed either by setting a retention window, or by publishing a DELETE event for that key. • In ESP, stateful events are implemented using a WINDOW © 2013 SAP AG or an SAP affiliate company. All rights reserved. 26
Stateless Events • An event is retained only while it is being processed. When an event moves from one stream to another it is removed from the first stream so it cannot be updated. • All events are treated as insert. • A stateless event ignores any PK attribute(s) and always outputs an insert • In ESP stateless events are implemented with a STREAM © 2013 SAP AG or an SAP affiliate company. All rights reserved. 27
ESP Elements ESP implements stateful and stateless events using 3 element types • WINDOW : used for stateful events. A WINDOW maintains state for a user defined period • STREAM : used for stateless events. A STREAM does not maintain or react to state • DELTA : used to process stateful events, but without maintaining state • All can be INPUT, LOCAL, OUTPUT • INPUT subscribe to input adaptors i. e. events are published to an INPUT. They are the entry points for events to enter an ESP project. • LOCAL perform processing logic. They can NOT be published or subscribed to • OUTPUT are derived elements which perform processing logic which CAN be subscribed to © 2013 SAP AG or an SAP affiliate company. All rights reserved. 28
Windows (stateful) • WINDOWS must have a PK set • Can have an optional retention period set • Maintain state in memory or persisted to disk. i. e. State can be recovered in the event of system failure • Can publish to STREAM, WINDOW or DELTA • Only a WINDOW can be used for aggregation • At lease one side of a join MUST be a WINDOW (unless using SPLASH). The side which maintains state depends on the type of the join. • Use WINDOW when developing models as they help debugging © 2013 SAP AG or an SAP affiliate company. All rights reserved. 29
Streams (stateless) • Stream • Cannot be persisted so any data is lost in the event of system failure • Can NOT publish to DELTA stream • Can publish to a WINDOW, but WINDOW must aggregate the input • Delta • Can only be LOCAL, OUTPUT • A stateless element (a STREAM) that understands and processes state (a WINDOW) • Can only use a WINDOW as a source • Must have a PK set • As it does not store events it can NOT be used for aggregation. • Cannot be persisted so any data is lost in the event of system failure © 2013 SAP AG or an SAP affiliate company. All rights reserved. 30
Example: Simple Position Maintenance Trades Stream Symbol* Time* Price Shares Aggregate Average. Trades Window Symbol* Last Price Weighted Avg Price Last Time Positions By Book Stream Position Window Book. Id* Symbol* Current Position Average Position Join Total Book Window Aggregate Book. Id* Current Position Average Position Book. Id* Symbol* Shares Held © 2013 SAP AG or an SAP affiliate company. All rights reserved. 31
Example: Simple Position Maintenance Trades Stream Symbol* Time* Price Shares MSFT 12: 05: 23 32. 45 200 IBM 12: 05: 24 57. 87 5000 CSCO 12: 06: 01 20. 04 100 MSFT 12: 06: 02 36. 55 900 Aggregate MSFT 32. 45 36. 55 33. 01 34. 00 12: 05: 23 12: 06: 02 Book 1 MSFT 97350 109650 99030 102000 Average. Prices Window IBM 57. 87 56. 90 12: 05: 24 Book 1 IBM 40509 39830 Symbol* Last Price Weighted Avg Price Last Time CSCO 20. 04 17. 00 12: 06: 01 Book 2 MSFT 32450 36550 3301034000 Individual Positions Stream Book. Id* Symbol* Current Position Average Position Join Position Window Book. Id* Symbol* Shares Held Book 1 MSFT 3000 Book 1 IBM 700 Book 2 MSFT 1000 © 2013 SAP AG or an SAP affiliate company. All rights reserved. Book Positions Window Aggregate Book. Id* Current Position Average Position Book 1 150159 139539 141830 138860 Book 2 36550 32450 34000 33010 32
ESP Solution Architektur
Event Processing Solution Architecture • Events don’t happen in isolation • Most Event Stream Scenarios need access to historic data • Pattern Detection • Risk Calculation • Optimisation algorithm • To ensure sub millisecond response times use in memory e. g. HANA • Many Event Stream Scenarios need access to master data • Data written directly to OLTP system • Updates to reference, semi-static data • For sub second updates without polling use log trawler e. g. Replication Server © 2013 SAP AG or an SAP affiliate company. All rights reserved. 34
Real-time ANPR Alerting Framework OCR Radars SQL Event Stream Processor Output Adapter Input Adapter Monitoring Business Inteligence CEP Engine SQL HANA In-database Algorithms - or y In Historic data em Analytical data M BO Predictive Analysis Predictive models (Ad-hoc analysis) Reference data SQL Business. Objects Data. Service External Databases © 2013 SAP AG or an SAP affiliate company. All rights reserved. 35
Real-time Retailing Framework Validate Event Detect Event HANA ESP GUI Is competitor price<95% of our price? Manage Exception Straight Through Processing Run Optimization Algorithm Check Results Output Is optimum price<95% of current our price? Phase 1: Phase 2: Phase 3: Phase 4: Event Detection & Validation Processing Validation & Exception Output © 2013 SAP AG or an SAP affiliate company. All rights reserved. 36
Integration with Hana I • Generic input / output ODBC adapter • Data can enter via a source stream © 2013 SAP AG or an SAP affiliate company. All rights reserved. 37
Integration with Hana II • Can be called from a Flex. Stream using SPLASH for event driven query: - ON TRADE { if (TRADE. Price <> optimal. Price) { [ string s; | ] rec; vector (typeof(rec) v : = new vector(typeof(rec)); get. Data(v, 'Hana. DSN', 'CALL DPTL. sp_price_opt (? , ? , ? )', TRADE. Id, TRADE. Datetime, TRADE. Sector, TRADE. Price); if (not (isnull(v))) { newrec : = v[0]; } // rest of method } } © 2013 SAP AG or an SAP affiliate company. All rights reserved. 38
Real-time Risk Framework Trading Systems Continual Mark t o Market Transformation of granular data • • • CEP Real-time Aggregation of Risk and P&L Cleansing Normalisation Enrichment Bus Trading Market Data Systems Analytics: • Heatmaps • Drill-downs • Reports Replication Server Trading Risk Systems Sensitivities ASE • © 2013 SAP AG or an SAP affiliate company. All rights reserved. Reference Data HANA • • Scenario Shifts VAR calculations Showing: • Concentration limits • Clustering • Outliers • Etc… By: • Instrument • Industry • Country • Debt Rating • Counterparty • Va. R Time Horizon • Etc… 39
Oil & Gas - Energy Efficiency Reporting Framework ECC (IS-U) Panopticon RA#1 Oracle RS (Replication Server) OKO HANA ESP RS SOI 2 BO 4. 0 (Event Stream Processor) HANA ODBC Non SAP (top level MES) Oracle RA#2 SOI 2 server SCADA © 2013 SAP AG or an SAP affiliate company. All rights reserved. Replication flow Event flow 40
Besten Dank Kontaktinformation: Thomas Dücker SAP Schweiz thomas. duecker@sap. com © 2013 SAP AG or an SAP affiliate company. All rights reserved.
- Slides: 41