Design Patterns For Real Time Streaming Analytics Sheetal

Design Patterns For Real Time Streaming Analytics Sheetal Dolas Principal Architect, Hortonworks Page 1 Feb 2015 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Who am I ? • Principal Architect @ Hortonworks • Most of the career has been in field, solving real life business problems • Last 5+ years in Big Data including Hadoop, Storm etc. • Co-developed Cisco Open. SOC ( http: //opensoc. github. io ) sheetal@hortonworks. com @sheetal_dolas Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Agenda • Streaming Architectural Patterns - Overview • Design Patterns o What o Why o Illustrations • QA Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Streaming Architectural Patterns Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Real Time Streaming Architecture Source Systems Sources Syslog Machine Data External Streams Other Page 5 Data Collection Messaging System Flume / Custom Kafka Real Time Processing Storm Storage Access Historic Analytic Tools Hive / HDFS Search Agent A Agent B Topic A Topic B Topology A Topology B Elastic Search / Solr Low Latency No. Sql Agent N © Hortonworks Inc. 2011 – 2014. All Rights Reserved Topic N Topology N HBase R / Python BI Tools Web Services REST API Web Apps Alerting Systems

Lambda Architecture New Data Stream Batch Layer Serving Layer All Data Batch View Pre-compute Views Batch View Query Speed Layer Stream Processing Page 6 Data Access © Hortonworks Inc. 2011 – 2014. All Rights Reserved Real Time View

Kappa Architecture Data Source Stream Processing System Job Version n Serving DB Output table n Data Stream Query Job Version n + 1 Page 7 Data Access © Hortonworks Inc. 2011 – 2014. All Rights Reserved Output table n + 1

Design Patterns Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Design Pattern – What is it? Reusable Solution Commonly Occurring Problem Contextua l Software Design A General reusable solution to a commonly occurring problem within a given context in software design. Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Design Patterns – Why ? • Streaming use cases have distinct characteristics o Unpredictable incoming data patterns o Correlating multiple streams o Out-of-sequence and late events • High scale and continuous streams pose new challenges Page 10 o Peaks and valleys o Changing data characteristics over period of time o Maintain the latency and throughput SLAs © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Streaming Patterns Architectural Patterns • Real-time Streaming • Near-real-time Streaming • Lambda Architecture Functional Patterns • Stream Joins • External Lookup • Top N (Trending) • Responsive Shuffling • Rolling Windows • Out-of. Sequence Events • Kappa Architecture Page 11 Data Management Patterns © Hortonworks Inc. 2011 – 2014. All Rights Reserved Stream Security Patterns • Message Encryption • Authorized Access • Secure Cluster Authentication

Streaming Patterns – Being Discussed Architectural Patterns • Real-time Streaming • Near-real-time Streaming • Lambda Architecture Functional Patterns • Stream Joins • External Lookup • Top N (Trending) • Responsive Shuffling • Rolling Windows • Out-of. Sequence Events • Kappa Architecture Page 12 Data Management Patterns © Hortonworks Inc. 2011 – 2014. All Rights Reserved Stream Security Patterns • Message encryption • Authorized Access • Secure Cluster Authentication

External Lookup Dynamic, High Speed Enrichments With External Data Lookup Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

External Lookup - Description Referencing frequently changing external system data for event enrichments, filters or validations by minimizing the event processing latencies, system bottlenecks and maintaining high throughput. Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Page 14

External Lookup - Challenges • Increased latency due to frequent external system calls • Insufficient memory to hold all reference data in memory • Scalability and performance issues with large data reference sets • Dynamic reference data needs frequent cache purge and refreshes • External systems can become a bottleneck Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Page 15

External Lookup – Potential Options Performance Always Fetch Cache Everything Partition and Cache on the go Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Scalability Fault Tolerance

External Lookup - A Reference Use Case • Real Time Credit Card Fraud Identification and Alert Page 17 o Credit card transaction data comes as stream (typically through Kafka) o External system holds information about the card holder’s recent location o Each credit card transaction is looked up against user’s current location o If the geographic distance between the credit card transaction location and user’s recent known location is significant, the credit card transaction is flagged as potential fraud © Hortonworks Inc. 2011 – 2014. All Rights Reserved Page 17

External Lookup - Topology Overview Source Stream External Reference Data Storm Looks up user’s current location from external system and finds geo distance between transaction location and user location Credit Card Transaction Spout Partitioner Bolt Partitions data based on area code of the mobile numbers Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Alerting System User Location Information Fraud Analyzer Bolt Locally caches the user location data. Cache validity is time bound Fraud Alert Email Page 18

External Lookup - Peek in the Bolts Storm Stream is partitioned based on area code Partitioner Bolt Instance 1 CA NV Fraud Analyzer Bolt Instance 1 NY CT Partitioner Bolt Instance 2 Partitioner Bolt Instance n Page 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved TX Local cache (time sensitive) (Use lightweight caching solution like Guava) MA Fraud Analyzer Bolt Instance 2 FL NC OH Fraud Analyzer Bolt Instance n Page 19

External Lookup - Benefits of the approach • Only required data is cached (on demand) • Each bolt caches only partition of reference data • Data is locally cached so trips to external system are reduced • Cache is time sensitive • On the go cache building handles failures elegantly Page 20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Page 20

External Lookup – Applicability • Stream processing depends on external data • External data is sufficiently large that could not be hold in memory of each task • External data keeps changing • External system has scalability limitations Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Responsive Shuffling Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Responsive Shuffling - Description Automatically adjust shuffling for better performance and throughput during peaks and varying data skews in streams Page 23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Responsive Shuffling - Challenges • Incoming data stream is unpredictable and can be skewed • Skew can change from time to time • Managing latency and throughput with skews is difficult • Since streams are continuously flowing, restarting topology with new shuffling logic is practically not possible Page 24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Shuffling – Potential Options Latency & Throughput Static Shuffle Responsive Shuffle Page 25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved System Reliability Uptime

External Lookup - A Reference Use Case • Optimized HBase Inserts Page 26 o Event data is stored in HBase after storm processing o Group events such that a bolts can insert more events in HBase with less trips to region servers o Over period of time HBase regions can split/merge o Automatically adjust the event grouping as HBase region layout changes over period of time © Hortonworks Inc. 2011 – 2014. All Rights Reserved Page 26

Example – HBase writes w/o responsive shuffling App Bolt Instance 1 (100 events) HBase Bolt Instance 1 (100 events) Region Server Instance 1 (100 events) App Bolt Instance 2 (100 events) HBase Bolt Instance 2 (100 events) Region Server Instance 2 (100 events) App Bolt Instance 3 (100 events) HBase Bolt Instance 3 (100 events) Region Server Instance 3 (100 events) 300 events sent Page 27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved 300 events sent 9 trips to region servers 300 events received

Responsive Shuffling - Design Page 28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Example – HBase writes with responsive shuffling HBase Bolt Instance 1 (100 events) Region Server Instance 1 (100 events) RS Aware Partitioner HBase Bolt Instance 2 (100 events) Region Server Instance 2 (100 events) RS Aware Partitioner HBase Bolt Instance 3 (100 events) Region Server Instance 3 (100 events) App Bolt RS Aware Instance 1 Partitioner (100 events) Partitioner App Bolt automatically Instance 2 adapts to splitting/mergi (100 events) ng HBase regions App Bolt Instance 3 (100 events) 300 events sent Page 29 © Hortonworks Inc. 2011 – 2014. All Rights Reserved 300 events sent 3 trips to region servers 300 events received

Responsive Shuffling - Benefits • Topology responds to changes in data patterns and adopts accordingly • Maintains high level of SLA and throughput adherence • Minimizes needs for maintenance & hence downtimes Page 31 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Responsive Shuffling - Applicability • Change in shuffle pattern does not impact final outcome • Data stream has varying skews • Target/Reference system specifications change over period of time Page 32 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Out-of-Sequence Events Page 33 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Out-of-Sequence Events - Description An out-of-sequence event is one that's received late, sufficiently late that you've already processed events that should have been processed after the out-of-sequence event was received. Page 34 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Out-of-Sequence Events - Challenges • Hard to determine if all events in given window have been received • Need referencing of relevant data for late events • Builds more pressure on processing components • Increased latency and degraded overall system performance Page 35 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Out-of-Sequence Events – Potential Options Latency Drop Wait Fan Out Page 36 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Result Accuracy Operational Ease

Out-of-Sequence Events - Processing Monitors currently being processed events and identifying out-of-sequence events Source Spout Event Filter Bolt Ordered events Typical Processing Bolt Out-of. Sequence events Special Handling Bolt Page 37 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Based on complexities in processing, this can be extended as different topology

Out-of-Sequence Events – Benefits • Separation of concerns • Maintain the overall throughput and latency requirements • Independent scaling of components Page 38 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Out-of-Sequence Events - Applicability • When order of events matter • Processing out-of-sequence events needs special and complex logic • Stream has relatively low volume of out-of-sequence events Page 39 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Thank You! sheetal@hortonworks. com @sheetal_dolas Page 40 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Appendix Page 41 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Data Security in Kafka Page 42 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Data Security in Kafka - Description Ability to use Kafka as secure data transfer mechanism. Apache Kafka is widely used messaging platform in streaming applications. Unfortunately Kafka does not have built in support for Authentication & Authorization (yet) Page 43 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Data Security in Kafka - Flow Source Systems Sources Syslog Page 44 Data Collection Messaging System Real Time Processing Custom Collector Kafka Storm Encryptin g Producer Encrypted Messages © Hortonworks Inc. 2011 – 2014. All Rights Reserved Kafka Spout Decryptin g Bolt App Bolt

Data Security in Kafka – Encryption Details Messaging System Data Collection Kafka Topic Event Producer Event Encrypt event(s) w/ AES Encrypt AES key w/ RSA Event(s) Envelope Page 45 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Storm Decrypting Bolt Event(s) Envelope Encrypted AES Key (w/ RSA) Encrypted Event (w/ AES) Real Time Processing Event(s) Envelope Decrypt AES key w/ RSA Decrypt event(s) w/ AES Event

Data Security in Kafka – Encryption Details • RSA public/private keys are generated ahead of time and securely shared with topology • AES key is randomly generated and periodically refreshed • Only user having appropriate RSA private key can read the data • One event or a batch of events can be encrypted together as per needs Page 46 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Data Security in Kafka - Applicability • Multiple applications want to use Kafka as their source to the stream • Data is sensitive and can not be shared between applications • Other components in the pipeline are secured Page 47 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching Page 48 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching - Description Micro-batching is a technique that allows a process or task to treat a stream as a sequence of small batches or chunks of data. For incoming streams, the events can be packaged into small batches and delivered to a batch system for processing Page 49 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching - Challenges • Data delivery reliability • Unnecessary data duplication • Increased latency • Complexity in time-bound batching Page 50 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching - Design Options • Thread-based Model • Controller stream to trigger batch flush • Use of Tick Tuples Page 51 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Tick Tuples Tick tuples are system generated tuples that Storm can send to your bolt if you need to perform some actions at a fixed interval Page 52 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching - Benefits • Takes advantages of system characteristic by batching events together • Adheres to processing latency needs by ensuring that batches are executed by certain intervals • Prevents data loss by acknowledging events only after successful processing • Simple, elegant and easy to maintain code Page 53 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching - Applicability • Target systems are more efficient with bulk transactions • Processing group of events is more efficient than individual event • End to end event latency is not super sensitive Page 54 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Micro Batching – Sample Code Page 55 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Thank You! sheetal@hortonworks. com @sheetal_dolas Page 56 © Hortonworks Inc. 2011 – 2014. All Rights Reserved