Kafka KSQL the Streaming SQL Engine for Apache

Kafka KSQL - the Streaming SQL Engine for Apache Kafka

Meet Kafka • A distributed commit log • A distributing streaming platform • Publish/subscribe messaging system • Unit: message (a row or a record) • Messages are categorized into «topics» (table) • Producers create new messages • Consumers (subscribers, readers) read messages • A single Kafka server is called a broker • Decoupling producer - consumer

The data-driven enterprise This is how it started. . .

Demo - Kafka

Pipelines

Pipelines • Kafka Connect (included in version 0. 9) • When to use Kafka Connect ? • When to use Producer/Consumer ?

Stream Processing

Stream Processing • Kafka was traditionally seen as a message bus only • Storm, Spark, Flink, Samza - stream processing systems • Kafka version 0. 10. 0 includes a stream-processing library

What is stream processing? • Data-stream – an abstraction representing an unbounded dataset • Event streams are ordered • Immutable data records • Event streams are replayable • Request/response (no) – batch (no) – stream processing (yes) • The processing has to be continous and ongoing

Stream Processing Use Cases • Customer Service (car-rental, hotel, etc. . ) • Internet of Things – predict when preventive maintenance is needed • Fraud Detection – «Can you start the anomaly-detection-job» All these cases are great for near real-time data analysis.

JOIN! CDC – Change Data Capture https: //www. confluent. io/blog/ksql-in-action-real-time-streaming-etl-from-oracle-transactional-data

Enter KSQL! (We love SQL)

Demo - KSQL