Past Present and Future of Apache Flink Aljoscha

(Past), Present, and Future of Apache Flink® Aljoscha Krettek, Sensei Software Engineer

What is Apache Flink? Data Stream Processing realtime results from data streams Batch Processing process static and historic data-driven actions and services Stateful Computations Over Data Streams 2 © 2018 data Artisans Event-driven Applications

What is Apache Flink? Stateful computations over streams real-time and historic fast, scalable, fault tolerant, in-memory, event time, large state, exactly-once Queries Application Streams Database Devices Stream etc. Historic Data File / Object Storage 3 © 2018 data Artisans

Present – New in Flink 1. 5 • FLIP-6 ‒ Tighter integration with the resource manager (YARN, Mesos, Kubernetes) ‒ Enables dynamic management of resources ‒ Rework of the client/cluster communication to be REST-based • Localised Failure Recovery ‒ Failures don‘t require restoring all state from distributed storage ‒ Task. Managers keep state on machines ‒ Failures that are not caused by machine failures lead to faster recovery • 50% Network Stack Rewrite ‒ Better throughput at very low latencies ‒ Much improved backpressure handling 4 © 2018 data Artisans

Present – New in Flink 1. 5 • Broadcast State ‒ API that enables new use cases such as applying dynamic CEP patterns on a stream or join • SQL CLI ‒ An interactive command-line interface for executing SQL queries on Flink • Unified Table Sources ‒ A new interface for defining sources for a Table API/SQL program that allows defining sources from a configuration file • Loads more automated testing/release verification ‒ Streamlined testing which will lead to lower overhead for releases 5 © 2018 data Artisans

Future – Flink 1. 6 and Beyond • Autoscaling ‒ Automatic and dynamic changes in the parallelism of Flink programs and individual operators • Hot-standby replication ‒ Replication of the state of operations to multiple machines so that we can instantly migrate computation in case of failures • Zero-downtime scaling and upgrades ‒ Parallelism changes, framework upgrades and user-code updates without any downtime 6 © 2018 data Artisans

Future – Flink 1. 6 and Beyond • More Table API/SQL connectors, integration with data bases ‒ Dynamic Tables based on a data base, not a stream • End-to-end batch/streaming integration ‒ Unification of the Data. Stream and Data. Set APIs ‒ Efficient execution of batch programs and streaming programs ‒ Dynamic switching of execution modes based on workload • Support for more programming languages ‒ Upcoming: Python and Go (via Apache Beam) ‒ Tensorflow for Machine Learning and AI (also via Apache Beam) 7 © 2018 data Artisans

Wrap up • Despite all the work, Flink is already the best open-source stream processing system, in production at a ton of companies • Flink 1. 5 has exciting new features • There are even more exciting features coming up 8 © 2018 data Artisans

Thank you! @aljoscha @data. Artisans @Apache. Flink We are hiring! data-artisans. com/careers

About Data Artisans Original creators of Apache Flink® 10 © 2018 data Artisans Open Source Apache Flink + d. A Application Manager

d. A platform 11 © 2018 data Artisans dataartisans. com/download

Powered by Apache Flink 12 © 2018 data Artisans

Download the free book info. data 13 © 2018 data Artisans