Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
1 like1,763 views
The document discusses modern stream processing using Apache Flink, highlighting its ability to handle fast-changing queries and stateful applications. It emphasizes the architecture and components of Flink, such as event sourcing, distributed applications, and support for both batch and streaming data. The text also touches on the development of stateful applications that react to time and data, paving the way for advanced analytics and real-time decision-making.
![Powerful Abstractions
9
Process Function (events, state, time)
DataStream API (streams, windows)
Stream SQL / Tables (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Layered abstractions to
navigate simple to complex use cases](https://image.slidesharecdn.com/modernstreamprocessingwithapacheflink-171117134105/85/Modern-Stream-Processing-With-Apache-Flink-GOTO-Berlin-2017-9-320.jpg)
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
- 1. Modern Stream Processing with Apache Flink® Till Rohrmann @stsffap GOTO Berlin 2017 1
- 2. 2 Original creators of Apache Flink® dA Platform 2 Open Source Apache Flink + dA Application Manager
- 3. What changes faster? Data or Query? 3 Data changes slowly compared to fast changing queries ad-hoc queries, data exploration, ML training and (hyper) parameter tuning Batch Processing Use Case Data changes fast application logic is long-lived continuous applications, data pipelines, standing queries, anomaly detection, ML evaluation, … Stream Processing Use Case
- 6. 6 Apache Flink in a Nutshell
- 7. Apache Flink in a Nutshell 7 Queries Applications Devices etc. Database Stream File / Object Storage Stateful computations over streams real-time and historic fast, scalable, fault tolerant, in-memory, event time, large state, exactly-once Historic Data Streams Application
- 8. 8 Event Streams State (Event) Time Snapshots The Core Building Blocks real-time and hindsight complex business logic consistency with out-of-order data and late data forking / versioning / time-travel
- 9. Powerful Abstractions 9 Process Function (events, state, time) DataStream API (streams, windows) Stream SQL / Tables (dynamic tables) Stream- & Batch Data Processing High-level Analytics API Stateful Event- Driven Applications val stats = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum((a, b) -> a.add(b)) def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = { // work with event and state (event, state.value) match { … } out.collect(…) // emit events state.update(…) // modify state // schedule a timer callback ctx.timerService.registerEventTimeTimer(event.timestamp + 500) } Layered abstractions to navigate simple to complex use cases
- 10. Hardened at scale 10 Athena X Streaming SQL Platform Service Streaming Platform as a Service Fraud detection Streaming Analytics Platform 100s jobs, 1000s nodes, TBs state metrics, analytics, real time ML Streaming SQL as a platform
- 12. Good old centralized architecture 12 The big mean central database $$$ The grumpy DBA Application Application Application Application
- 13. Write log and tables 13 From the Apache Cassandra docs
- 14. Modern distributed app. architecture 14 Event Sourcing Turning the DB inside out CQRS
- 15. Modern distributed app. architecture 15 Application Sensor APIs Application Application Application The limit to what you can do is how sophisticated can you compute over the stream! Boils down to: How well do you handle state and time
- 17. Event Sourcing + Memory Image 17 event log persists events (temporarily) event / command Process main memory update local variables/structures periodically snapshot the memory
- 18. Event Sourcing + Memory Image 18 Recovery: Restore snapshot and replay events since snapshot event log persists events (temporarily) Process
- 19. Distributed Memory Image 19 Distributed application, many memory images. Snapshots are all consistent together.
- 20. Stateful Event & Stream Processing 20 Scalable embedded state Access at memory speed & scales with parallel operators
- 21. Compute, State, and Storage 21 Classic tiered architecture Streaming architecture database layer compute layer application state + backup compute + stream storage and snapshot storage (backup) application state
- 22. Performance 22 synchronous reads/writes across tier boundary asynchronous writes of large blobs all modifications are local Classic tiered architecture Streaming architecture
- 23. Consistency 23 distributed transactions at scale typically at-most / at-least once exactly once per state =1 =1 Classic tiered architecture Streaming architecture
- 24. Scaling a Service 24 separately provision additional database capacity provision compute and state together Classic tiered architecture Streaming architecture provision compute
- 25. Rolling out a new Service 25 provision a new database (or add capacity to an existing one) simply occupies some additional backup space Classic tiered architecture Streaming architecture provision compute and state together
- 26. What users built on checkpoints… § Upgrades and Rollbacks § Cross Datacenter Failover § State Archiving § Application Migration § Spot Instance Region Chasing § A/B testing § … 26
- 27. 27 What is the next wave of stream processing applications?
- 28. What changes faster? Data or Query? 28 Data changes slowly compared to fast changing queries ad-hoc queries, data exploration, ML training and tuning Batch Processing Use Case Data changes fast application logic is long-lived continuous applications, data pipelines, standing queries, anomaly detection, ML evaluation, … Stream Processing Use Case
- 29. Analytics & Business Logic 29 Stream Source analytics metrics Business Logic Stream Processor events ApplicationDatabase
- 30. Blending Analytics & Business Logic 30 Stream Source Streaming Application Stream Processor events RPCs (Actor) Messages …
- 32. 32 Can one build an entire sophisticated web application (say a social network) on a stream processor? (Yes, we can!™)
- 33. 33 Social network implemented using event sourcing and CQRS (Command Query Responsibility Segregation) on Kafka/Flink/Elasticsearch/Redis More: https://data-artisans.com/blog/drivetribe-cqrs-apache-flink @
- 34. 34 The next wave of stream processing applications… … is all types of stateful applications that react to data and time!
- 35. 35 Stateful stream applications versioning, upgrading, rollback, duplicating, migrating, … Continuous applications
- 36. The dA Platform Architecture 36 dA Platform 2 Apache Flink Stateful stream processing Kubernetes Container platform Logging Streams from Kafka, Kinesis, S3, HDFS, Databases, ... dA Application Manager Application lifecycle management Metrics CI/CD Real-time Analytics Anomaly- & Fraud Detection Real-time Data Integration Reactive Microservices (and more)
- 37. Versioned Applications, not Jobs/Jars 37 Stream Processing Application Version 3 Version 2 Version 1 Code and Application Snapshot upgrade upgrade New Application Version 3a Version 2a fork / duplicate
- 38. Deployments, not Flink Clusters 38 Testing / QA Kubernetes Cluster Production Kubernetes Cluster Threat Metrics App. Testing Activity Monitor Application Fraud Detection Application Fraud Detection App. Testing
- 39. Hooks for CI/CD pipelines 39 Kubernetes Cluster Application Version 1 CI Service dA Application Manager push update trigger CI upgrade API stateful upgrade Application Version 2