Introduction to Stateful Stream Processing with Apache Flink.

1
Kostas Kloudas
@kkloudas
HUG @ Warsaw
JULY 3, 2017
Stateful Stream Processing
with Apache Flink®

2
Original creators of Apache
Flink®
Providers of the
dA Platform

3 questions and some history
▪ What is stateful stream processing?
▪ Why care about it?
▪ How does Flink do it?
▪ The evolution of Flink.
3

5
Continuous Processing
for Continuously Arriving Data

6
Batch
jobs
t
2017-06-14
01:00am
2017-06-14
00:00am
2017-06-13
11:00pm
2017-06-13
10:00pm
... ● Continuously ingesting
data
● Time-bounded batch
files
● Periodic batch jobs
The ol’ traditional batch way

7
intermediate
state
t
2017-06-14
01:00am
2017-06-14
00:00am
2017-06-13
11:00pm
2017-06-13
10:00pm
...
● Compute a counter:
#(A) per hour / 2 min
● What if:
● interval crosses batch
boundaries?
→ carry intermediate
results to next batch
● events out of order?
→ ???

▪ So, for a simple counting program:
• Custom logic for handling state
• Custom logic for handling time
• Custom logic for fault tolerance
8

▪ So, for a simple counting program:
• Custom logic for handling state
• Custom logic for handling time
• Custom logic for fault tolerance
9
Difficult and has nothing to do with your
program.

Why should we care?
▪...this is just for continuous data, right?
10

Why should we care?
▪...this is just for continuous data, right?
11
Most datasets are
continuously arriving streams.

Stream Processing
12
Computation over an endless stream of data
Your
Code
...

Distributed Stream Processing
13
Your
Code
...
...
...
Your
Code
Your
Code
● Partitions input by some key
● Distributes computation
across multiple instances
● Each instance is responsible
for some keys

qwe
14
...
...
Your
Code
Your
Code
update local
variables/structures
var x = …
if (condition(x)) {
…
}

15
...
...
Your
Code
Your
Code
qwe
update local
variables/structures
var x = …
if (condition(x)) {
…
}
● Embedded local state
● State co-partitioned with
the input stream by key

A practical stream processor
16
state
●Fault-tolerance
●Scalability
●Efficiency
●Event-time
(out-of-order events)
●Allows you to work in
event-time (e.g. timers)
time

17
Stateful Stream Processor
that handles
consistently, robustly, and efficiently
Large
Distributed State
Time / Order /
Completeness
● Stateful stream processing as
a new paradigm to
continuously process
continuously arriving data
● Produce accurate results
● Real-time is only a natural
consequence of the model
A practical stream processor

This is where Flink shines...
▪ Supports out-of-order streams
▪ Manages state transparently
• exactly-once processing
▪ Offers high throughput and low latency
▪ Scales to large deployments
• https://data-artisans.com/blog/blink-flink-alibaba-search
• https://data-artisans.com/blog/rbea-scalable-real-time-analytics-at-king
18

About time ...
20
...
...
Your
Code
Your
Code
When are my results complete?

21
...
...
Your
Code
Your
Code
When are my results complete?
Processing Time drawbacks:
• Incorrect results
• Irreproducible results
About time ...

Event Time: Watermarks
23
● Special markers,
called Watermarks
● Flow with elements
● A watermark of
timestamp t means
that no records with
timestamp < t should
be expected

25
Documentation:
https://ci.apache.org/projects/flink/flink-docs-release-
1.3/dev/event_time.html
Event Time

Fault tolerance
▪ How to ensure exactly-once semantics?
26

Fault tolerance simple case
27
event log
single process
main memoryperiodically take a
Snapshot of the memory

28
event log
single process
main memoryRecovery
restore snapshot and replay
events since snapshot
persists events
(temporarily)
Fault tolerance simple case

Fault tolerance distributed
▪ How to create consistent snapshots of
distributed state?
▪ How to do it efficiently?
29

Distributed Snapshots
30
Coordination via markers, injected into the streams

31
State index
(Hash Table
or
RocksDB)
Events flow without replication or synchronous writes
stateful
operation
source

32
Trigger checkpoint
Inject checkpoint barrier
stateful
operation
source

33
stateful
operation
source
Take state snapshot
Trigger state
copy-on-write

34
stateful
operation
source
DFS
Durably persist
snapshots
asynchronously
Processing pipeline continues

35
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
● Consistent snapshotting:
Fault tolerance

36
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
checkpointed
state
checkpointed
state
checkpointed
state
File System
Checkpoint
● Consistent snapshotting:
Fault tolerance

37
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
checkpointed
state
checkpointed
state
checkpointed
state
File System
Restore
● Recover all embedded state
● Reset position in input stream
Fault tolerance

38
Documentation:
https://ci.apache.org/projects/flink/flink-docs-release-
1.3/internals/stream_checkpointing.html
Fault tolerance

State Management: misc.
39
▪ Savepoints
▪ Rescaling
▪ Queryable State

Apache Flink Ecosystem
40
Integration
POSIX Java/Scala
Collections
POSIX

Apache Flink Stack
41
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming Data Flow
Libraries
Streaming and batch as first class citizens.

Levels of abstraction
42
Process Function (events, state, time)
DataStream API (streams, windows)
Table API (dynamic tables)
Stream SQL
low-level (stateful
stream processing)
stream processing &
analytics
declarative DSL
high-level langauge

API and Execution
43
Source
DataStream<String> lines = env.addSource(new FlinkKafkaConsumer010(…));
DataStream<Event> events = lines.map(line -> parse(line));
DataStream<Statistic> stats = stream
.keyBy("id")
.timeWindow(Time.seconds(5))
.sum(new MyAggregationFunction());
stats.addSink(new BucketingSink(path));
map()
[1]
keyBy()/
window()/
apply()
[1]
Transformation
Transformation
Sink
Streaming
DataflowkeyBy()/
window()/
apply()
[2]
map()
[1]
map()
[2]
Source
[1]
Source
[2]
Sink
[1]

TL;DR
▪ Stateful stream processing as a paradigm for
continuous data processing
▪ Flink is a sophisticated and tested stateful stream
processor
▪ Efficiency, management, and operational issues for
state are taken very seriously
48

4
Thank you!
@kkloudas
@ApacheFlink
@dataArtisans

50
Stream Processing 
and Apache Flink®'s
approach to it
@StephanEwen
Apache Flink PMC 
CTO @ data ArtisansFLINK FORWARD IS COMING BACK TO BERLIN
SEPTEMBER 11-13, 2017
BERLIN.FLINK-FORWARD.ORG -

We are hiring!
data-artisans.com/careers

Introduction to Stateful Stream Processing with Apache Flink.

Recommended

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Introduction to Stateful Stream Processing with Apache Flink. (20)

Recently uploaded (20)

Introduction to Stateful Stream Processing with Apache Flink.