Streaming SQL to unify batch and stream processing: Theory and practice with Apache Flink at Uber

1
Streaming SQL
to Unify Batch and Stream Processing:
Theory and Practice with Apache Flink at Uber
Strata Data Conference,
San Jose
March, 7th 2018
Fabian Hueske Shuyi Chen

What is Apache Flink?
2
Batch Processing
process static and
historic data
Data Stream
Processing
realtime results
from data streams
Event-driven
Applications
data-driven actions
and services
Stateful Computations Over Data Streams

What is Apache Flink?
3
Queries
Applications
Devices
etc.
Database
Stream
File / Object
Storage
Stateful computations over streams
real-time and historic
fast, scalable, fault tolerant, in-memory,
event time, large state, exactly-once
Historic
Data
Streams
Application

Hardened at scale
4
Streaming Platform Service
billions messages per day
A lot of Stream SQL
Streaming Platform as a Service
3700+ container running Flink,
1400+ nodes, 22k+ cores, 100s of jobs
Fraud detection
Streaming Analytics Platform
100s jobs, 1000s nodes, TBs state,
metrics, analytics, real time ML,
Streaming SQL as a platform

Powerful Abstractions
5
Process Function (events, state, time)
DataStream API (streams, windows)
SQL / Table API (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Layered abstractions to
navigate simple to complex use cases

Apache Flink’s Relational APIs
Unified APIs for batch & streaming data
A query specifies exactly the same result
regardless whether its input is
static batch data or streaming data.
6
tableEnvironment
.scan("clicks")
.groupBy('user)
.select('user, 'url.count as 'cnt)
SELECT user, COUNT(url) AS cnt
FROM clicks
GROUP BY user
LINQ-style Table APIANSI SQL

Query Translation
7
tableEnvironment
.scan("clicks")
.groupBy('user)
.select('user, 'url.count as 'cnt)
SELECT user, COUNT(url) AS cnt
FROM clicks
GROUP BY user
Input data is
bounded
(batch)
Input data is
unbounded
(streaming)

What if “clicks” is a file?
8
Clicks
user cTime url
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
user cnt
Mary 2
Bob 1
Liz 1
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
Input data is
read at once
Result is produced
at once

What if “clicks” is a stream?
9
user cTime url
user cnt
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
Clicks
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
Bob 1
Liz 1
Mary 1Mary 2
Input data is
continuously read
Result is continuously
produced
The result is identical!

Why is stream-batch unification important?
 Usability
• ANSI SQL syntax: No custom “StreamSQL” syntax.
• ANSI SQL semantics: No stream-specific results.
 Portability
• Run the same query on bounded and unbounded data
• Run the same query on recorded and real-time data
 Do we need to soften SQL semantics for streaming? 10
now
bounded query
unbounded query
past future
bounded query
start of the stream
unbounded query

DBMSs Run Queries on Streams
 Materialized views (MV) are similar to regular views,
but persisted to disk or memory
• Used to speed-up analytical queries
• MVs need to be updated when the base tables change
 MV maintenance is very similar to SQL on streams
• Base table updates are a stream of DML statements
• MV definition query is evaluated on that stream
• MV is query result and continuously updated
11

Continuous Queries in Flink
 Core concept is a “Dynamic Table”
• Dynamic tables are changing over time
 Queries on dynamic tables
• produce new dynamic tables (which are updated based on input)
• do not terminate
 Stream ↔ Dynamic table conversions
12

Stream ↔ Dynamic Table Conversions
 Append Conversions
• Records are only inserted/appended
 Upsert Conversions
• Records are inserted/updated/deleted and have a
(composite) unique key
 Changelog Conversions
• Records are inserted/updated/deleted
13

SQL Feature Set in Flink 1.5.0
 SELECT FROM WHERE
 GROUP BY / HAVING
• Non-windowed, TUMBLE, HOP, SESSION windows
 JOIN
• Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
• Non-windowed INNER JOIN
 Scalar, aggregation, table-valued UDFs
 SQL CLI Client (beta)
 [streaming only] OVER / WINDOW
• UNBOUNDED / BOUNDED PRECEDING
 [batch only] UNION / INTERSECT / EXCEPT / IN / ORDER BY
14

What can I build with this?
 Data Pipelines
• Transform, aggregate, and move events in real-time
 Low-latency ETL
• Convert and write streams to file systems, DBMS, K-V stores, indexes, …
• Convert appearing files into streams
 Stream & Batch Analytics
• Run analytical queries over bounded and unbounded data
• Query and compare historic and real-time data
 Data Preparation for Live Dashboards
• Compute and update data to visualize in real-time 15

The New York Taxi Rides Data Set
 The New York City Taxi & Limousine Commission provides a public data
set about taxi rides in New York City
 We can derive a streaming table from the data
 Table: TaxiRides
rideId: BIGINT // ID of the taxi ride
isStart: BOOLEAN // flag for pick-up (true) or drop-off (false) event
lon: DOUBLE // longitude of pick-up or drop-off location
lat: DOUBLE // latitude of pick-up or drop-off location
rowtime: TIMESTAMP // time of pick-up or drop-off event
16

Identify popular pick-up / drop-off locations
SELECT cell,
isStart,
HOP_END(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE) AS hopEnd,
COUNT(*) AS cnt
FROM (SELECT rowtime, isStart, toCellId(lon, lat) AS cell
FROM TaxiRides
WHERE isInNYC(lon, lat))
GROUP BY cell,
isStart,
HOP(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE) 17
 Compute every 5 minutes for each location the
number of departing and arriving taxis
of the last 15 minutes.

Average ride duration per pick-up location
SELECT pickUpCell,
AVG(TIMESTAMPDIFF(MINUTE, e.rowtime, s.rowtime) AS avgDuration
FROM (SELECT rideId, rowtime, toCellId(lon, lat) AS pickUpCell
FROM TaxiRides
WHERE isStart) s
JOIN
(SELECT rideId, rowtime
FROM TaxiRides
WHERE NOT isStart) e
ON s.rideId = e.rideId AND
e.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL '1' HOUR
GROUP BY pickUpCell
18
 Join start ride and end ride events on rideId and
compute average ride duration per pick-up location.

Building a Dashboard
19
Elastic
Search
Kafka
SELECT cell,
isStart,
HOP_END(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE) AS hopEnd,
COUNT(*) AS cnt
FROM (SELECT rowtime, isStart, toCellId(lon, lat) AS cell
FROM TaxiRides
WHERE isInNYC(lon, lat))
GROUP BY cell,
isStart,
HOP(rowtime, INTERVAL '5' MINUTE, INTERVAL '15' MINUTE)

Flink SQL in Production @
UBER
20

Uber's business is Real-Time
21
Uber

Challenges
Infrastructure
 100s of Billions of
messages / day
 At-least-once
processing
 Exactly-once state
processing
 99.99% SLA on
availability
 99.99% SLA on
latency
Productivity
 Target audience
 Operation people
 Data scientists
 Engineers
 Integrations
 Logging
 Backend services
 Storage systems
 Data management
 Monitoring
22
Operation
 ~1000 streaming
jobs
 Multiple DCs

Stream processing @ Uber
 Apache Samza (Since Jul. 2015)
• Scalable
• At-least-once message processing
• Managed state
• Fault tolerance
 Apache Flink (Since May, 2017)
• All of above
• Exactly-once stateful computation
• Accurate
• Unified stream & batch processing with SQL
23

24
Lifecycle of building a streaming job

Writing the job
25
Business
Logics
Input
Output
Testing Debugging
• Java/Scala
• Streaming/batch
• Duplicate code

Running the job
26
Resource
estimation
Deployment
Monitoring
& Alerts
Logging Maintenance
• Manual process
• Hard to scale beyond > 10 jobs

27
Job from idea to production takes days

28
How can we improve efficiency as a platform?

Flink SQL to be savior
29
SELECT AVG(…) FROM eats_order
WHERE …

Connectors
30
HTTP
SELECT AVG(…) FROM eats_order
WHERE …
Pinot

UI & Backend services
 To make it self-service
• SQL composition & validation
• Connectors management
31

• Job compilation and generation
• Resource estimation
32
Analyze input Analyze query
Test
deployment
Kafka input rate
Hive metastore data SELECT * FROM ...
YARN containers
CPU
Heap memory

• Job deployment
33
Sandbox
• Functional correctness
• Play around with SQL
Staging
• System generated estimate
• Production like load
Production
• Managed
Promote

34
• Job management

 To support Uber scale
• Monitoring and alert automation
• Auto-scaling
• Job recovery
• DC failover
35
Watchdog

AthenaX
 Uber's Self-service stream and batch
processing platform
• SQL
• Connectors
• UI & backend services
36

Real-time Machine Learning - UberEats ETD
38
SELECT restaurant_id,
AVG(etd) AS avg_etd
FROM restaurant_stream
GROUP BY TUMBLE(proctime, INTERVAL '5' MINUTE),
restaurant_id

Powering Restaurant Manager
39
Better Data -> Better
Food -> Better
Business = A Winning
Recipe
Eats restaurant
manager blog

AthenaX wins
 SQL abstraction with Flink
• Non-engineers to use stream processing
 E2E Self service
 Job from idea to production take minutes/hours
 Centralized place to track streaming dataflows
 Minimal human intervention, and scale
operationally to ~ 1000 jobs in production
40

AthenaX Open Source
 Uber engineering blog
 Open source repository
41

15% discount code: StrataFlink

Flink Forward SF 2018 Presenters
43

Thank you!
@fhueske
@ApacheFlink
Available on O’Reilly Early Release!

Streaming SQL to unify batch and stream processing: Theory and practice with Apache Flink at Uber

More Related Content

What's hot (20)

Similar to Streaming SQL to unify batch and stream processing: Theory and practice with Apache Flink at Uber (20)

More from Fabian Hueske (6)

Recently uploaded (20)

Streaming SQL to unify batch and stream processing: Theory and practice with Apache Flink at Uber