Data Analysis with Apache Flink (Hadoop Summit, 2015)

Aljoscha Krettek / Till Rohrmann
Flink committers
Co-founders @ data Artisans
aljoscha@apache.org / trohrmann@apache.org
Data Analysis With
Apache Flink

What is Apache Flink?
1
Functional
API
Relational
API
Graph API
Machine
Learning
…
Iterative Dataflow Engine

Apache Flink Stack
2
Python
Gelly
Table
FlinkML
SAMOA
Batch Optimizer
DataSet (Java/Scala) DataStream (Java/Scala)
Stream Builder
Hadoop
M/R
Distributed Runtime
Local Remote Yarn Tez Embedded
Dataflow
Dataflow
*current Flink master + few PRs
Table

Example Use Case: Log
Analysis
3

What Seems to be the Problem?
 Collect clicks from a
webserver log
 Find interesting URLs
 Combine with user
data
4
Web server
log
user
data base
Interesting
User Data
Extract
Clicks
Combine
Massage

The Execution Environment
 Entry point for all Flink programs
 Creates DataSets from data sources
5
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();

Getting at Those Clicks
6
DataSet<String> log = env.readTextFile("hdfs:///log");
DataSet<Tuple2<String, Integer>> clicks = log.flatMap(
(String line, Collector<Tuple2<String, Integer>> out) ->
String[] parts = in.split("*magic regex*");
if (isClick(parts)) {
out.collect(new Tuple2<>(parts[1],Integer.parseInt(parts[2])));
}
}
)
post /foo/bar… 313
get /data/pic.jpg 128
post /bar/baz… 128
post /hello/there… 42

The Table Environment
 Environment for dealing with Tables
 Converts between DataSet and Table
7
TableEnvironment tableEnv = new TableEnvironment();

Counting those Clicks
8
Table clicksTable = tableEnv.toTable(clicks, "url, userId");
Table urlClickCounts = clicksTable
.groupBy("url, userId")
.select("url, userId, url.count as count");

Getting the User Information
9
Table userInfo = tableEnv.toTable(…, "name, id, …");
Table resultTable = urlClickCounts.join(userInfo)
.where("userId = id && count > 10")
.select("url, count, name, …");

The Final Step
10
class Result {
public String url;
public int count;
public String name;
…
}
DataSet<Result> set =
tableEnv.toSet(resultTable, Result.class);
DataSet<Result> result =
set.groupBy("url").reduceGroup(new ComplexOperation());
result.writeAsText("hdfs:///result");
env.execute();

API in a Nutshell
 Element-wise
• map, flatMap, filter
 Group-wise
• groupBy, reduce, reduceGroup, combineGroup,
mapPartition, aggregate, distinct
 Binary
• join, coGroup, union, cross
 Iterations
• iterate, iterateDelta
 Physical re-organization
• rebalance, partitionByHash, sortPartition
 Streaming
• window, windowMap, coMap, ...
11

What happens under the
hood?
12

From Program to Dataflow
13
Flink Program
Dataflow Plan
Optimized Plan

Distributed Execution
14
Orchestration
Recovery
Master
Memory
Management
Serialization
Worker
Streaming
Network

Advanced Analysis:
Website Recommendation
15

Going Further
 Log analysis result:
Which user visited how
often which web site
 Which other websites
might they like?
 Recommendation by
collaborative filtering
16

Collaborative Filtering
 Recommend items based on users with
similar preferences
 Latent factor models capture underlying
characteristics of items and preferences of
user
 Predicted preference:
17
ˆru,i = xu
T
yi

Matrix Factorization
18
minX,Y ru,i - xu
T
yi( )
2
+l nu xu
2
+ ni yi
2
i
å
u
å
æ
è
ç
ö
ø
÷
ru,i¹0
å
R » XT
Y

Alternating least squares
 Iterative approximation
1. Fix X and optimize Y
2. Fix Y and optimize X
 Communication and
computation intensive
19
R=YX x
R=YX x

Matrix Factorization Pipeline
20
val featureExtractor = HashingFT()
val factorizer = ALS()
val pipeline = featureExtractor.chain(factorizer)
val clickstreamDS =
env.readCsvFile[(String, String, Int)](clickStreamData)
val parameters = ParameterMap()
.add(HashingFT.NumFeatures, 1000000)
.add(ALS.Iterations, 10)
.add(ALS.NumFactors, 50)
.add(ALS.Lambda, 1.5)
val factorization = pipeline.fit(clickstreamDS, parameters)
Clickstream
Data
Hashing
Feature
Extractor
ALS
Matrix
factorization

Does it Scale?
21
• 40 node GCE cluster, highmem-8
• 10 ALS iteration with 50 latent factors
• Based on Spark MLlib’s implementation
Scale of Netflix or Spotify

What Else Can You Do?
 Classification using SVMs
• Conversion goal prediction
 Clustering
• Visitor segmentation
 Multiple linear regression
• Visitor prediction
22

What Have You Seen?
 Flink is a general-purpose analytics system
 Highly expressive Table API
 Advanced analysis with Flink’s machine learning
library
 Jobs are executed on powerful distributed
dataflow engine
24

Flink Roadmap for 2015
 Additions to Machine Learning library
 Streaming Machine Learning
 Support for interactive programs
 Optimization for Table API queries
 SQL on top of Table API
25

WordCount in DataSet API
29
case class Word (word: String, frequency: Int)
val env = ExecutionEnvironment.getExecutionEnvironment()
val lines = env.readTextFile(...)
lines
.flatMap {line => line.split(" ").map(word => Word(word,1))}
.groupBy("word").sum("frequency”)
.print()
env.execute()
Java and Scala APIs offer the same functionality.

Log Analysis Code
30
ExecutionEnvironment env = TableEnvironment tableEnv = new TableEnvironment();
TableEnvironment tableEnv = new TableEnvironment();
DataSet<String> log = env.readTextFile("hdfs:///log");
DataSet<Tuple2<String, Integer>> clicks = log.flatMap(
new FlatMapFunction<String, Tuple2<String, Integer>>() {
public void flatMap(String in, Collector<Tuple2<>> out) {
String[] parts = in.split("*magic regex*");
if (parts[0].equals("click")) {
out.collect(new Tuple2<>(parts[1], Integer.parseInt(parts[4])));
}
}
});
Table clicksTable = tableEnv.toTable(clicks, "url, userId");
Table urlClickCounts = clicksTable
.groupBy("url, userId")
.select("url, userId, url.count as count");
Table userInfo = tableEnv.toTable(…, "name, id, …");
Table resultTable = urlClickCounts.join(userInfo)
.where("userId = id && count > 10")
.select("url, count, name, …");
DataSet<Result> result = tableEnv.toSet(resultTable, Result.class);
result.writeAsText("hdfs:///result");
env.execute();

Log Analysis Dataflow Graph
31
Log
Map
AggUsers
Join
Result
Group
Log
Map
AggUsers
Join
combine
partition
sort
merge
sort
Result
Group
partition
sort

Pipelined Execution
32
Only 1 Stage
(depending on join strategy)
Data transfer in-memory
and disk if needed
Note: Intermediate DataSets
are not necessarily “created”!

Data Analysis with Apache Flink (Hadoop Summit, 2015)

More Related Content

What's hot (20)

Similar to Data Analysis with Apache Flink (Hadoop Summit, 2015) (20)

More from Aljoscha Krettek (16)

Recently uploaded (20)

Data Analysis with Apache Flink (Hadoop Summit, 2015)

Editor's Notes