Real-time Stream Processing with Apache Flink
29 likes6,300 views
The document provides an overview of Apache Flink, a framework designed for real-time stream processing and batch data management, highlighting its features such as true streaming with adjustable latency-throughput trade-off and rich functional APIs. It compares Flink with other stream processing systems like Apache Storm, Spark Streaming, and Samza, detailing their strengths and weaknesses. The document also outlines specific use cases, APIs, fault tolerance mechanisms, and performance optimizations in using Flink for data processing.
Real-time Stream Processing with Apache Flink
- 1. Marton Balassi – data Artisans Gyula Fora - SICS Flink committers [email protected] / [email protected] Real-time Stream Processing with Apache Flink
- 2. Stream Processing 2 Data stream: Infinite sequence of data arriving in a continuous fashion. Stream processing: Analyzing and acting on real-time streaming data, using continuous queries
- 3. Streaming landscape 3 Apache Storm •True streaming, low latency - lower throughput •Low level API (Bolts, Spouts) + Trident Spark Streaming •Stream processing on top of batch system, high throughput - higher latency •Functional API (DStreams), restricted by batch runtime Apache Samza •True streaming built on top of Apache Kafka, state is first class citizen •Slightly different stream notion, low level API Apache Flink •True streaming with adjustable latency-throughput trade-off •Rich functional API exploiting streaming runtime; e.g. rich windowing semantics
- 4. Apache Storm 4 True streaming, low latency - lower throughput Low level API (Bolts, Spouts) + Trident At-least-once processing guarantees Issues Costly fault tolerance Serialization Low level API
- 5. Spark Streaming 5 Stream processing emulated on a batch system High throughput - higher latency Functional API (DStreams) Exactly-once processing guarantees Issues Restricted streaming semantics Windowing High latency
- 6. Apache Samza 6 True streaming built on top of Apache Kafka Slightly different stream notion, low level API At-least-once processing guarantees with state Issues High disk IO Low level API
- 7. Apache Flink 7 True streaming with adjustable latency and throughput Rich functional API exploiting streaming runtime Flexible windowing semantics Exactly-once processing guarantees with (small) state Issues Limited state size HA issue
- 9. What is Flink 9 A "use-case complete" framework to unify batch and stream processing Event logs Historic data ETL Relational Graph analysis Machine learning Streaming analysis
- 10. Flink Historic data Kafka, RabbitMQ, ... HDFS, JDBC, ... ETL, Graphs, Machine Learning Relational, … Low latency windowing, aggregations, ... Event logs Real-time data streams What is Flink An engine that puts equal emphasis to streaming and batch 10
- 11. Flink stack 11 Python Gelly Table FlinkML SAMOA Batch Optimizer DataSet (Java/Scala) DataStream (Java/Scala)Hadoop M/R Flink Runtime Local Remote Yarn Tez Embedded Dataflow Dataflow *current Flink master + few PRs Streaming Optimizer
- 13. Overview of the API Data stream sources • File system • Message queue connectors • Arbitrary source functionality Stream transformations • Basic transformations: Map, Reduce, Filter, Aggregations… • Binary stream transformations: CoMap, CoReduce… • Windowing semantics: Policy based flexible windowing (Time, Count, Delta…) • Temporal binary stream operators: Joins, Crosses… • Native support for iterations Data stream outputs For the details please refer to the programming guide: • http://flink.apache.org/docs/latest/streaming_guide.html 13 Reduce Merge Filter Sum Map Src Sink Src
- 14. Use-case: Financial analytics 14 Reading from multiple inputs • Merge stock data from various sources Window aggregations • Compute simple statistics over windows of data Data driven windows • Define arbitrary windowing semantics Combine with sentiment analysis • Enrich your analytics with social media feeds (Twitter) Streaming joins • Join multiple data streams Detailed explanation and source code on our blog • http://flink.apache.org/news/2015/02/09/streaming-example.html
- 15. Reading from multiple inputs case class StockPrice(symbol : String, price : Double) val env = StreamExecutionEnvironment.getExecutionEnvironment val socketStockStream = env.socketTextStream("localhost", 9999) .map(x => { val split = x.split(",") StockPrice(split(0), split(1).toDouble) }) val SPX_Stream = env.addSource(generateStock("SPX")(10) _) val FTSE_Stream = env.addSource(generateStock("FTSE")(20) _) val stockStream = socketStockStream.merge(SPX_Stream, FTSE_STREAM) 15 (1) (2) (4) (3) (1) (2) (3) (4) "HDP, 23.8" "HDP, 26.6" StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(HDP, 23.8) StockPrice(HDP, 26.6)
- 16. Window aggregations val windowedStream = stockStream .window(Time.of(10, SECONDS)).every(Time.of(5, SECONDS)) val lowest = windowedStream.minBy("price") val maxByStock = windowedStream.groupBy("symbol").maxBy("price") val rollingMean = windowedStream.groupBy("symbol").mapWindow(mean _) 16 (1) (2) (4) (3) (1) (2) (4) (3) StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(HDP, 23.8) StockPrice(HDP, 26.6) StockPrice(HDP, 23.8) StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(HDP, 26.6) StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(HDP, 25.2)
- 17. Data-driven windows case class Count(symbol : String, count : Int) val priceWarnings = stockStream.groupBy("symbol") .window(Delta.of(0.05, priceChange, defaultPrice)) .mapWindow(sendWarning _) val warningsPerStock = priceWarnings.map(Count(_, 1)) .groupBy("symbol") .window(Time.of(30, SECONDS)) .sum("count") 17 (1) (2) (4) (3) (1) (2) (4) (3) StockPrice(SPX, 2113.9) StockPrice(FTSE, 6931.7) StockPrice(HDP, 23.8) StockPrice(HDP, 26.6) Count(HDP, 1)StockPrice(HDP, 23.8) StockPrice(HDP, 26.6)
- 18. Combining with a Twitter stream val tweetStream = env.addSource(generateTweets _) val mentionedSymbols = tweetStream.flatMap(tweet => tweet.split(" ")) .map(_.toUpperCase()) .filter(symbols.contains(_)) val tweetsPerStock = mentionedSymbols.map(Count(_, 1)).groupBy("symbol") .window(Time.of(30, SECONDS)) .sum("count") 18 "hdp is on the rise!" "I wish I bought more YHOO and HDP stocks" Count(HDP, 2) Count(YHOO, 1)(1) (2) (4) (3) (1) (2) (4) (3)
- 19. Streaming joins val tweetsAndWarning = warningsPerStock.join(tweetsPerStock) .onWindow(30, SECONDS) .where("symbol") .equalTo("symbol"){ (c1, c2) => (c1.count, c2.count) } val rollingCorrelation = tweetsAndWarning .window(Time.of(30, SECONDS)) .mapWindow(computeCorrelation _) 19 Count(HDP, 2) Count(YHOO, 1) Count(HDP, 1) (1,2) (1) (2) (1) (2) 0.5
- 20. Fault tolerance Exactly once semantics • Asynchronous barrier snapshotting • Checkpoint barriers streamed from the sources • Operator state checkpointing + source backup • Pluggable backend for state management 20 1 1 2 3 JM SM State manager Job manager Operator Snapshot barrier Event channel Data channel Checkpoint JM SM
- 21. Performance 21 Performance optimizations • Effective serialization due to strongly typed topologies • Operator chaining (thread sharing/no serialization) • Different automatic query optimizations Competitive performance • ~ 1.5m events / sec / core • As a comparison Storm promises ~ 1m tuples / sec / node
- 22. Roadmap 22 Persistent, high-throughput state backend Job manager high availability Application libraries • General statistics over streams • Pattern matching • Machine learning pipelines library • Streaming graph processing library Integration with other frameworks • Zeppelin (Notebook) • SAMOA (Online ML)
- 23. Summary Flink is a use-case complete framework to unify batch and stream processing True streaming runtime with high-level APIs Flexible, data-driven windowing semantics Competitive performance We are just getting started! 23
- 24. Flink Community 24 0 20 40 60 80 100 120 Jul-09 Nov-10 Apr-12 Aug-13 Dec-14 May-16 Unique git contributors
Editor's Notes
- #14: 3 main components to the system: Connectors(sources), Operators(trafos), Sinks (outputs) The source interface is as general as it gets + preimplemented connectors Rich set of operators, designed for true streaming analytics (long-standing, stateful, windowing) Sinks are very general, same as sources. Simple interfaces + pre-implemented
- #15: -The goal is to showcase the main features of the api on a “real world” example -Use-case: Analyze streams of stock market data, which consists of (Stock symbol, Stock price) pairs -Sentiment analysis: Combine the market information with information aquired from social media feeds (in this case the nr of times the stock symbol was mentioned in the twitter stream) -Use stream joins for this ^
- #16: - As a first step we need to connect to our data streams, and parse our inputs Here we use a simple socket stream and convert it to a case-class -> Could have used kafka or any other message queue + or more advanced stock representation
- #17: Talk about policy based windowing -> eviction(window size), trigger (slide size) Window operations: reduce, mapWindow Grouped vs non-grouped windowing
- #18: Flexible windowing -> awesome features Delta policy Case-class Count for convenience Mention other cool use-cases: detecting user sessions
- #19: Simple windowed word count on the filtered tweet stream for each symbol