Reintroducing the Stream Processor: A universal tool for continuous data analysis
1 like625 views
The document discusses the capabilities and benefits of stream processors for continuous data analysis, emphasizing speed, history, durability, and interactivity. It highlights how these processors can handle both live and historical data with low latency, exactly-once processing, and flexible state management. Additionally, it mentions popular open-source tools in the ecosystem, such as Apache Flink and Kafka, that facilitate these processes.
Reintroducing the Stream Processor: A universal tool for continuous data analysis
- 1. Re-introducing the Stream ProcessorA Universal Tool for Continuous Data Analytical Needs A Universal Tool for Continuous Data Analysis Paris Carbone Committer @ Apache Flink PhD Candidate @ KTH
- 2. Data Stream Processors Data Stream Processor can set up any data pipeline for you http://edge.alluremedia.com.au/m/l/2014/10/CoolingPipes.jpg
- 3. Is this really a step forward in data processing? A growing open-source ecosystem: kafkaflink beam apex e.g. General Idea of the tech: • Processes pipeline computation in a cluster • Computation is continuous and parallel (like data) • Event-processing logic <-> Application state • It’s production-ready and aims to simplify analytics Data Stream Processors streams
- 4. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer
- 5. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer
- 6. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer 1. Speed stream processor
- 7. 1. Speed Low-Latency Data Processing Traditionally the sole reason stream processing was used • No intermediate scheduling (you let it run) • No physical blocking (pre-compute on the go) • Copy-on-write for state and output How do stream processors achieve low latency? But Is this is only relevant for live data? CEP semantics etc. are nowadays provided as additional libraries for stream processors
- 8. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer 1. Speed 2. History stream processor
- 9. 2. History Offline Data Processing It is possible and better over bulk historical data analysis • Ability to define custom state to build up models • Large-scale support is a given (inherits cluster computing benefits) • Separation of notions of time and out-of-order processing What can stream processors do for historical data? But isn’t streaming hard to deal with failures? session windows event-timewindowse.g.,
- 10. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer 1. Speed 2. History 3. Durability stream processor
- 11. 3. Durability Exactly-Once Data Processing Traditionally streaming ~ lossy, approximate processing This is no longer true. Forget the ‘lambda architecture’. • Input records are durably stored and indexed in logs (e.g., Kafka) • Systems handle state snapshotting & transactions with external stores transparently. • Idempontent and transactional writes to external stores part 1 part 2 part 3 part 4 on Flink each stream computation either completes or repeats e.g.
- 12. 3. Durability Exactly-Once Data Processing input streams application states stream processor rollback
- 13. complex event proc fast approximate streaming ETL event logs production database 4 Aspects of Data Processing rules data warehouses + historical data application state + failover “microservices" complex analytics large-scale processing systems interactive queries data science reports dev user analyst data engineer 1. Speed 2. History 3. Durability stream processor 4. Interactivity
- 14. 4. Interactivity Querying Data Processing State Stream Processor ~ Inverse DBMS Application state holds fresh knowledge we want to query: • In some systems (e.g. Kafka-Streams) we can use the changelog • In other systems (i.e., Flink) we can query the state externally…or stream queries on custom query processor on-top of them* Alice Bob? Bob=… *https://techblog.king.com/rbea-scalable-real-time-analytics-king/
- 15. 4 Aspects of Data Processing 1. Speed 2. History 3. Durability 4. Interactivity stream processor • no physical blocking/staging • no rescheduling • efficient pipelining • copy-on-write data structures • different notions of time • flexible stateful processing • high throughput • durable input logging is a standard • automated state management • exactly-once processing • output commit & Idempotency • external access to state/ changelogs • ability to ‘stream queries’ over state
- 16. @SenorCarbone Try out Stream Processing https://flink.apache.org/ https://kafka.apache.org/ https://beam.apache.org/