Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
12 likes9,020 views
The document discusses streaming analytics using Apache Flink, highlighting its capabilities for real-time and batch processing through features like low latency, fault tolerance, and various time semantics. It covers the architecture, data flow, and examples of case classes for event processing, as well as techniques for stateful streaming and complex event processing. The document concludes with insights on future enhancements and use cases of Flink in large-scale data environments.
1 of 50
Downloaded 195 times
![Programs and Dataflows
8
Source
Transformation
Transformation
Sink
val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…))
val events: DataStream[Event] = lines.map((line) => parse(line))
val stats: DataStream[Statistic] = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum(new MyAggregationFunction())
stats.addSink(new RollingSink(path))
Source
[1]
map()
[1]
keyBy()/
window()/
apply()
[1]
Sink
[1]
Source
[2]
map()
[2]
keyBy()/
window()/
apply()
[2]
Streaming
Dataflow](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-8-320.jpg)
![Time-Windowed Aggregations
11
case class Event(sensor: String, measure: Double)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-11-320.jpg)
![Time-Windowed Aggregations
12
case class Event(sensor: String, measure: Double)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("sensor")
.timeWindow(Time.seconds(60), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-12-320.jpg)
![Session-Windowed Aggregations
13
case class Event(sensor: String, measure: Double)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("sensor")
.window(EventTimeSessionWindows.withGap(Time.seconds(60)))
.max("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-13-320.jpg)
![Pattern Detection
14
case class Event(producer: String, evtType: Int, msg: String)
case class Alert(msg: String)
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("producer")
.flatMap(new RichFlatMapFuncion[Event, Alert]() {
lazy val state: ValueState[Int] = getRuntimeContext.getState(…)
def flatMap(event: Event, out: Collector[Alert]) = {
val newState = state.value() match {
case 0 if (event.evtType == 0) => 1
case 1 if (event.evtType == 1) => 0
case x => out.collect(Alert(event.msg, x)); 0
}
state.update(newState)
}
})](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-14-320.jpg)
![Pattern Detection
15
case class Event(producer: String, evtType: Int, msg: String)
case class Alert(msg: String)
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("producer")
.flatMap(new RichFlatMapFuncion[Event, Alert]() {
lazy val state: ValueState[Int] = getRuntimeContext.getState(…)
def flatMap(event: Event, out: Collector[Alert]) = {
val newState = state.value() match {
case 0 if (event.evtType == 0) => 1
case 1 if (event.evtType == 1) => 0
case x => out.collect(Alert(event.msg, x)); 0
}
state.update(newState)
}
})
Embedded key/value
state store](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-15-320.jpg)
![Example: Windowing by Time
19
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-19-320.jpg)
![Processing Time
23
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(ProcessingTime)
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-23-320.jpg)
![Ingestion Time
24
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(IngestionTime)
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-24-320.jpg)
![Event Time
25
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(EventTime)
val stream: DataStream[Event] = env.addSource(…)
stream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-25-320.jpg)
![Event Time
26
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(EventTime)
val stream: DataStream[Event] = env.addSource(…)
val tsStream = stream.assignTimestampsAndWatermarks(
new MyTimestampsAndWatermarkGenerator())
tsStream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-26-320.jpg)
![Watermarks in Parallel
28
Source
(1)
Source
(2)
map
(1)
map
(2)
window
(1)
window
(2)
29
29
17
14
14
29
14
14
W(33)
W(17)
W(17)
A|30B|31
C|30
D|15
E|30
F|15G|18H|20
K|35
Watermark
Event Time
at the operator
Event
[id|timestamp]
Event Time
at input streams
Watermark
Generation
M|39N|39Q|44
L|22O|23R|37](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-28-320.jpg)
![Back to the Aggregation Example
31
case class Event(id: String, measure: Double, timestamp: Long)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream: DataStream[Event] = env.addSource(
new FlinkKafkaConsumer09(topic, schema, properties))
stream
.keyBy("id")
.timeWindow(Time.seconds(15), Time.seconds(5))
.sum("measure")
Stateful](https://image.slidesharecdn.com/streaminganalyticswithapacheflinkstephanewen-160511192751/85/Advanced-Streaming-Analytics-with-Apache-Flink-and-Apache-Kafka-Stephan-Ewen-31-320.jpg)
Ad
Recommended
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks This document discusses the integration of Apache Spark with Kubernetes on Google Cloud, highlighting its advantages for running data engineering and machine learning workloads within existing infrastructure. It outlines benefits such as improved cost optimization, faster scaling, and enhanced resource management through Google Kubernetes Engine (GKE) and Dataproc, while detailing implementation steps and monitoring options. Additionally, it covers the compatibility with big data ecosystem tools, job execution, and enterprise security features.
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Databricks Structured Streaming provides stateful stream processing capabilities in Spark SQL through built-in operations like aggregations and joins as well as user-defined stateful transformations. It handles state automatically through watermarking to limit state size by dropping old data. For arbitrary stateful logic, MapGroupsWithState requires explicit state management by the user.
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...Databricks The document discusses the evolution and future of data management, focusing on the 'lakehouse' paradigm, which integrates benefits of data lakes and warehouses to simplify data architecture. It highlights challenges faced by data professionals regarding data quality, timeliness, and scalability, alongside solutions provided by technologies like Delta Lake and MLflow for enhanced analytics. The content emphasizes the importance of centralized model management in machine learning development and the role of MLflow in facilitating this process.
Free Training: How to Build a Lakehouse
Free Training: How to Build a LakehouseDatabricks Doug Bateman, a principal data engineering instructor at Databricks, presented on how to build a Lakehouse architecture. He began by introducing himself and his background. He then discussed the goals of describing key Lakehouse features, explaining how Delta Lake enables it, and developing a sample Lakehouse using Databricks. The key aspects of a Lakehouse are that it supports diverse data types and workloads while enabling using BI tools directly on source data. Delta Lake provides reliability, consistency, and performance through its ACID transactions, automatic file consolidation, and integration with Spark. Bateman concluded with a demo of creating a Lakehouse.
Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureDatabricks This document provides an introduction and overview of SQL Analytics on Lakehouse Architecture. It discusses the instructor Doug Bateman's background and experience. The course goals are outlined as describing key features of a data Lakehouse, explaining how Delta Lake enables a Lakehouse architecture, and defining features of the Databricks SQL Analytics user interface. The course agenda is then presented, covering topics on Lakehouse Architecture, Delta Lake, and a Databricks SQL Analytics demo. Background is also provided on Lakehouse architecture, how it combines the benefits of data warehouses and data lakes, and its key features.
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks The document provides a detailed comparison of Delta Lake, Apache Iceberg, and Apache Hudi, focusing on their key features such as transaction support, data mutation, and streaming capabilities. Delta Lake offers strong integration with Spark, Iceberg excels in design abstraction for large datasets, and Hudi specializes in streaming data ingestion. The analysis covers aspects like schema evolution, performance optimization, and tooling, concluding that each technology has its unique strengths and use cases.
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward The document outlines the design and manual migration steps for multi-cluster Kafka source in Apache Flink, detailing the necessary tasks for swapping and upgrading clusters while minimizing downtime. It highlights the challenges of scaling multiple Kafka clusters across hybrid cloud environments, emphasizing complex operability and failover issues. The document also discusses benefits such as automated migrations and future work opportunities regarding watermark alignment and optimizations.
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotXiang Fu The document discusses real-time analytics using Trino and Apache Pinot, focusing on balancing latency and flexibility without sacrificing performance. It covers various ETL trade-offs like pre-joined, pre-aggregated, and pre-cubed tables, and highlights Apache Pinot's capabilities in handling high query loads and fast ingestion. It also details the Trino Pinot connector's functionalities, including aggregation pushdown and the configuration of data handling for optimized query performance.
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...Databricks The document details stateful stream processing in Apache Spark's Structured Streaming, highlighting its fast, scalable, and fault-tolerant capabilities with high-level APIs. It covers the anatomy of streaming queries, including sources, transformations, and sinks, while emphasizing the importance of state management, watermarking, and fault recovery mechanisms. Additionally, it discusses various operations such as streaming aggregation, deduplication, and user-defined stateful processing using APIs like mapGroupsWithState.
Kafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema RegistryJean-Paul Azar The document discusses Confluent Schema Registry, which stores and manages Avro schemas for Kafka clients. It allows producers and consumers to serialize and deserialize Kafka records to and from Avro format. The Schema Registry performs compatibility checks between the schema used by producers and consumers, and handles schema evolution if needed to allow schemas to change over time in a backwards compatible manner. It provides APIs for registering, retrieving, and checking compatibility of schemas.
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent Kafka Streams is a client library for building distributed applications that process streaming data stored in Apache Kafka. It provides a high-level streams DSL that allows developers to express streaming applications as set of processing steps. Alternatively, developers can use the lower-level processor API to implement custom business logic. Kafka Streams handles tasks like fault-tolerance, scalability and state management. It represents data as streams for unbounded data or tables for bounded state. Common operations include transformations, aggregations, joins and table operations.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward The document discusses the implementation of a near real-time, exactly-once financial data processing pipeline using Kafka, Flink, and Pinot at Stripe, addressing the challenges of processing large volumes of transactions without missing or duplicating them. Key requirements include low latency and operational efficiency, with focus on deduplication strategies and handling Kafka transactions carefully to prevent data loss. Additionally, it outlines lessons learned and offers best practices for managing the architecture effectively.
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021StreamNative The document provides an overview of Trino, a fast distributed SQL query engine designed for large datasets across various sources, highlighting its evolution from Presto and integration with Apache Pulsar. It discusses the advantages of Trino, including its query federation capabilities, and the development history along with community engagement. The document also includes links to community resources and contributions related to Trino.
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent The document provides an overview of Kafka, highlighting its developer best practices and the event streaming paradigm that enables efficient data integration. It details the architecture of Kafka, including producers, consumers, brokers, and topics, and emphasizes the importance of using schemas and configuring producer properties for data consistency and performance. Confluent is positioned as a key player in the enterprise event streaming market, enhancing Kafka's capabilities for large-scale deployments.
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta LakeDatabricks The document discusses the enhancement of Apache Spark with Delta Lake, focusing on the integration of data lakes for machine learning and data analytics. It outlines the challenges associated with data lakes, such as messy data and updates, and presents Delta Lake as a solution that improves data quality and processing efficiency. The document also emphasizes Delta Lake's capabilities for handling large datasets and its growing adoption among organizations worldwide.
Introduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang The document provides an overview of stream processing using Kafka, outlining its key concepts, architecture, and the advantages of using Kafka for real-time data streaming. It discusses the Kafka Streams library introduced in Apache Kafka since version 0.10, highlighting features such as event-at-a-time processing, state management, and fault tolerance. The document also includes examples of Kafka Streams DSL and explains the challenges of stream processing such as ordering, partitioning, and state management.
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner The document discusses various real-life use cases and architectures for event streaming using Apache Kafka and Confluent, highlighting its applications across multiple industries such as financial services, healthcare, and automotive. Key benefits include improving customer experience, increasing operational efficiency, and enabling real-time data processing for applications like fraud detection, predictive maintenance, and digital transformation. The presentation outlines how organizations can leverage event streaming for business value while managing challenges in security and integration.
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner This document discusses the comparison between Kappa and Lambda architectures for event streaming and data processing, highlighting their use cases, trade-offs, and the technologies involved. It emphasizes the benefits of Kappa architecture, such as simplicity, real-time processing, and scalability, while addressing concerns related to data availability and consistency. The text also includes examples of Kappa architecture implementations across various companies and applications in the domain of data analytics and real-time event handling.
Optimizing Delta/Parquet Data Lakes for Apache Spark
Optimizing Delta/Parquet Data Lakes for Apache SparkDatabricks Matthew Powers gave a presentation on optimizing Delta and Parquet data lakes. He discussed the benefits of using Delta lakes such as built-in time travel, compacting, and vacuuming capabilities. Delta lakes provide these features for free on top of Parquet files and a transaction log. Powers demonstrated how to create, compact, vacuum, partition, filter, and update Delta lakes in Spark. He showed that partitioning data significantly improves query performance by enabling data skipping and filtering at the partition level.
Apache Airflow
Apache AirflowSumit Maheshwari The document outlines a discussion by Sumit Maheshwari from Qubole about various workflow solutions, focusing on Apache Airflow as a preferred choice due to its comprehensive features and Python code base. It compares multiple options like Oozie, Pinball, and Luigi, highlighting their pros and cons, and also details Airflow's architecture and community engagement. The presentation emphasizes how Airflow, which originated at Airbnb and is now under Apache's incubation, is actively developed and used by several prominent companies.
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the HoodDatabricks Photon is a new query engine developed by Databricks designed to enhance SQL query performance on modern cloud infrastructure. It is 100% compatible with Apache Spark and offers features such as native C++ execution, efficient memory management, and support for various data use cases, with upcoming support for nested types and ANSI SQL. Photon aims to balance speed and resilience while addressing common SQL operations and enhancing query execution efficiency in Databricks environments.
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Databricks The document discusses designing ETL pipelines using Spark's structured streaming to handle real-time data processing efficiently. It outlines various design patterns for building streaming data pipelines, including handling unstructured input, key-value outputs, and joining multiple data streams while addressing common mistakes. Additionally, it emphasizes the use of Delta Lake for ACID transactions and implementing best practices for building robust streaming applications.
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connectconfluent The document serves as a practical guide for becoming proficient with Kafka Connect, detailing its integration capabilities with various data sources and sinks such as Amazon S3 and PostgreSQL. It includes instructions for configuring connectors, handling serialization, applying transformations, deploying connectors, and troubleshooting common issues. Additionally, it highlights topics like scaling, fault tolerance, and error handling with dead letter queues.
Owning Your Own (Data) Lake House
Owning Your Own (Data) Lake HouseData Con LA The document discusses the evolution of data architectures towards lake house models, integrating data warehouses and data lakes for more comprehensive analytics. It highlights the capabilities of Amazon Redshift and related AWS services like Redshift Spectrum, which allow seamless querying of data across various platforms without data duplication. It also provides information on service performance, cost benefits, and tools available for implementing these data strategies.
Making Data Timelier and More Reliable with Lakehouse Technology
Making Data Timelier and More Reliable with Lakehouse TechnologyMatei Zaharia The document discusses the challenges of modern data analytics due to complex architectures and how lakehouse technology can simplify these systems by combining data warehousing and data lake features. It highlights that current issues like data quality and timeliness arise from unnecessary complexity and multiple storage systems. Lakehouse technologies, such as metadata layers and optimized query engines, aim to provide reliable, timely access to both structured and unstructured data while reducing costs and ETL steps.
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
Spark Operator—Deploy, Manage and Monitor Spark clusters on KubernetesDatabricks The document discusses the Spark Operator, which allows deploying, managing, and monitoring Spark clusters on Kubernetes. It describes how the operator extends Kubernetes by defining custom resources and reacting to events from those resources, such as SparkCluster, SparkApplication, and SparkHistoryServer. The operator takes care of common tasks to simplify running Spark on Kubernetes and hides the complexity through an abstract operator library.
Continuous Processing with Apache Flink - Strata London 2016
Continuous Processing with Apache Flink - Strata London 2016Stephan Ewen The document discusses continuous processing with Apache Flink, emphasizing its capability to handle real-time and historical data efficiently through streaming technology. It highlights key features such as low latency, event time processing, and the importance of savepoints for fault tolerance and data reprocessing. The conclusion asserts that streaming architecture simplifies data applications and requires a powerful stream processor like Apache Flink.
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache FlinkVerverica The document discusses stateful stream processing using Apache Flink, detailing its benefits over traditional batch processing. It highlights key concepts such as fault tolerance, state management, and event-time awareness, which are fundamental to achieving reliable and efficient stream processing. Additionally, it emphasizes Flink's capabilities in handling large state efficiently and its sophisticated APIs for developers.
More Related Content
What's hot (20)
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotXiang Fu The document discusses real-time analytics using Trino and Apache Pinot, focusing on balancing latency and flexibility without sacrificing performance. It covers various ETL trade-offs like pre-joined, pre-aggregated, and pre-cubed tables, and highlights Apache Pinot's capabilities in handling high query loads and fast ingestion. It also details the Trino Pinot connector's functionalities, including aggregation pushdown and the configuration of data handling for optimized query performance.
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...Databricks The document details stateful stream processing in Apache Spark's Structured Streaming, highlighting its fast, scalable, and fault-tolerant capabilities with high-level APIs. It covers the anatomy of streaming queries, including sources, transformations, and sinks, while emphasizing the importance of state management, watermarking, and fault recovery mechanisms. Additionally, it discusses various operations such as streaming aggregation, deduplication, and user-defined stateful processing using APIs like mapGroupsWithState.
Kafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema RegistryJean-Paul Azar The document discusses Confluent Schema Registry, which stores and manages Avro schemas for Kafka clients. It allows producers and consumers to serialize and deserialize Kafka records to and from Avro format. The Schema Registry performs compatibility checks between the schema used by producers and consumers, and handles schema evolution if needed to allow schemas to change over time in a backwards compatible manner. It provides APIs for registering, retrieving, and checking compatibility of schemas.
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent Kafka Streams is a client library for building distributed applications that process streaming data stored in Apache Kafka. It provides a high-level streams DSL that allows developers to express streaming applications as set of processing steps. Alternatively, developers can use the lower-level processor API to implement custom business logic. Kafka Streams handles tasks like fault-tolerance, scalability and state management. It represents data as streams for unbounded data or tables for bounded state. Common operations include transformations, aggregations, joins and table operations.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward The document discusses the implementation of a near real-time, exactly-once financial data processing pipeline using Kafka, Flink, and Pinot at Stripe, addressing the challenges of processing large volumes of transactions without missing or duplicating them. Key requirements include low latency and operational efficiency, with focus on deduplication strategies and handling Kafka transactions carefully to prevent data loss. Additionally, it outlines lessons learned and offers best practices for managing the architecture effectively.
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021StreamNative The document provides an overview of Trino, a fast distributed SQL query engine designed for large datasets across various sources, highlighting its evolution from Presto and integration with Apache Pulsar. It discusses the advantages of Trino, including its query federation capabilities, and the development history along with community engagement. The document also includes links to community resources and contributions related to Trino.
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent The document provides an overview of Kafka, highlighting its developer best practices and the event streaming paradigm that enables efficient data integration. It details the architecture of Kafka, including producers, consumers, brokers, and topics, and emphasizes the importance of using schemas and configuring producer properties for data consistency and performance. Confluent is positioned as a key player in the enterprise event streaming market, enhancing Kafka's capabilities for large-scale deployments.
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta LakeDatabricks The document discusses the enhancement of Apache Spark with Delta Lake, focusing on the integration of data lakes for machine learning and data analytics. It outlines the challenges associated with data lakes, such as messy data and updates, and presents Delta Lake as a solution that improves data quality and processing efficiency. The document also emphasizes Delta Lake's capabilities for handling large datasets and its growing adoption among organizations worldwide.
Introduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang The document provides an overview of stream processing using Kafka, outlining its key concepts, architecture, and the advantages of using Kafka for real-time data streaming. It discusses the Kafka Streams library introduced in Apache Kafka since version 0.10, highlighting features such as event-at-a-time processing, state management, and fault tolerance. The document also includes examples of Kafka Streams DSL and explains the challenges of stream processing such as ordering, partitioning, and state management.
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner The document discusses various real-life use cases and architectures for event streaming using Apache Kafka and Confluent, highlighting its applications across multiple industries such as financial services, healthcare, and automotive. Key benefits include improving customer experience, increasing operational efficiency, and enabling real-time data processing for applications like fraud detection, predictive maintenance, and digital transformation. The presentation outlines how organizations can leverage event streaming for business value while managing challenges in security and integration.
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner This document discusses the comparison between Kappa and Lambda architectures for event streaming and data processing, highlighting their use cases, trade-offs, and the technologies involved. It emphasizes the benefits of Kappa architecture, such as simplicity, real-time processing, and scalability, while addressing concerns related to data availability and consistency. The text also includes examples of Kappa architecture implementations across various companies and applications in the domain of data analytics and real-time event handling.
Optimizing Delta/Parquet Data Lakes for Apache Spark
Optimizing Delta/Parquet Data Lakes for Apache SparkDatabricks Matthew Powers gave a presentation on optimizing Delta and Parquet data lakes. He discussed the benefits of using Delta lakes such as built-in time travel, compacting, and vacuuming capabilities. Delta lakes provide these features for free on top of Parquet files and a transaction log. Powers demonstrated how to create, compact, vacuum, partition, filter, and update Delta lakes in Spark. He showed that partitioning data significantly improves query performance by enabling data skipping and filtering at the partition level.
Apache Airflow
Apache AirflowSumit Maheshwari The document outlines a discussion by Sumit Maheshwari from Qubole about various workflow solutions, focusing on Apache Airflow as a preferred choice due to its comprehensive features and Python code base. It compares multiple options like Oozie, Pinball, and Luigi, highlighting their pros and cons, and also details Airflow's architecture and community engagement. The presentation emphasizes how Airflow, which originated at Airbnb and is now under Apache's incubation, is actively developed and used by several prominent companies.
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the HoodDatabricks Photon is a new query engine developed by Databricks designed to enhance SQL query performance on modern cloud infrastructure. It is 100% compatible with Apache Spark and offers features such as native C++ execution, efficient memory management, and support for various data use cases, with upcoming support for nested types and ANSI SQL. Photon aims to balance speed and resilience while addressing common SQL operations and enhancing query execution efficiency in Databricks environments.
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Databricks The document discusses designing ETL pipelines using Spark's structured streaming to handle real-time data processing efficiently. It outlines various design patterns for building streaming data pipelines, including handling unstructured input, key-value outputs, and joining multiple data streams while addressing common mistakes. Additionally, it emphasizes the use of Delta Lake for ACID transactions and implementing best practices for building robust streaming applications.
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connectconfluent The document serves as a practical guide for becoming proficient with Kafka Connect, detailing its integration capabilities with various data sources and sinks such as Amazon S3 and PostgreSQL. It includes instructions for configuring connectors, handling serialization, applying transformations, deploying connectors, and troubleshooting common issues. Additionally, it highlights topics like scaling, fault tolerance, and error handling with dead letter queues.
Owning Your Own (Data) Lake House
Owning Your Own (Data) Lake HouseData Con LA The document discusses the evolution of data architectures towards lake house models, integrating data warehouses and data lakes for more comprehensive analytics. It highlights the capabilities of Amazon Redshift and related AWS services like Redshift Spectrum, which allow seamless querying of data across various platforms without data duplication. It also provides information on service performance, cost benefits, and tools available for implementing these data strategies.
Making Data Timelier and More Reliable with Lakehouse Technology
Making Data Timelier and More Reliable with Lakehouse TechnologyMatei Zaharia The document discusses the challenges of modern data analytics due to complex architectures and how lakehouse technology can simplify these systems by combining data warehousing and data lake features. It highlights that current issues like data quality and timeliness arise from unnecessary complexity and multiple storage systems. Lakehouse technologies, such as metadata layers and optimized query engines, aim to provide reliable, timely access to both structured and unstructured data while reducing costs and ETL steps.
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
Spark Operator—Deploy, Manage and Monitor Spark clusters on KubernetesDatabricks The document discusses the Spark Operator, which allows deploying, managing, and monitoring Spark clusters on Kubernetes. It describes how the operator extends Kubernetes by defining custom resources and reacting to events from those resources, such as SparkCluster, SparkApplication, and SparkHistoryServer. The operator takes care of common tasks to simplify running Spark on Kubernetes and hides the complexity through an abstract operator library.
Similar to Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen (20)
Continuous Processing with Apache Flink - Strata London 2016
Continuous Processing with Apache Flink - Strata London 2016Stephan Ewen The document discusses continuous processing with Apache Flink, emphasizing its capability to handle real-time and historical data efficiently through streaming technology. It highlights key features such as low latency, event time processing, and the importance of savepoints for fault tolerance and data reprocessing. The conclusion asserts that streaming architecture simplifies data applications and requires a powerful stream processor like Apache Flink.
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache FlinkVerverica The document discusses stateful stream processing using Apache Flink, detailing its benefits over traditional batch processing. It highlights key concepts such as fault tolerance, state management, and event-time awareness, which are fundamental to achieving reliable and efficient stream processing. Additionally, it emphasizes Flink's capabilities in handling large state efficiently and its sophisticated APIs for developers.
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...Ververica The document provides an overview of Apache Flink, a platform for stream processing that supports batch and real-time data processing. It discusses key features such as stateful computations, checkpointing for fault tolerance, and integration with Apache Kafka, emphasizing its scalability and low-latency capabilities for event-driven applications. The document also highlights various use cases and recent advancements in Flink, as well as its potential applications in machine learning and real-time optimization.
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans
Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data ArtisansEvention The document discusses Apache Flink, a stateful stream processing framework, highlighting its capabilities for handling continuous data streams and fault tolerance. It covers the architecture, including the DataStream API and the ability to consistently snapshot state for fault tolerance. Recent updates since version 1.4 include improved support for exactly-once semantics and enhancements for managing large states efficiently.
Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)
Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)Apache Flink Taiwan User Group This document provides an overview of Apache Flink and stream processing. It discusses how stream processing has changed data infrastructure by enabling real-time analysis with low latency. Traditional batch processing had limitations like high latency of hours. Flink allows analyzing streaming data with sub-second latency using mechanisms like windows, state handling, and fault tolerance through distributed snapshots. The document benchmarks Flink performance against other frameworks on a Yahoo! production use case, finding Flink can achieve over 15 million messages/second throughput.
Introduction to Stateful Stream Processing with Apache Flink.
Introduction to Stateful Stream Processing with Apache Flink.Konstantinos Kloudas Kostas Kloudas presented on stateful stream processing with Apache Flink. He discussed how Flink handles state management, fault tolerance, and time semantics to allow for continuous and accurate processing of streaming data. Flink embeds local state with keyed streams, takes consistent snapshots of distributed state, and uses watermarks to process events in event time to produce correct results even for out-of-order data. This allows Flink to provide a robust stream processing engine that scales to large deployments.
Unified Stream and Batch Processing with Apache Flink
Unified Stream and Batch Processing with Apache FlinkDataWorks Summit/Hadoop Summit The document discusses Apache Flink, an open source stream processing framework. It provides high throughput and low latency processing of both streaming and batch data. Flink allows for explicit handling of event time, stateful stream processing with exactly-once semantics, and high performance. It also supports features like windowing, sessionization, and complex event processing that are useful for building streaming applications.
Zurich Flink Meetup
Zurich Flink MeetupKonstantinos Kloudas The document discusses stateful stream processing with Apache Flink, highlighting its capability to handle continuously arriving data for accurate real-time results. It addresses the challenges of traditional batch processing, such as state and time handling, and presents Flink's solutions for fault tolerance and efficiency. Ultimately, it emphasizes Flink's sophisticated nature as a stateful stream processor that simplifies complex data management for developers.
QCon London - Stream Processing with Apache Flink
QCon London - Stream Processing with Apache FlinkRobert Metzger Robert Metzger presented on Apache Flink, an open source stream processing framework. He discussed how streaming data enables real-time analysis with low latency compared to traditional batch processing. Flink provides unique building blocks like windows, state handling, and fault tolerance to process streaming data reliably at high throughput. Benchmark results showed Flink achieving throughputs over 15 million messages/second, outperforming Storm by 35x.
Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)
Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)ucelebi Apache Flink is an open-source stream processing framework that offers low latency, high throughput, and robust event time handling. It enables efficient batch and stream processing while ensuring state and fault tolerance, capable of handling both stateless and stateful operations with precise time semantics. Recent developments include enhancements in performance, complex event processing, and upcoming features like SQL support and dynamic scaling.
GOTO Night Amsterdam - Stream processing with Apache Flink
GOTO Night Amsterdam - Stream processing with Apache FlinkRobert Metzger This document discusses Apache Flink, an open source stream processing framework. It provides an overview of Flink and how it enables low-latency stream processing compared to traditional batch processing systems. Key aspects covered include windowing, state handling, fault tolerance, and performance benchmarks showing Flink can achieve high throughput. The document demonstrates how Flink addresses challenges like out-of-order events, state management, and exactly-once processing through features like event-time processing, managed state, and distributed snapshots.
Apache Flink @ Tel Aviv / Herzliya Meetup
Apache Flink @ Tel Aviv / Herzliya MeetupRobert Metzger Apache Flink is an open-source stream processing framework designed for low latency and high throughput in event-driven applications. It provides building blocks for stateful computations over data streams and supports various use cases including streaming analytics, fraud detection, and machine learning. The document outlines key concepts, APIs, and implementation details relevant to leveraging Flink for real-time data processing.
Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...
Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...Robert Metzger The document discusses Apache Flink, emphasizing its capabilities in stream and batch processing, highlighting its community growth and development timeline. It outlines Flink's architecture, including features like low latency, high throughput, and exactly-once guarantees, while providing examples of its usage in data processing tasks. The content also touches on Flink's integration with other systems and its future developments, leading to increased stability and improved features.
Stream Processing with Apache Flink
Stream Processing with Apache FlinkC4Media The document discusses stream processing using Apache Flink, emphasizing its low latency, high throughput, and stateful processing capabilities. It contrasts traditional batch processing with stream processing, highlighting the advantages of real-time data analysis and various architectural components involved in Flink. The presentation details performance benchmarks, fault tolerance mechanisms, windowing strategies, and the overall benefits of adopting stream processing in modern data infrastructures.
Debunking Common Myths in Stream Processing
Debunking Common Myths in Stream ProcessingDataWorks Summit/Hadoop Summit The document presents a detailed discussion on common myths about stream processing with Apache Flink, debunking misconceptions regarding approximation, exactly-once guarantees, throughput versus latency tradeoffs, and the misconception that streaming is solely for real-time data. It illustrates how Flink provides capabilities like low-latency processing, high throughput, and stateful streaming, promoting the efficient handling of unbounded datasets. The conclusion emphasizes that streaming frameworks like Flink can deliver accurate results under high data volumes without needing a batch processing system.
Flink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseKostas Tzoumas This document provides an overview of Apache Flink, an open-source stream processing framework. It discusses the rise of stream processing and how Flink enables low-latency applications through features like pipelining, operator state, fault tolerance using distributed snapshots, and integration with batch processing. The document also outlines Flink's roadmap, which includes graduating its DataStream API, fully managing windowing and state, and unifying batch and stream processing.
Flink Streaming @BudapestData
Flink Streaming @BudapestDataGyula Fóra This document provides an overview of Apache Flink, an open-source platform for distributed stream and batch data processing. Flink allows for unified batch and stream processing with a simple yet powerful programming model. It features native stream processing, exactly-once fault tolerance based on consistent snapshots, and high performance optimized for streaming workloads. The document outlines Flink's APIs, state management, fault tolerance approach, and roadmap for continued improvements in 2015.
Data Stream Processing with Apache Flink
Data Stream Processing with Apache FlinkFabian Hueske Apache Flink is an open-source platform designed for scalable stream and batch processing, offering low latency, high throughput, and exactly-once consistency. It supports complex event processing and handles out-of-order streams, providing an intuitive API similar to batch processing. With a vibrant community and robust operational features, Flink aims to simplify data infrastructure and enhance real-time analytics.
Apache Flink @ NYC Flink Meetup
Apache Flink @ NYC Flink MeetupStephan Ewen This document provides an overview of Apache Flink and streaming analytics. It discusses key concepts in streaming such as event time vs processing time, watermarks, windows, and fault tolerance using checkpoints and savepoints. It provides examples of time-windowed and session-windowed aggregations as well as pattern detection using state. The document also covers mixing event time and processing time, window triggers, and reprocessing data from savepoints in streaming jobs.
Apache Flink at Strata San Jose 2016
Apache Flink at Strata San Jose 2016Kostas Tzoumas This document discusses continuous counting on data streams using Apache Flink. It begins by introducing streaming data and how counting is an important but challenging problem. It then discusses issues with batch-oriented and lambda architectures for counting. The document presents Flink's streaming architecture and DataStream API as solutions. It discusses requirements for low-latency, high-efficiency counting on streams, as well as fault tolerance, accuracy, and queryability. Benchmark results show Flink achieving sub-second latencies and high throughput. The document closes by overviewing upcoming features in Flink like SQL and dynamic scaling.
Ad
More from confluent (20)
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)confluent Hands-on Workshop:
Stream Processing Made Easy with Flink SQL on Confluent Cloud
Webinar Think Right - Shift Left - 19-03-2025.pptx
Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent Créer des Data Product réutilisables avec intégration Data Warehouse / Data Lake :
Think Right, Shift Left
Migration, backup and restore made easy using Kannika
Migration, backup and restore made easy using Kannikaconfluent The document outlines the features and benefits of Kannika Armory, a backup and restore tool designed for event-driven architecture, emphasizing data protection and recoverability. It explains the processes for migrating data between Kafka clusters and highlights the complementary roles of Kannika Armory and cluster linking in ensuring data security and disaster recovery. Additionally, it details features such as an intuitive interface, support for plugins, and the importance of maintaining a cold backup for Kafka data.
Five Things You Need to Know About Data Streaming in 2025
Five Things You Need to Know About Data Streaming in 2025confluent The document outlines key insights about data streaming in 2025, emphasizing the importance of real-time data processing and governance. It highlights Confluent's role as a leading data streaming platform, detailing its financial performance, customer impact, and technological advancements. Additionally, it discusses the necessity for companies to treat data as a product to enhance data accessibility, improve decision-making speed, and derive operational efficiencies.
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent The document discusses the challenges and solutions in last-mile data streaming for real-time applications, particularly focusing on the integration of Confluent and Lightstreamer technologies. It outlines various technical solutions for enhancing data delivery and scalability, including the use of intelligent proxies and WebSocket-based protocols. Additionally, case studies highlight practical applications in industries like finance, aerospace, and media, showcasing Lightstreamer's capabilities in managing high-concurrency interactions and real-time data streaming.
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent The document discusses the significance of real-time data streaming for financial services organizations, emphasizing that it is essential for competitiveness and innovation. Confluent's data streaming platform facilitates the accessibility and management of data through features like Kafka integration, real-time processing, and governance, which enhance operational efficiency and enable advanced applications such as fraud detection and personalized services. The document also highlights the challenges of traditional data handling methods and how Confluent's solutions aim to streamline these processes, thus driving growth and improved customer experience.
Data in Motion Tour 2024 Riyadh, Saudi Arabia
Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent The document outlines an agenda for a data and AI event featuring various speakers from Confluent, Google Cloud, and Elastic, focusing on innovations in data streaming, generative AI, and observability. It highlights discussions about real-time data processing, the Kafka platform, and the integration of AI in workflows, as well as customer success stories in the financial sector. The event emphasizes the importance of transforming data into products to enhance organizational efficiency and decision-making.
Build a Real-Time Decision Support Application for Financial Market Traders w...
Build a Real-Time Decision Support Application for Financial Market Traders w...confluent The document discusses new features and updates from Confluent, including tech talks with partners, improved fully managed data stream integrations, and enhancements to AI model capabilities. It highlights the expansion of programs aimed at facilitating seamless transitions to Confluent’s platform and deploying data streaming solutions. Additionally, it introduces Quix's real-time data engineering framework and how it complements Flink for operational analytics and ML applications.
Strumenti e Strategie di Stream Governance con Confluent Platform
Strumenti e Strategie di Stream Governance con Confluent Platformconfluent The document discusses stream governance and technical features of the Confluent platform, including schema management, data contracts, and the importance of schema adherence to prevent data quality issues. It outlines a roadmap for future developments such as data cataloging and encryption, emphasizing the need for reliable data processing and security measures. The content emphasizes the significance of enforcing data contracts to enhance data trustworthiness and reduce operational complexities.
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent The document outlines updates and features of Confluent's services, emphasizing the Q3 2024 launch with new integrations and enhancements in data streaming and management. It details the introduction of the OEM partner program and migration accelerator to facilitate smooth transitions to Confluent. Additionally, Mia-Platform is highlighted as a comprehensive internal developer platform that supports cloud-native complexity and enhances data integration and governance.
Building Real-Time Gen AI Applications with SingleStore and Confluent
Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent The document discusses best practices for debugging client applications, particularly Kafka producers and consumers, as well as Kafka Streams applications. It also highlights innovations and updates from Confluent, including new partner integrations, security enhancements, and features for managing data streams effectively. Additionally, it outlines the importance of utilizing AI and real-time data processing in modern applications while mentioning the collaboration between Confluent and SingleStore.
Unlocking value with event-driven architecture by Confluent
Unlocking value with event-driven architecture by Confluentconfluent The document outlines a presentation on event-driven architecture (EDA) and its benefits for businesses, emphasizing how EDA can enhance real-time data processing and decision-making through various applications and use cases. It discusses the transition from monoliths to event-driven microservices, highlighting the advantages of loose coupling, fault tolerance, and cost-effectiveness, along with addressing challenges such as ordering and complexity. The document also introduces Confluent's data streaming platform as a solution for managing EDA and streamlining operational efficiency across various industries.
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneconfluent The document outlines the importance of real-time data streaming for enhancing generative AI applications, emphasizing the integration of operational data across enterprises to provide reliable and scalable insights. It highlights how large language models (LLMs) and vector stores can be modularized for improved performance, requiring contextual and trustworthy data to drive value. Additionally, it details the process for AI application development, including data augmentation, inference, workflows, and post-processing, facilitated by Confluent's platform.
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent The document discusses best practices for debugging client applications, particularly those involving Kafka producers and consumers, while highlighting the benefits of data streaming for various applications such as real-time analytics and customer experiences. It emphasizes the integration of generative AI with data streaming technologies, showcasing solutions for creating scalable, event-driven architectures that efficiently handle large traffic and deliver insights. Key components include the use of real-time data updates, microservice architectures, and AI-driven operations for enhanced performance and responsiveness.
Break data silos with real-time connectivity using Confluent Cloud Connectors
Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent The document outlines a webinar discussing Confluent Cloud connectors and the advantages of using fully managed connectors over self-managed ones. It highlights the complexities of data integration, the benefits of Kafka Connect for reliable data movement, and the vast portfolio of over 80 fully managed connectors available for various applications. Additionally, it covers secure networking options, connector management, and the ease of connecting disparate data systems to operate in real-time.
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureconfluent The document discusses best practices for debugging client applications, specifically for Kafka producers and consumers, emphasizing the importance of real-time data infrastructure. It also highlights the advantages of using REST proxies for Kafka, enabling HTTP-connected devices to interact with Kafka clusters, along with introducing the capabilities of the Gravitee API management platform for event-driven APIs. Furthermore, the document presents insights into the emerging trends and features in API management, particularly in the context of Kafka integration.
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent The document discusses best practices for debugging client applications, particularly focusing on Kafka producers and consumers. It emphasizes the need for real-time analytics and operational intelligence, highlighting how companies can leverage Confluent's data streaming solutions to improve performance and resolve issues quickly. Additionally, it showcases various customer implementations of real-time data applications using Tinybird in conjunction with Confluent technology.
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent The document discusses the importance of real-time data governance in the context of generative AI and streaming data architecture. It highlights how organizations can overcome traditional data architecture challenges by unifying operational and analytical data streams to provide instant, trustworthy insights. Case studies of banks utilizing Confluent's platform demonstrate improvements in real-time experiences, data availability, and governance, leading to significant operational benefits.
Ad
Recently uploaded (20)
Complete guidance book of Asp.Net Web API
Complete guidance book of Asp.Net Web APIShabista Imam Unlock the full potential of modern web development with the Complete Guidance Book of ASP.NET Web API—your all-in-one resource for mastering RESTful services using Microsoft’s powerful ASP.NET Core framework. This book takes you on a step-by-step journey from beginner to expert, covering everything from routing and controllers to security, performance optimization, and real-world architecture.
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptxresming1 This gives an introduction to how NLP has evolved from the time of World War II till this date through the advances in approaches, architectures and word representations. From rule based approaches, it advanced to statistical approaches. from traditional machine learning algorithms it advanced to deep neural network architectures. Deep neural architectures include recurrent neural networks, long short term memory, gated recurrent units, seq2seq models, encoder decoder models, transformer architecture, upto large language models and vision language models which are multimodal in nature.
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...resming1 Lecture delivered in 2021. This gives an introduction to Natural Language Processing. It describes the use cases of NLP in daily life. It discusses the stages in NLP Pipeline. It highlights the challenges involved covering the different levels of ambiguity that could arise. It also gives a brief note on the present scenario with the latest language models, tools and frameworks/libraries for NLP.
How to Un-Obsolete Your Legacy Keypad Design
How to Un-Obsolete Your Legacy Keypad DesignEpec Engineered Technologies For any number of circumstances, obsolescence risk is ever present in the electronics industry. This is especially true for human-to-machine interface hardware, such as keypads, touchscreens, front panels, bezels, etc. This industry is known for its high mix and low-volume builds, critical design requirements, and high costs to requalify hardware. Because of these reasons, many programs will face end-of-life challenges both at the component level as well as at the supplier level.
Redesigns and qualifications can take months or even years, so proactively managing this risk is the best way to deter this. If an LED is obsolete or a switch vendor has gone out of business, there are options to proceed.
In this webinar, we cover options to redesign and reverse engineer legacy keypad and touchscreen designs.
For more information on our HMI solutions, visit https://www.epectec.com/user-interfaces.
Data Structures Module 3 Binary Trees Binary Search Trees Tree Traversals AVL...
Data Structures Module 3 Binary Trees Binary Search Trees Tree Traversals AVL...resming1 This covers binary trees, binary search trees, BST insertion, deletion, tree traversals - inorder, preorder, postorder, AVL trees, AVL tree insertion, deletion, BTrees - insertion, deletion, hash tables etc.
Rapid Prototyping for XR: Lecture 1 Introduction to Prototyping
Rapid Prototyping for XR: Lecture 1 Introduction to PrototypingMark Billinghurst Lecture 1 of a course on Rapid Prototyping for XR taught by Mark Billinghurst at Oulu University on June 9th, 2025. This lecture presents an Introduction to Prototyping.
Generative AI & Scientific Research : Catalyst for Innovation, Ethics & Impact
Generative AI & Scientific Research : Catalyst for Innovation, Ethics & ImpactAlqualsaDIResearchGr Invited keynote at the Artificial Intelligence Symposium on AI-powered Research Innovation, taking place at ENSEM (L'École Nationale Supérieure d'Électricité et de Mécanique), Casablanca on June 21, 2025. I’ll be giving a keynote titled: "Generative AI & Scientific Research: Catalyst for Innovation, Ethics & Impact". Looking forward to engaging with researchers and doctoral students on how Generative AI is reshaping the future of science, from discovery to governance — with both opportunities and responsibilities in focus.
#AI hashtag#GenerativeAI #ScientificResearch #Innovation #Ethics #Keynote #AIinScience #GAI #ResearchInnovation #Casablanca
1. Thinking, Creative Thinking, Innovation
2. Societies Evolution from 1.0 to 5.0
3. AI - 3P Approach, Use Cases & Innovation
4. GAI & Creativity
5. TrustWorthy AI
6. Guidelines on The Responsible use of GAI In Research
Microwatt: Open Tiny Core, Big Possibilities
Microwatt: Open Tiny Core, Big PossibilitiesIBM Microwatt is a lightweight, open-source core based on the OpenPOWER ISA.
It’s designed for FPGAs and easy experimentation in chip design.
Ideal for education, prototyping, and custom silicon development.
Fully open, it empowers developers to learn, modify, and innovate.
دراسة حاله لقرية تقع في جنوب غرب السودان
دراسة حاله لقرية تقع في جنوب غرب السودانمحمد قصص فتوتة دراسة حاله لقرية تقع في جنوب غرب السودان لتوفير إحتياجاتها من الطاقه من مصادر الطاقات المتجدده المتوفره بها
CST413 KTU S7 CSE Machine Learning Clustering K Means Hierarchical Agglomerat...
CST413 KTU S7 CSE Machine Learning Clustering K Means Hierarchical Agglomerat...resming1 This covers CST413 KTU S7 CSE Machine Learning Module 4 topics - Clustering, K Means clustering, Hierarchical Agglomerative clustering, Principal Component Analysis, and Expectation Maximization.
تقرير عن التحليل الديناميكي لتدفق الهواء حول جناح.pdf
تقرير عن التحليل الديناميكي لتدفق الهواء حول جناح.pdfمحمد قصص فتوتة تقرير عن التحليل الديناميكي لتدفق الهواء حول جناح مروحه ريحيه
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
Rapid Prototyping for XR: Lecture 5 - Cross Platform DevelopmentMark Billinghurst This is lecture 5 in the course about Rapid Prototyping for XR, taught by Mark Billinghurst. This lecture is about Cross Platform Development.
May 2025: Top 10 Read Articles in Data Mining & Knowledge Management Process
May 2025: Top 10 Read Articles in Data Mining & Knowledge Management ProcessIJDKP Data mining and knowledge discovery in databases have been attracting a significant amount of research, industry, and media attention of late. There is an urgent need for a new generation of computational theories and tools to assist researchers in extracting useful information from the rapidly growing volumes of digital data.
This Journal provides a forum for researchers who address this issue and to present their work in a peer-reviewed open access forum. Authors are solicited to contribute to the Journal by submitting articles that illustrate research results, projects, surveying works and industrial experiences that describe significant advances in the following areas, but are not limited to these topics only.
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
- 1. Stephan Ewen @stephanewen Streaming Analytics with Apache Flink
- 2. Apache Flink Stack 2 DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow LibrariesApache Beam Streaming and batch as first class citizens.
- 3. Today 3 Streaming and batch as first class citizens. DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow LibrariesApache Beam
- 4. 4 Streaming technology is enabling the obvious: continuous processing on data that is continuously produced
- 5. Continuous Processing with Batch Continuous ingestion Periodic (e.g., hourly) files Periodic batch jobs 5
- 6. λ Architecture "Batch layer": what we had before "Stream layer": approximate early results 6
- 7. A Stream Processing Pipeline 7 collect log analyze serve & store
- 8. Programs and Dataflows 8 Source Transformation Transformation Sink val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…)) val events: DataStream[Event] = lines.map((line) => parse(line)) val stats: DataStream[Statistic] = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum(new MyAggregationFunction()) stats.addSink(new RollingSink(path)) Source [1] map() [1] keyBy()/ window()/ apply() [1] Sink [1] Source [2] map() [2] keyBy()/ window()/ apply() [2] Streaming Dataflow
- 9. Why does Flink stream flink? 9 Low latency High Throughput Well-behaved flow control (back pressure) Make more sense of data Works on real-time and historic data True Streaming Event Time APIs Libraries Stateful Streaming Globally consistent savepoints Exactly-once semantics for fault tolerance Windows & user-defined state Flexible windows (time, count, session, roll-your own) Complex Event Processing
- 10. Streaming Analytics by Example 10
- 11. Time-Windowed Aggregations 11 case class Event(sensor: String, measure: Double) val env = StreamExecutionEnvironment.getExecutionEnvironment val stream: DataStream[Event] = env.addSource(…) stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum("measure")
- 12. Time-Windowed Aggregations 12 case class Event(sensor: String, measure: Double) val env = StreamExecutionEnvironment.getExecutionEnvironment val stream: DataStream[Event] = env.addSource(…) stream .keyBy("sensor") .timeWindow(Time.seconds(60), Time.seconds(5)) .sum("measure")
- 13. Session-Windowed Aggregations 13 case class Event(sensor: String, measure: Double) val env = StreamExecutionEnvironment.getExecutionEnvironment val stream: DataStream[Event] = env.addSource(…) stream .keyBy("sensor") .window(EventTimeSessionWindows.withGap(Time.seconds(60))) .max("measure")
- 14. Pattern Detection 14 case class Event(producer: String, evtType: Int, msg: String) case class Alert(msg: String) val stream: DataStream[Event] = env.addSource(…) stream .keyBy("producer") .flatMap(new RichFlatMapFuncion[Event, Alert]() { lazy val state: ValueState[Int] = getRuntimeContext.getState(…) def flatMap(event: Event, out: Collector[Alert]) = { val newState = state.value() match { case 0 if (event.evtType == 0) => 1 case 1 if (event.evtType == 1) => 0 case x => out.collect(Alert(event.msg, x)); 0 } state.update(newState) } })
- 15. Pattern Detection 15 case class Event(producer: String, evtType: Int, msg: String) case class Alert(msg: String) val stream: DataStream[Event] = env.addSource(…) stream .keyBy("producer") .flatMap(new RichFlatMapFuncion[Event, Alert]() { lazy val state: ValueState[Int] = getRuntimeContext.getState(…) def flatMap(event: Event, out: Collector[Alert]) = { val newState = state.value() match { case 0 if (event.evtType == 0) => 1 case 1 if (event.evtType == 1) => 0 case x => out.collect(Alert(event.msg, x)); 0 } state.update(newState) } }) Embedded key/value state store
- 16. Many more Joining streams (e.g. combine readings from sensor) Detecting Patterns (CEP) Applying (changing) rules or models to events Training and applying online machine learning models … 16
- 18. 18 The biggest change in moving from batch to streaming is handling time explicitly
- 19. Example: Windowing by Time 19 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment val stream: DataStream[Event] = env.addSource(…) stream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure")
- 20. Different Notions of Time 20 Event Producer Message Queue Flink Data Source Flink Window Operator partition 1 partition 2 Event Time Ingestion Time Window Processing Time
- 21. Time and the Dataflow Model Event Time semantics in Flink follow the Dataflow model (Apache Beam (incub.)) See previous talk by Frances Perry & Tyler Akidau For the sake of time (no pun intended) I, only briefly recapitulate on the basic concept 21
- 22. 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time Event Time vs. Processing Time 22
- 23. Processing Time 23 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(ProcessingTime) val stream: DataStream[Event] = env.addSource(…) stream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure")
- 24. Ingestion Time 24 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(IngestionTime) val stream: DataStream[Event] = env.addSource(…) stream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure")
- 25. Event Time 25 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(EventTime) val stream: DataStream[Event] = env.addSource(…) stream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure")
- 26. Event Time 26 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(EventTime) val stream: DataStream[Event] = env.addSource(…) val tsStream = stream.assignTimestampsAndWatermarks( new MyTimestampsAndWatermarkGenerator()) tsStream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure")
- 27. Watermarks 27 7 W(11)W(17) 11159121417122220 171921 Watermark Event Event timestamp Stream (in order) 7 W(11)W(20) Watermark 991011141517 Event Event timestamp 1820 192123 Stream (out of order)
- 28. Watermarks in Parallel 28 Source (1) Source (2) map (1) map (2) window (1) window (2) 29 29 17 14 14 29 14 14 W(33) W(17) W(17) A|30B|31 C|30 D|15 E|30 F|15G|18H|20 K|35 Watermark Event Time at the operator Event [id|timestamp] Event Time at input streams Watermark Generation M|39N|39Q|44 L|22O|23R|37
- 29. Per Kafka Partition Watermarks 29 Source (1) Source (2) map (1) map (2) window (1) window (2) 33 17 29 29 17 14 14 29 14 14 W(33) W(17) W(17) A|30B|73 C|33 D|18 E|31 F|15G|91H|94 K|77 Watermark Generation L|35N|39 O|97 M|89 I|21Q|23 T|99 S|97
- 30. Matters of State (Fault Tolerance, Reinstatements, etc) 30
- 31. Back to the Aggregation Example 31 case class Event(id: String, measure: Double, timestamp: Long) val env = StreamExecutionEnvironment.getExecutionEnvironment val stream: DataStream[Event] = env.addSource( new FlinkKafkaConsumer09(topic, schema, properties)) stream .keyBy("id") .timeWindow(Time.seconds(15), Time.seconds(5)) .sum("measure") Stateful
- 32. Fault Tolerance Prevent data loss (reprocess lost in-flight events) Recover state consistency (exactly-once semantics) • Pending windows & user-defined (key/value) state Checkpoint based fault tolerance • Periodically create checkpoints • Recovery: resume from last completed checkpoint • Async. Barrier Snapshots (ABS) Algorithm 32
- 33. Checkpoints 33 data stream event newer records older records State of the dataflow at point Y State of the dataflow at point X
- 34. Checkpoint Barriers Markers, injected into the streams 34
- 37. Savepoints A "Checkpoint" is a globally consistent point-in-time snapshot of the streaming program (point in stream, state) A "Savepoint" is a user-triggered retained checkpoint Streaming programs can start from a savepoint 37 Savepoint B Savepoint A
- 38. (Re)processing data (in batch) Re-processing data (what-if exploration, to correct bugs, etc.) Usually by running a batch job with a set of old files Tools that map files to times 38 2016-3-1 12:00 am 2016-3-1 1:00 am 2016-3-1 2:00 am 2016-3-11 11:00pm 2016-3-12 12:00am 2016-3-12 1:00am… Collection of files, by ingestion time 2016-3-11 10:00pm To the batch processor
- 39. Unclear Batch Boundaries 39 2016-3-1 12:00 am 2016-3-1 1:00 am 2016-3-1 2:00 am 2016-3-11 11:00pm 2016-3-12 12:00am 2016-3-12 1:00am… 2016-3-11 10:00pm To the batch processor ? ? What about sessions across batches?
- 40. (Re)processing data (streaming) Draw savepoints at times that you will want to start new jobs from (daily, hourly, …) Reprocess by starting a new job from a savepoint • Defines start position in stream (for example Kafka offsets) • Initializes pending state (like partial sessions) 40 Savepoint Run new streaming program from savepoint
- 41. Continuous Data Sources 41 2016-3-1 12:00 am 2016-3-1 1:00 am 2016-3-1 2:00 am 2016-3-11 11:00pm 2016-3-12 12:00am 2016-3-12 1:00am 2016-3-11 10:00pm … partition partition Savepoint Savepoint Stream of Kafka Partitions Stream view over sequence of files Kafka offsets + Operator state File mod timestamp + File position + Operator state WIP (target: Flink 1.1)
- 42. Complex Event Processing Primer Demo Time 42
- 43. Demo Scenario Pattern validation & violation detection: Events should follow a certain pattern, or an alert should be raised Think cybersecurity, process monitoring, etc 43
- 44. An Outlook on Things to Come 44
- 45. Flink in the wild 45 30 billion events daily 2 billion events in 10 1Gb machines data integration & distribution platform See talks by at
- 46. Roadmap Dynamic Scaling, Resource Elasticity Stream SQL CEP enhancements Incremental & asynchronous state snapshotting Mesos support More connectors, end-to-end exactly once API enhancements (e.g., joins, slowly changing inputs) Security (data encryption, Kerberos with Kafka) 46
- 47. 47 Apache Flink Meetup - Thursday, April, 28th
- 48. Flink Forward 2016, Berlin Submission deadline: June 30, 2016 Early bird deadline: July 15, 2016 www.flink-forward.org
- 50. 50 I stream, do you?