Changelog Stream Processing with Apache Flink
0 likes582 views
The document discusses Apache Flink, a powerful tool for stream processing and changing data collection, highlighting its APIs and features used in handling streams and dynamic tables. It covers how data processing is fundamentally a stream of changes, how Flink SQL operates with these streams, and the implications of using different changelog modes. Additionally, it introduces the changelog insights and the flexibility of integrating with various data systems, ultimately summarizing Flink's capabilities as a changelog processor.
1 of 27
Downloaded 25 times
![Stream-Table Duality - Example
15
An applied changelog becomes a real (materialized) table.
name amount
Alice 56
Bob 10
Alice 89
name total
Alice 56
Bob 10
changelog
+I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56]
145
materialization
CREATE TABLE Revenue
(name STRING, total INT)
WITH (…)
INSERT INTO Revenue
SELECT name, SUM(amount)
FROM Transactions
GROUP BY name
CREATE TABLE Transactions
(name STRING, amount INT)
WITH (…)](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-15-320.jpg)
![Stream-Table Duality - Example
16
An applied changelog becomes a real (materialized) table.
name amount
Alice 56
Bob 10
Alice 89
name total
Alice 56
Bob 10
+I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56]
145
materialization
CREATE TABLE Revenue
(PRIMARY KEY(name) …)
WITH (…)
INSERT INTO Revenue
SELECT name, SUM(amount)
FROM Transactions
GROUP BY name
CREATE TABLE Transactions
(name STRING, amount INT)
WITH (…)
Save ~50% of traffic if downstream system supports upserting!](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-16-320.jpg)
![Retract vs. Upsert
18
Retract
● No primary key requirements
● Works for almost every external system
● Supports duplicate rows
● In distributed system often unavoidable
à most flexible changelog mode
à default mode
Upsert
● Traffic + computation optimization
● In-place updates (idempotency)
SELECT c, COUNT(*) FROM (
SELECT COUNT(*) AS c
FROM T
GROUP BY user
)
GROUP BY c
Count 1
Subtask 1
Count 2
Subtask 1
Subtask 2
+U[1]
+U[2]
+I[…]
1=>1
2=>1
Subtask 2
+I[…]](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-18-320.jpg)
![Changelog Insights – Append-only
19
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …); // accepts all changes
INSERT INTO Result SELECT * FROM Transactions T JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-19-320.jpg)
![Changelog Insights – Updating
20
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-20-320.jpg)
![Changelog Insights – Updating with PK
21
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE], upsertMaterialize=[true])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-21-320.jpg)
![Changelog Insights – Updating with PK
22
CREATE TABLE Transaction (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
CREATE TABLE Payment (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[UniqueKey], rightInputSpec=[UniqueKey], changelogMode=[I,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])](https://image.slidesharecdn.com/215pmchangelogprocessingwithapacheflink-220809110543-62c3f603/85/Changelog-Stream-Processing-with-Apache-Flink-22-320.jpg)
Ad
Recommended
Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward The document discusses optimizing streaming data ingestion to Apache Iceberg by addressing the small files problem through a smart shuffling operator in Flink. It highlights the issues caused by too many small files, such as poor read performance and long checkpoint durations, while presenting strategies to balance data distribution across partitions. The proposed smart shuffling aims to enhance data clustering, reduce the number of files generated, and improve overall performance metrics like checkpoint duration and CPU utilization.
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergFlink Forward The document outlines Goldman Sachs' batch processing framework using Flink and Iceberg to create scalable data pipelines with a focus on decoupling producers and consumers. It discusses metadata management, merging strategies, and performance optimizations that support various data consumption patterns while ensuring efficient data access and retention. The implementation leverages Iceberg's capabilities for metadata handling and improved query performance, highlighting the benefits of modern batch processing strategies.
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkFlink Forward The document introduces the new async sink for Apache Flink, detailing its internal architecture and functionality including buffering, batching, retry mechanisms, and managing throughput. It outlines configuration options and current limitations like ordering guarantees and thread pool management. Future enhancements mentioned include rate limiting features and integration with Amazon DynamoDB.
Apache Flink 101 - the rise of stream processing and beyond
Apache Flink 101 - the rise of stream processing and beyondBowen Li This document provides an overview and summary of Apache Flink. It discusses how Flink enables stateful stream processing and beyond. Key points include that Flink allows for stateful computations over event streams in an expressive, scalable, fault-tolerant way through layered APIs. It also supports batch processing, machine learning, and serving as a stream processor that unifies streaming and batch. The document highlights many use cases of Flink at Alibaba and how it powers critical systems like real-time analytics and recommendations.
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward The document discusses various types of skew in Apache Flink, including data skew, key skew, state skew, scheduling skew, and event time/watermark skew, highlighting their causes and impacts such as workload imbalance and low resource utilization. It presents strategies and solutions to mitigate these issues, including data repartitioning, using custom operators, and adjusting parallelism settings. Key takeaways emphasize understanding and addressing skew for improved streaming performance.
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Flink Forward The document outlines the architecture and implementation details of a managed stream processing platform built on Apache Flink at LinkedIn. It describes the responsibilities of users and the stream team, the capabilities of the Flink SQL language, deployment processes using Kubernetes, and various use cases such as search and caching. Furthermore, it highlights lessons learned regarding testability, proactive platform building, and automated alert systems, aiming to enhance the efficiency and reliability of streaming applications.
Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark Wu
Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark WuFlink Forward The document presents a comprehensive overview of Flink SQL, detailing its architecture and operational mechanisms before and after version 1.9, including various SQL optimizations and the integration of a changelog mechanism for stream processing. The discussion covers how Flink SQL operates through logical and physical plans and highlights recent advancements like the Blink planner which significantly improves performance. Future enhancements aim to unify batch and streaming processing with new interfaces and increased compatibility with Hive.
Flink powered stream processing platform at Pinterest
Flink powered stream processing platform at PinterestFlink Forward Pinterest's Xenon platform is a stateful stream processing solution that enables real-time data handling for various applications, including ads, user signals, and safety measures, while supporting around 100 engineers in managing over 100 applications. The focus is on stability, developer velocity, and cloud efficiency, aiming to enhance real-time processing capabilities and operational resilience. Ongoing initiatives include automated scaling, resource optimization, and improving the developer experience with better support and tools.
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward The document outlines the design and manual migration steps for multi-cluster Kafka source in Apache Flink, detailing the necessary tasks for swapping and upgrading clusters while minimizing downtime. It highlights the challenges of scaling multiple Kafka clusters across hybrid cloud environments, emphasizing complex operability and failover issues. The document also discusses benefits such as automated migrations and future work opportunities regarding watermark alignment and optimizations.
Apache Flink internals
Apache Flink internalsKostas Tzoumas This document provides an overview of Apache Flink internals. It begins with an introduction and recap of Flink programming concepts. It then discusses how Flink programs are compiled into execution plans and executed in a pipelined fashion, as opposed to being executed eagerly like regular code. The document outlines Flink's architecture including the optimizer, runtime environment, and data storage integrations. It also covers iterative processing and how Flink handles iterations both by unrolling loops and with native iterative datasets.
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io ScaleGrid offers a fully managed PostgreSQL solution that automates high availability for on-premise and cloud deployments, allowing developers to focus on their products without vendor lock-in. It compares high availability frameworks like patroni, repmgr, and paf, outlining their features, benefits, and required interventions. ScaleGrid's platform also supports multiple databases and integrations, enabling efficient management at scale.
Fundamentals of Apache Kafka
Fundamentals of Apache KafkaChhavi Parasher The document presents an overview of Apache Kafka, describing it as a high throughput distributed messaging system designed for data pipelines and real-time event processing. It covers Kafka's architecture, terminologies, benefits such as scalability and durability, and its applications in various industries. Additionally, it discusses Kafka's production implementation at Zapr, including configurations, challenges, and tools for management and monitoring.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward The document discusses the implementation of a near real-time, exactly-once financial data processing pipeline using Kafka, Flink, and Pinot at Stripe, addressing the challenges of processing large volumes of transactions without missing or duplicating them. Key requirements include low latency and operational efficiency, with focus on deduplication strategies and handling Kafka transactions carefully to prevent data loss. Additionally, it outlines lessons learned and offers best practices for managing the architecture effectively.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward The document discusses autoscaling in Apache Flink, emphasizing the evolution of its capabilities from the introduction of rescalable state in Flink 1.2 to the deployment of reactive mode in Flink 1.15. Reactive mode allows Flink to automatically adjust the number of task managers based on workload, requiring external tools for management and scaling metrics. It outlines the pros and cons of different deployment strategies and introduces the adaptive scheduler for improving scaling decisions.
Flink SQL & TableAPI in Large Scale Production at Alibaba
Flink SQL & TableAPI in Large Scale Production at AlibabaDataWorks Summit The document discusses the implementation and impact of Flink SQL and Table API in large-scale production at Alibaba, focusing on the development of Blink, Alibaba's version of Flink, designed to enhance performance for big data applications. It highlights improvements in efficiency and user experience by unifying batch and stream processing, and showcases successful examples of its application in real-time data analysis, particularly during high-traffic events like China Singles' Day. The document also details contributions to the Flink community and enhancements like user-defined functions and aggregates that support Alibaba's extensive operational needs.
FlinkML: Large Scale Machine Learning with Apache Flink
FlinkML: Large Scale Machine Learning with Apache FlinkTheodoros Vasiloudis The document discusses Apache Flink, a large-scale data processing engine designed for batch and real-time streaming analysis, and introduces FlinkML, an effort to implement scalable machine learning on Flink with easy-to-use APIs. It covers various machine learning approaches such as supervised learning, recommendation systems, and optimization algorithms, while also addressing challenges in large-scale learning like stragglers and model parallelism. Future developments for FlinkML include new algorithms and tools for streaming ML and advanced optimization techniques.
Introduction to Apache Flink
Introduction to Apache Flinkmxmxm This document provides an introduction to Apache Flink. It begins with an overview of the presenters and structure of the presentation. It then discusses Flink's APIs, architecture, and execution model. Key concepts are explained like streaming vs batch processing, scaling, the job manager and task managers. It provides a demo of Flink's DataSet API for batch processing and explains a WordCount example program. The goal is to get attendees started with Apache Flink.
Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.
Dynamic filtering for presto join optimisation
Dynamic filtering for presto join optimisationOri Reshef The document discusses dynamic filtering for join optimization in Presto, highlighting how it improves performance by minimizing CPU and memory usage during large table joins. It outlines implementation requirements, such as adding new plan nodes and optimizing the collection and application of dynamic filters. Performance analysis demonstrates significant execution time and resource usage improvements when using dynamic filtering compared to regular joins.
Practical learnings from running thousands of Flink jobs
Practical learnings from running thousands of Flink jobsFlink Forward The document discusses practical learnings from running thousands of Flink jobs, focusing on job stability and state backend selection and tuning. Key issues highlighted include memory management challenges, job instability causes, and a comparison between HashMap and RocksDB for state backends. It concludes with optimization strategies for RocksDB to improve performance.
Introduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent ksqldb is a stream-relational database system that provides SQL abstraction over mutable tables and immutable streams, enabling online stream processing through continuous queries and state lookups. It integrates with Apache Kafka, leveraging its partitioned log structure to maintain scalability and fault tolerance. The system supports various query types, including persistent continuous queries, transient client queries, and stream-table joins, while also addressing future improvements in language syntax and query optimization.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward Stripe utilizes Change Data Capture (CDC) with Flink to efficiently capture and process data changes from databases while maintaining system reliability and scalability. The journey entails addressing complex challenges in aggregating change events across various applications and ensuring quality standards such as freshness, coverage, and correctness. CDC underpins crucial financial operations at Stripe, including dashboards and reporting, emphasizing the need for a strong infrastructure that abstracts database complexities.
How to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor NettyVMware Tanzu The document discusses common mistakes when using Reactor Netty including logging, memory leaks, timeouts, connection closed issues, and connection pools. It provides examples of logging output that show a request-response lifecycle and handling of multiple concurrent connections. The presentation covers configuring logging, avoiding object retention, setting response timeouts, handling closed connections, and sizing connection pools properly.
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraFlink Forward The document presents insights from David Moravek on the evolution of Apache Flink in a cloud-native context, highlighting its advancements from version 1.11.x to 1.15.x with features like unaligned checkpoints, reactive scaling, and generalized incremental checkpoints. It emphasizes principles such as fault tolerance, stateless deployments, resource elasticity, and automation, underlining the changes in managing stateful computations in modern architectures. Additionally, it discusses automation and tooling through declarative APIs and OpenAPI specifications for improved developer experience.
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Unlocking the Power of Apache Flink: An Introduction in 4 Acts
Unlocking the Power of Apache Flink: An Introduction in 4 ActsHostedbyConfluent The document provides an overview of Apache Flink, emphasizing its capabilities for real-time stream processing and analytics. Key features include support for various programming languages, high performance and scalability, and fault tolerance through checkpointing mechanisms. Flink is geared towards enabling real-time data pipelines and event-driven applications that can efficiently handle both bounded and unbounded streams.
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022HostedbyConfluent The document is a presentation by Timo Walther on Apache Flink, focusing on its capabilities for stream processing and the use of Flink SQL to manage data as dynamic tables. It describes the structure of Flink, including its APIs, data processing models, and concepts like changelog entry updates, stream-table duality, and operational modes. The presentation highlights Flink's usefulness for integrating various data systems and has links to additional resources and projects related to Flink.
CDC Stream Processing with Apache Flink
CDC Stream Processing with Apache FlinkTimo Walther The document provides an overview of Apache Flink, highlighting its capabilities for stream processing and the unique aspects that distinguish it in the field. It discusses the various APIs available for users to interact with Flink, focusing on the stream-table duality and the innovative use of changelogs. The speaker also emphasizes the flexibility of Flink's SQL engine for integrating diverse systems and mentions additional resources available for users.
More Related Content
What's hot (20)
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward The document outlines the design and manual migration steps for multi-cluster Kafka source in Apache Flink, detailing the necessary tasks for swapping and upgrading clusters while minimizing downtime. It highlights the challenges of scaling multiple Kafka clusters across hybrid cloud environments, emphasizing complex operability and failover issues. The document also discusses benefits such as automated migrations and future work opportunities regarding watermark alignment and optimizations.
Apache Flink internals
Apache Flink internalsKostas Tzoumas This document provides an overview of Apache Flink internals. It begins with an introduction and recap of Flink programming concepts. It then discusses how Flink programs are compiled into execution plans and executed in a pipelined fashion, as opposed to being executed eagerly like regular code. The document outlines Flink's architecture including the optimizer, runtime environment, and data storage integrations. It also covers iterative processing and how Flink handles iterations both by unrolling loops and with native iterative datasets.
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io ScaleGrid offers a fully managed PostgreSQL solution that automates high availability for on-premise and cloud deployments, allowing developers to focus on their products without vendor lock-in. It compares high availability frameworks like patroni, repmgr, and paf, outlining their features, benefits, and required interventions. ScaleGrid's platform also supports multiple databases and integrations, enabling efficient management at scale.
Fundamentals of Apache Kafka
Fundamentals of Apache KafkaChhavi Parasher The document presents an overview of Apache Kafka, describing it as a high throughput distributed messaging system designed for data pipelines and real-time event processing. It covers Kafka's architecture, terminologies, benefits such as scalability and durability, and its applications in various industries. Additionally, it discusses Kafka's production implementation at Zapr, including configurations, challenges, and tools for management and monitoring.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward The document discusses the implementation of a near real-time, exactly-once financial data processing pipeline using Kafka, Flink, and Pinot at Stripe, addressing the challenges of processing large volumes of transactions without missing or duplicating them. Key requirements include low latency and operational efficiency, with focus on deduplication strategies and handling Kafka transactions carefully to prevent data loss. Additionally, it outlines lessons learned and offers best practices for managing the architecture effectively.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward The document discusses autoscaling in Apache Flink, emphasizing the evolution of its capabilities from the introduction of rescalable state in Flink 1.2 to the deployment of reactive mode in Flink 1.15. Reactive mode allows Flink to automatically adjust the number of task managers based on workload, requiring external tools for management and scaling metrics. It outlines the pros and cons of different deployment strategies and introduces the adaptive scheduler for improving scaling decisions.
Flink SQL & TableAPI in Large Scale Production at Alibaba
Flink SQL & TableAPI in Large Scale Production at AlibabaDataWorks Summit The document discusses the implementation and impact of Flink SQL and Table API in large-scale production at Alibaba, focusing on the development of Blink, Alibaba's version of Flink, designed to enhance performance for big data applications. It highlights improvements in efficiency and user experience by unifying batch and stream processing, and showcases successful examples of its application in real-time data analysis, particularly during high-traffic events like China Singles' Day. The document also details contributions to the Flink community and enhancements like user-defined functions and aggregates that support Alibaba's extensive operational needs.
FlinkML: Large Scale Machine Learning with Apache Flink
FlinkML: Large Scale Machine Learning with Apache FlinkTheodoros Vasiloudis The document discusses Apache Flink, a large-scale data processing engine designed for batch and real-time streaming analysis, and introduces FlinkML, an effort to implement scalable machine learning on Flink with easy-to-use APIs. It covers various machine learning approaches such as supervised learning, recommendation systems, and optimization algorithms, while also addressing challenges in large-scale learning like stragglers and model parallelism. Future developments for FlinkML include new algorithms and tools for streaming ML and advanced optimization techniques.
Introduction to Apache Flink
Introduction to Apache Flinkmxmxm This document provides an introduction to Apache Flink. It begins with an overview of the presenters and structure of the presentation. It then discusses Flink's APIs, architecture, and execution model. Key concepts are explained like streaming vs batch processing, scaling, the job manager and task managers. It provides a demo of Flink's DataSet API for batch processing and explains a WordCount example program. The goal is to get attendees started with Apache Flink.
Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.
Dynamic filtering for presto join optimisation
Dynamic filtering for presto join optimisationOri Reshef The document discusses dynamic filtering for join optimization in Presto, highlighting how it improves performance by minimizing CPU and memory usage during large table joins. It outlines implementation requirements, such as adding new plan nodes and optimizing the collection and application of dynamic filters. Performance analysis demonstrates significant execution time and resource usage improvements when using dynamic filtering compared to regular joins.
Practical learnings from running thousands of Flink jobs
Practical learnings from running thousands of Flink jobsFlink Forward The document discusses practical learnings from running thousands of Flink jobs, focusing on job stability and state backend selection and tuning. Key issues highlighted include memory management challenges, job instability causes, and a comparison between HashMap and RocksDB for state backends. It concludes with optimization strategies for RocksDB to improve performance.
Introduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent ksqldb is a stream-relational database system that provides SQL abstraction over mutable tables and immutable streams, enabling online stream processing through continuous queries and state lookups. It integrates with Apache Kafka, leveraging its partitioned log structure to maintain scalability and fault tolerance. The system supports various query types, including persistent continuous queries, transient client queries, and stream-table joins, while also addressing future improvements in language syntax and query optimization.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward Stripe utilizes Change Data Capture (CDC) with Flink to efficiently capture and process data changes from databases while maintaining system reliability and scalability. The journey entails addressing complex challenges in aggregating change events across various applications and ensuring quality standards such as freshness, coverage, and correctness. CDC underpins crucial financial operations at Stripe, including dashboards and reporting, emphasizing the need for a strong infrastructure that abstracts database complexities.
How to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor NettyVMware Tanzu The document discusses common mistakes when using Reactor Netty including logging, memory leaks, timeouts, connection closed issues, and connection pools. It provides examples of logging output that show a request-response lifecycle and handling of multiple concurrent connections. The presentation covers configuring logging, avoiding object retention, setting response timeouts, handling closed connections, and sizing connection pools properly.
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraFlink Forward The document presents insights from David Moravek on the evolution of Apache Flink in a cloud-native context, highlighting its advancements from version 1.11.x to 1.15.x with features like unaligned checkpoints, reactive scaling, and generalized incremental checkpoints. It emphasizes principles such as fault tolerance, stateless deployments, resource elasticity, and automation, underlining the changes in managing stateful computations in modern architectures. Additionally, it discusses automation and tooling through declarative APIs and OpenAPI specifications for improved developer experience.
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataWorks Summit/Hadoop Summit This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Unlocking the Power of Apache Flink: An Introduction in 4 Acts
Unlocking the Power of Apache Flink: An Introduction in 4 ActsHostedbyConfluent The document provides an overview of Apache Flink, emphasizing its capabilities for real-time stream processing and analytics. Key features include support for various programming languages, high performance and scalability, and fault tolerance through checkpointing mechanisms. Flink is geared towards enabling real-time data pipelines and event-driven applications that can efficiently handle both bounded and unbounded streams.
Similar to Changelog Stream Processing with Apache Flink (20)
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022HostedbyConfluent The document is a presentation by Timo Walther on Apache Flink, focusing on its capabilities for stream processing and the use of Flink SQL to manage data as dynamic tables. It describes the structure of Flink, including its APIs, data processing models, and concepts like changelog entry updates, stream-table duality, and operational modes. The presentation highlights Flink's usefulness for integrating various data systems and has links to additional resources and projects related to Flink.
CDC Stream Processing with Apache Flink
CDC Stream Processing with Apache FlinkTimo Walther The document provides an overview of Apache Flink, highlighting its capabilities for stream processing and the unique aspects that distinguish it in the field. It discusses the various APIs available for users to interact with Flink, focusing on the stream-table duality and the innovative use of changelogs. The speaker also emphasizes the flexibility of Flink's SQL engine for integrating diverse systems and mentions additional resources available for users.
Flink's SQL Engine: Let's Open the Engine Room!
Flink's SQL Engine: Let's Open the Engine Room!HostedbyConfluent The document presents an overview of Apache Flink's SQL engine, highlighting its capabilities for stream processing using dynamic tables instead of traditional databases. It explains key concepts such as timestamp management, changelog modes, and SQL execution planning, demonstrating how SQL statements translate into physical execution plans and stream transformations. Various examples illustrate workflows, including joins and watermarks, showcasing how users can implement these features in their applications.
Why and how to leverage the power and simplicity of SQL on Apache Flink
Why and how to leverage the power and simplicity of SQL on Apache FlinkFabian Hueske The document presents the advantages and functionalities of Apache Flink, focusing on SQL's integration for stream and batch processing. It highlights Flink's ability to handle real-time data streams and the importance of unifying streaming and batch query capabilities using ANSI SQL syntax. The presentation discusses use cases, ongoing developments within the Flink community, and invites feedback on new features like the SQL query service.
Streaming SQL
Streaming SQLJulian Hyde The document discusses Streaming SQL, emphasizing its role in managing and querying streaming data alongside traditional relational data. It details query planning, the duality of streams and tables, and the importance of time-sensitive data, while also covering different windowing techniques and aggregation methods. Additionally, it highlights the need for optimizing streaming queries to achieve high-quality results in real-time applications.
2019-01-29 - Demystifying Kotlin Coroutines
2019-01-29 - Demystifying Kotlin CoroutinesEamonn Boyle This document provides an overview of Kotlin coroutines through examples. It begins with examples of synchronous and asynchronous programming approaches and their drawbacks. Coroutines are then introduced as a better solution that allows writing asynchronous code in a synchronous-looking way using suspend functions and other features. Key benefits of coroutines include avoiding callback hell and enabling easy control of thread dispatch. Examples are provided of using coroutines for tasks like interleaving reading and processing of data from a file. Under the hood, coroutines implement asynchronous behavior using state machines and continuation passing.
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/TridentDataWorks Summit/Hadoop Summit Streaming SQL allows users to query streaming data using standard SQL. Some key benefits include:
- SQL is a widely used language that lets users focus on what data is needed rather than how to process it. The system can optimize queries to meet quality of service needs.
- Streaming queries can join streams with static relations or aggregate streams using windows. Monotonic columns like timestamps help the system make progress while maintaining accuracy.
- Materialized views allow querying recent historical data from streams. This enables applications like dashboards that summarize the last hour of data.
- Streaming SQL provides a unified way to query both streaming and static data sources using a single language. This simplifies development of applications that use both streaming
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/TridentJulian Hyde This document discusses streaming SQL and how it can be used to query streaming data sources like IoT devices, web servers, and databases. Some key points discussed include:
- Streaming SQL extends standard SQL to work over both streaming and static data sources. It allows queries to be executed continuously over streaming data.
- The replay principle states that streaming queries should produce the same results as equivalent non-streaming queries over the same static data. Techniques like watermarks and monotonic columns help ensure this.
- Windowing functions allow aggregating over sliding windows of records in a stream. Various window types like tumbling and hopping windows are described.
- Apache Calcite is an open source framework that can optimize streaming SQL queries
The Current State of Table API in 2022
The Current State of Table API in 2022Flink Forward The document discusses the recent developments in the Apache Flink Table API, including the separation of the legacy planner, the introduction of features like window functions and JSON support, and enhancements for SQL capabilities. It emphasizes the interoperability between datastreams and tables, with examples showcasing various use cases such as ETL and analytics. Additionally, it outlines ongoing efforts to improve version upgrade support and the integration of built-in table storage and SQL gateway functionalities.
Streaming SQL w/ Apache Calcite
Streaming SQL w/ Apache Calcite Hortonworks This document discusses streaming SQL with Apache Calcite. It begins by introducing SQL and query planning for streaming data sources like devices, services, and databases. It then discusses why SQL is useful for streaming queries, covering topics like stream-table duality, windows, aggregations, joins between streams and tables, and optimizing streaming queries. The document concludes by introducing Apache Calcite's architecture and relational algebra approach for planning both conventional and streaming queries.
Streaming SQL with Apache Calcite
Streaming SQL with Apache CalciteJulian Hyde The document discusses the development of Streaming SQL using Apache Calcite, emphasizing its potential for combining real-time and historical data through a standardized SQL approach. It highlights the need for a consensus-driven standard and the roles of various open-source projects like Samza, Storm, and Flink in this ecosystem. The main benefits include optimized query execution for streaming data and improved access for both traditional database and message-oriented applications.
Omid: scalable and highly available transaction processing for Apache Phoenix
Omid: scalable and highly available transaction processing for Apache PhoenixDataWorks Summit Omid is a transaction processing service for Apache Phoenix designed to enhance scalability and high availability in OLTP and operational analytics on Hadoop. It allows for ACID-compliant transactions, efficient query processing through HBase, and has been integrated with Phoenix for effective transaction handling at high throughput. The document outlines the architecture, operational benefits, and experimental results that demonstrate Omid's capability to support over 500,000 transactions per second.
The Ring programming language version 1.5.2 book - Part 7 of 181
The Ring programming language version 1.5.2 book - Part 7 of 181Mahmoud Samir Fayed The document summarizes updates and new features in Ring version 1.5, including:
1) A calculator application added to the Applications folder. Better Ring Notepad updated with new styles and a main file toolbar.
2) The Ring Notepad output window displays new lines correctly and contains a clear button. It also opens and switches between large files while preparing function/class lists.
3) New functions added to the StdLib - Print2Str(), ListAllFiles(), and SystemCmd(). The WebLib updated with an HTMLPage class to create HTML outside of web applications.
4) Better support for RingQt classes including QStringRef and Bluetooth classes. An example parses an XML response from
ARIES Recovery Algorithms
ARIES Recovery AlgorithmsPulasthi Lankeshwara The document presents a comprehensive overview of transaction management in databases, focusing on recovery methods such as Write Ahead Logging (WAL) and Shadow Paging, as well as the ARIES recovery algorithm. It details how these methods handle concurrency, recovery efficiency, and maintenance of data integrity in the event of crashes. The ARIES methodology encompasses three recovery phases: analysis, redo, and undo, which systematically restore the database to a consistent state after failures.
PyCon Ukraine 2017: Operational Transformation
PyCon Ukraine 2017: Operational Transformation Max Klymyshyn Operational transformation (OT) is a technology for advanced collaborative software systems, allowing real-time collaboration despite high-latency environments. It optimistically assumes that operations from different clients won't conflict, using transformation functions to reconcile operations for consistency. Despite its importance in systems like Google Docs, OT is complex and challenging to implement correctly, with many algorithms and potential pitfalls.
Streaming SQL
Streaming SQLJulian Hyde The document discusses the features and benefits of Streaming SQL in the context of Apache Samza, highlighting its ability to execute queries over both streams and traditional relational data. It emphasizes the importance of querying strategies such as aggregation, window functions, and temporal joins, which allow for efficient data processing and retrieval. Additionally, it notes that Streaming SQL optimizes quality of service and enables the integration of streaming and historical data while adhering to standard SQL principles.
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...Flink Forward This document summarizes recent improvements to Flink SQL and Table API by Blink, Alibaba's distribution of Flink. Key improvements include support for stream-stream joins, user-defined functions, table functions and aggregate functions, retractable streams, and over/group aggregates. Blink aims to make Flink work well at large scale for Alibaba's search and recommendation systems. Many of the improvements will be included in upcoming Flink releases.
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...Flink Forward This document discusses Flink's Table and SQL APIs, which provide a unified way to write batch and streaming queries. It motivates the need for a relational API by explaining that while Flink's DataStream API is powerful, it requires more technical skills. The Table and SQL APIs allow users to focus on business logic by writing declarative queries. It describes how the APIs work, including translating queries to logical and execution plans and supporting batch, streaming and windowed queries. Finally, it outlines the current capabilities and opportunities for contributors to help expand Flink's relational features.
«Практическое применение Akka Streams» — Алексей Романчук, 2ГИС
«Практическое применение Akka Streams» — Алексей Романчук, 2ГИС2ГИС Технологии The document describes the key abstractions in Akka Streams - Source, Sink, and Flow. It provides examples of creating Sources from single values, iterators, and ticks. It also gives examples of Sinks like foreach, head, fold, and reduce. Various Flow combinators are demonstrated like map, filter, take, drop, and flatMapConcat. The document then gives an example problem of reading log files from HDFS in parallel to find 100 fraudulent transactions using Akka Streams.
Практическое применения Akka Streams
Практическое применения Akka StreamsAlexey Romanchuk The document describes the key abstractions in Akka Streams - Source, Sink, and Flow. It provides examples of creating Sources from single values, iterators, and ticks. It also gives examples of Sinks like foreach, head, fold, and reduce. Various Flow operations are demonstrated like map, filter, take, drop, and flatMapConcat. The document then gives an example problem of reading log files from HDFS in parallel to find 100 fraudulent transactions using Akka Streams.
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/TridentDataWorks Summit/Hadoop Summit
Ad
More from Flink Forward (17)
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...Flink Forward The document discusses an end-to-end near-real time model building and evaluation platform for Amazon Alexa, emphasizing continual learning for adaptability and cost efficiency. It outlines the use of Kinesis Data Analytics and Flink for handling streaming data, including challenges faced such as dataflow transparency. The roadmap includes improving existing systems and enhancing infrastructure components.
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward The document presents 'binarysortedstate', a new state primitive for Apache Flink aimed at enhancing application performance, particularly in stream sorting and event-time joins. It outlines the motivation, key functionalities, and future directions for this implementation, as well as its benefits over existing state management techniques like mapstate and liststate. The authors, Nico Kruber and David Anderson, discuss the efficient handling of user-defined keys and time-based operations, and detail experimental results indicating significant performance improvements.
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorFlink Forward The document outlines the Apache Flink Kubernetes Operator, detailing its functionalities for managing Flink applications and session clusters, including deployment, upgrades, and monitoring. Key features include support for different deployment modes, lifecycle management, and observability metrics. It also highlights the project's status, contributions from the community, and a roadmap for future enhancements.
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward The document discusses tuning Apache Kafka connectors for Flink, focusing on the migration from test to production and addressing issues related to message size and batch processing in Kafka. It outlines steps to resolve common errors, optimize configurations, and improve performance through adjustments in topic settings, batch compression, and custom partitioning strategies. The author emphasizes the necessity of adapting Kafka sink/source configurations based on varying data loads and business requirements.
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward The document discusses performance troubleshooting in Apache Flink, highlighting the common bottlenecks in processing records, checkpointing, and recovery. It provides a step-by-step approach to identify and analyze performance issues, including the significance of analyzing machine metrics and leveraging profiling tools. Recommendations for improving checkpoint performance and recovery times are also outlined, emphasizing the importance of addressing backpressure and resource overload.
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentFlink Forward The document discusses the Apache Flink Kubernetes Operator and its application in production environments, emphasizing its benefits such as self-management of services and upgrades. It outlines essential topics including the operator maturity model, enhancements in security and stability, and a live demo showcasing the operator's functionalities. The presentation concluded with a Q&A session addressing common challenges in deploying Flink in Kubernetes.
Flink SQL on Pulsar made easy
Flink SQL on Pulsar made easyFlink Forward Flink Forward San Francisco 2022.
Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects:
1. Two different modes of use Pulsar as a metadata store
2. Data format transformation and management
3. SQL semantics support within Pulsar context
by
Sijie Guo & Neng Lu
Dynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data AlertsFlink Forward The document discusses the implementation of a dynamic rule-based real-time market data alert system using Apache Flink. It outlines the alert system's architecture, including event detection, rule management, and various alert delivery methods. Key lessons learned emphasize the importance of simplifying input data and understanding SQL-like query limitations in complex event processing.
Processing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial ServicesFlink Forward The document discusses the challenges of achieving in-order processing in stream processing frameworks like Flink and Beam, particularly in financial services which require precise event ordering. It explains the concept of 'sequenced streams' and outlines the differences between traditional event processing and the proposed sequencer architecture to ensure global event order. It calls for improvements in stream processing frameworks to explicitly maintain order from the source during processing, highlighting the importance of preserving order for applications in the financial industry.
Welcome to the Flink Community!
Welcome to the Flink Community!Flink Forward The document provides an overview of the Flink community, highlighting its history, significant milestones, and key contributors such as Caito Scherr from Ververica. It outlines how users can get involved with Flink, including participation in events, joining mailing lists, and contributing to development. Additionally, it emphasizes community resources and support channels available for both new and experienced Flink users.
Extending Flink SQL for stream processing use cases
Extending Flink SQL for stream processing use casesFlink Forward 1. For streaming data, Flink SQL uses STREAMs for append-only queries and CHANGELOGs for upsert queries instead of tables.
2. Stateless queries on streaming data, such as projections and filters, result in new STREAMs or CHANGELOGs.
3. Stateful queries, such as aggregations, produce STREAMs or CHANGELOGs depending on whether they are windowed or not. Join queries between streaming sources also result in STREAM outputs.
The top 3 challenges running multi-tenant Flink at scale
The top 3 challenges running multi-tenant Flink at scaleFlink Forward The document discusses the top three challenges of running multi-tenant Flink at scale, including infrastructure resource management, observability, and authentication with external systems. It emphasizes the importance of optimized configurations, centralized management for cost efficiency, and effective error classification to improve the user experience. Moreover, it highlights the need for secure handling of sensitive information while maintaining comprehensive operational insights.
Using Queryable State for Fun and Profit
Using Queryable State for Fun and ProfitFlink Forward The presentation discusses the use of queryable state in digital observability, focusing on New Relic's Lookout feature that provides insights through golden metrics. It presents strategies for reducing query times and compares cost estimates between traditional and modern solutions using Redis and Flink. Key learnings emphasize the importance of queryable state for data management and decision-making.
Large Scale Real Time Fraudulent Web Behavior Detection
Large Scale Real Time Fraudulent Web Behavior DetectionFlink Forward Flink Forward San Francisco 2022.
Neuro-ID analyzes web behavior at a large scale to determine visitors' intent on web pages, specifically in the online lending industry. When users interact with an online loan application, our software analyzes their behavior to determine if the applicant may be potentially fraudulent. Lenders can then request various scores describing the applicant's intentions in real-time to use to make decisions during the application flow. Flink gives our product the ability to observe behavior in a stateful manner. As an applicant interacts with an online loan application, a Flink application is used to compare earlier actions to later actions. This processing in Flink can determine the applicant's intent throughout the process of the application.
by
Jeff Niemann & Randy Hanak
Building Reliable Lakehouses with Apache Flink and Delta Lake
Building Reliable Lakehouses with Apache Flink and Delta LakeFlink Forward The document outlines the principles and architecture of building reliable data lakes using Apache Flink and Delta Lake, emphasizing the advantages of lakehouses over traditional data warehouses and data lakes. It highlights key features such as ACID transactions, scalable metadata, and unified batch/streaming support, along with the evolving ecosystem of Delta connectors. The authors, Scott Sandre and Denny Lee from Databricks, also discuss the importance of community engagement in the growth and development of Delta Lake.
Near real-time statistical modeling and anomaly detection using Flink!
Near real-time statistical modeling and anomaly detection using Flink!Flink Forward The document discusses a near real-time statistical modeling and anomaly detection system using Flink, focusing on the architecture and processes for data ingestion, alert generation, and evaluation. It highlights the transition from static to dynamic thresholds for improved anomaly detection and outlines the advantages of streaming statistical computations. The conclusion emphasizes the system's robustness, scalability, and future improvements including optimizations and machine learning applications.
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward The document explains how to build a streaming lakehouse using Apache Flink and Hudi for real-time analytics and data ingestion challenges. It discusses key features of the Hudi framework, including fast stream ingestion, efficient upserts/deletes, and support for ACID transactions, culminating in composite data management strategies. Additionally, it mentions various components such as the metaserver and lake cache aimed at optimizing data layout and ingestion processes.
Ad
Recently uploaded (20)
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and SwecoSafe Software Managing airport and airspace data is no small task, especially when you’re expected to deliver it in AIXM format without spending a fortune on specialized tools. But what if there was a smarter, more affordable way?
Join us for a behind-the-scenes look at how Sweco partnered with Swedavia, the Swedish airport operator, to solve this challenge using FME and Esri.
Learn how they built automated workflows to manage periodic updates, merge airspace data, and support data extracts – all while meeting strict government reporting requirements to the Civil Aviation Administration of Sweden.
Even better? Swedavia built custom services and applications that use the FME Flow REST API to trigger jobs and retrieve results – streamlining tasks like securing the quality of new surveyor data, creating permdelta and baseline representations in the AIS schema, and generating AIXM extracts from their AIS data.
To conclude, FME expert Dean Hintz will walk through a GeoBorders reading workflow and highlight recent enhancements to FME’s AIXM (Aeronautical Information Exchange Model) processing and interpretation capabilities.
Discover how airports like Swedavia are harnessing the power of FME to simplify aviation data management, and how you can too.
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdfPriyanka Aash Coordinated Disclosure for ML - What's Different and What's the Same
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft developmentshyamraj55 CurieTech AI in Action – Accelerate MuleSoft Development
Overview:
This presentation demonstrates how CurieTech AI’s purpose-built agents empower MuleSoft developers to create integration workflows faster, more accurately, and with less manual effort
linkedin.com
+12
curietech.ai
+12
meetups.mulesoft.com
+12
.
Key Highlights:
Dedicated AI agents for every stage: Coding, Testing (MUnit), Documentation, Code Review, and Migration
curietech.ai
+7
curietech.ai
+7
medium.com
+7
DataWeave automation: Generate mappings from tables or samples—95%+ complete within minutes
linkedin.com
+7
curietech.ai
+7
medium.com
+7
Integration flow generation: Auto-create Mule flows based on specifications—speeds up boilerplate development
curietech.ai
+1
medium.com
+1
Efficient code reviews: Gain intelligent feedback on flows, patterns, and error handling
youtube.com
+8
curietech.ai
+8
curietech.ai
+8
Test & documentation automation: Auto-generate MUnit test cases, sample data, and detailed docs from code
curietech.ai
+5
curietech.ai
+5
medium.com
+5
Why Now?
Achieve 10× productivity gains, slashing development time from hours to minutes
curietech.ai
+3
curietech.ai
+3
medium.com
+3
Maintain high accuracy with code quality matching or exceeding manual efforts
curietech.ai
+2
curietech.ai
+2
curietech.ai
+2
Ideal for developers, architects, and teams wanting to scale MuleSoft projects with AI efficiency
Conclusion:
CurieTech AI transforms MuleSoft development into an AI-accelerated workflow—letting you focus on innovation, not repetition.
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdfEarley Information Science As AI reshapes expectations in retail and B2B commerce, organizations are recognizing a critical reality: meaningful AI outcomes depend on well-structured, adaptable infrastructure. In this session, Seth Earley is joined by Phil Ryan - AI strategist, search technologist, and founder of Glass Leopard Technologies - for a candid conversation on what it truly means to engineer systems for scale, agility, and intelligence.
Phil draws on more than two decades of experience leading search and AI initiatives for enterprise organizations. Together, he and Seth explore the challenges businesses face when legacy architectures limit personalization, agility, and real-time decisioning - and what needs to change to support agentic technologies and next-best-action capabilities.
Key themes from the webinar include:
Composability as a prerequisite for AI - Why modular, loosely coupled systems are essential for adapting to rapid innovation and evolving business needs
Search and relevance as foundational to AI - How techniques honed-in enterprise search have laid the groundwork for more responsive and intelligent customer experiences
From MDM and CDP to agentic systems - How data platforms are evolving to support richer customer context and dynamic orchestration
Engineering for business alignment - Why successful AI programs require architectural decisions grounded in measurable outcomes
The conversation is practical and forward-looking, connecting deep technical understanding with real-world business needs. Whether you’re modernizing your commerce stack or exploring how AI can enhance product discovery, personalization, or customer journeys, this session provides a clear-eyed view of the capabilities, constraints, and priorities that matter most.
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes ProbableSaikat Basu Imagine combining the "brains" of Artificial Intelligence (AI) with the "super muscles" of Quantum Computing. That's Quantum AI!
It's a new field that uses the mind-bending rules of quantum physics to make AI even more powerful.
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical UniversesSaikat Basu Embark on a cosmic journey exploring the intersection of quantum
computing, consciousness, and ancient wisdom. Together we'll uncover the
recursive patterns that bind our reality.
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
2025_06_18 - OpenMetadata Community Meeting.pdf
2025_06_18 - OpenMetadata Community Meeting.pdfOpenMetadata The community meetup was held Wednesday June 18, 2025 @ 9:00 AM PST.
Catch the next OpenMetadata Community Meetup @ https://www.meetup.com/openmetadata-meetup-group/
In this month's OpenMetadata Community Meetup, "Enforcing Quality & SLAs with OpenMetadata Data Contracts," we covered data contracts, why they matter, and how to implement them in OpenMetadata to increase the quality of your data assets!
Agenda Highlights:
👋 Introducing Data Contracts: An agreement between data producers and consumers
📝 Data Contracts key components: Understanding a contract and its purpose
🧑🎨 Writing your first contract: How to create your own contracts in OpenMetadata
🦾 An OpenMetadata MCP Server update!
➕ And More!
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's SeatSafe Software Finding a specific car online can be a time-consuming task, especially when checking multiple dealer websites. A few years ago, I faced this exact problem while searching for a particular vehicle in New Zealand. The local classified platform, Trade Me (similar to eBay), wasn’t yielding any results, so I expanded my search to second-hand dealer sites—only to realise that periodically checking each one was going to be tedious. That’s when I noticed something interesting: many of these websites used the same platform to manage their inventories. Recognising this, I reverse-engineered the platform’s structure and built an FME workspace that automated the search process for me. By integrating API calls and setting up periodic checks, I received real-time email alerts when matching cars were listed. In this presentation, I’ll walk through how I used FME to save hours of manual searching by creating a custom car-finding automation system. While FME can’t buy a car for you—yet—it can certainly help you find the one you’re after!
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core InnovationsIBM penPOWER offers a fully open, royalty-free CPU architecture for custom chip design.
It enables both lightweight FPGA cores (like Microwatt) and high-performance processors (like POWER10).
Developers have full access to source code, specs, and tools for end-to-end chip creation.
It supports AI, HPC, cloud, and embedded workloads with proven performance.
Backed by a global community, it fosters innovation, education, and collaboration.
Techniques for Automatic Device Identification and Network Assignment.pdf
Techniques for Automatic Device Identification and Network Assignment.pdfPriyanka Aash Techniques for Automatic Device Identification and Network Assignment
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/mpu-a-transformative-solution-for-next-gen-ai-at-the-edge-a-presentation-from-fotonation/
Petronel Bigioi, CEO of FotoNation, presents the “MPU+: A Transformative Solution for Next-Gen AI at the Edge” tutorial at the May 2025 Embedded Vision Summit.
In this talk, Bigioi introduces MPU+, a novel programmable, customizable low-power platform for real-time, localized intelligence at the edge. The platform includes an AI-augmented image signal processor that enables leading image and video quality.
In addition, it integrates ultra-low-power object and motion detection capabilities to enable always-on computer vision. A programmable neural processor provides flexibility to efficiently implement new neural networks. And additional specialized engines facilitate image stabilization and audio enhancements.
Lessons Learned from Developing Secure AI Workflows.pdf
Lessons Learned from Developing Secure AI Workflows.pdfPriyanka Aash Lessons Learned from Developing Secure AI Workflows
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...Josef Weingand IBM LTO10
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance
Changelog Stream Processing with Apache Flink
- 1. Changelog Stream Processing with Apache Flink Timo Walther @twalthr – Flink Forward 2022 2022-08-03
- 2. About me Open source ● Long-term committer since 2014 (before ASF) ● Member of the project management committee (PMC) ● Top 5 contributor (commits), top 1 contributor (additions) ● Among core architects of Flink SQL Career ● Early software engineer @ DataArtisans ● SDK team @ DataArtisans/Ververica (acquisition by Alibaba) ● SQL team lead @ Ververica ● Co-founder @ Immerok 2
- 3. What is Apache Flink? 3
- 4. Building Blocks for Stream Processing 4 Time ● Synchronize ● Progress ● Wait ● Timeout ● Fast-forward ● Replay State ● Store ● Buffer ● Cache ● Model ● Grow ● Expire Streams ● Pipeline ● Distribute ● Join ● Enrich ● Control ● Replay Snapshots ● Backup ● Version ● Fork ● A/B test ● Time-travel ● Restore
- 5. What is Apache Flink used for? 5 Transactions Logs IoT Interactions Events … Analytics Event-driven Applications Data Integration ETL Messaging Systems Files Databases Key/Value Stores Applications Messaging Systems Files Databases Key/Value Stores
- 7. API Stack 7 Dataflow Runtime Low-Level Stream Operator API Optimizer / Planner Table / SQL API DataStream API Stateful Functions
- 8. DataStream API 8 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setRuntimeMode(STREAMING); DataStream<Integer> stream = env.fromElements(1, 2, 3); stream.executeAndCollect().forEachRemaining(System.out::println); Properties ● Exposes the building blocks for stream processing ● Arbitrary operator topologies using map(), process(), connect(), ... ● Business logic is written in user-defined functions ● Arbitrary user-defined record types flow in-between ● Conceptually always an append-only / insert-only log! 1 2 3 Output
- 9. Table / SQL API 9 TableEnvironment env = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); // Programmatic Table table = env.fromValues(row(1), row(2), row(3)); // SQL Table table = env.sqlQuery("SELECT * FROM (VALUES (1), (2), (3))"); table.execute().print(); Properties ● Abstracts the building blocks for stream processing ● Operator topology is determined by planner ● Business logic is declared in SQL and/or Table API ● Internal record types flow, Flink’s Row type is exposed in Table API ● Conceptually a table, but a changelog under the hood! +----+-------------+ | op | f0 | +----+-------------+ | +I | 1 | | +I | 2 | | +I | 3 | Output
- 10. DataStream API ↔Table / SQL API 10 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); // Stream -> Table DataStream<?> inStream1 = ... Table appendOnlyTable = tableEnv.fromDataStream(inStream1) DataStream<Row> inStream2 = ... Table anyTable = tableEnv.fromChangelogStream(inStream2) // Table -> Stream DataStream<T> appendOnlyStream = tableEnv.toDataStream(insertOnlyTable, T.class) DataStream<Row> changelogStream = tableEnv.toChangelogStream(anyTable) Mix and match APIs!
- 11. Changelog Stream Processing 11 Change is the law of life and those who look only to the past or present are certain to miss the future. John F. Kennedy
- 12. Data Processing is a Stream of Changes 12 ● Business data is always a stream: bounded or unbounded ● Every record is a changelog entry: insertion as the default ● Batch processing is just a special case in the runtime now past future start end of stream bounded stream unbounded stream unbounded stream
- 13. How do I Work with Streams in Flink SQL? 13 ● You don’t. You work with dynamic tables! ● A concept similar to materialized views CREATE TABLE Revenue (name STRING, total INT) WITH (…) INSERT INTO Revenue SELECT name, SUM(amount) FROM Transactions GROUP BY name CREATE TABLE Transactions (name STRING, amount INT) WITH (…) name amount Alice 56 Bob 10 Alice 89 name total Alice 145 Bob 10 So, is Flink SQL a database? No, bring your own data and systems!
- 14. Stream-Table Duality - Basics 14 ● A stream is the changelog of a dynamic table ● Sources, operators, and sinks work on changelogs under the hood ● Each component declares the kind of changes it consumes/produces only +I Appending/Insert-only contains -… Updating contains -U Retracting never –U but +U Upserting Short name Long name +I Insertion Default for scans + output of bounded results. -U Update Before Retracts a previously emitted result. +U Update After Updates a previously emitted result. Requires a primary key if -U is omitted for idempotent updates. -D Delete Removes the last result.
- 15. Stream-Table Duality - Example 15 An applied changelog becomes a real (materialized) table. name amount Alice 56 Bob 10 Alice 89 name total Alice 56 Bob 10 changelog +I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56] 145 materialization CREATE TABLE Revenue (name STRING, total INT) WITH (…) INSERT INTO Revenue SELECT name, SUM(amount) FROM Transactions GROUP BY name CREATE TABLE Transactions (name STRING, amount INT) WITH (…)
- 16. Stream-Table Duality - Example 16 An applied changelog becomes a real (materialized) table. name amount Alice 56 Bob 10 Alice 89 name total Alice 56 Bob 10 +I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56] 145 materialization CREATE TABLE Revenue (PRIMARY KEY(name) …) WITH (…) INSERT INTO Revenue SELECT name, SUM(amount) FROM Transactions GROUP BY name CREATE TABLE Transactions (name STRING, amount INT) WITH (…) Save ~50% of traffic if downstream system supports upserting!
- 17. Stream-Table Duality - Propagation 17 ● Sources declares set of emitted changes i.e. changelog mode ● Optimizer tracks changelog mode and primary key through pipeline ● Sink declares changes it can digest CREATE TABLE … … WITH ('connector'='filesystem') … WITH ('connector'='kafka') … WITH ('connector'='kafka-upsert') … WITH ('connector'='jdbc') … WITH ('connector'='kafka', 'format' = 'debezium-json') +I +I +I -D +I -U +U -D +I (for sources)
- 18. Retract vs. Upsert 18 Retract ● No primary key requirements ● Works for almost every external system ● Supports duplicate rows ● In distributed system often unavoidable à most flexible changelog mode à default mode Upsert ● Traffic + computation optimization ● In-place updates (idempotency) SELECT c, COUNT(*) FROM ( SELECT COUNT(*) AS c FROM T GROUP BY user ) GROUP BY c Count 1 Subtask 1 Count 2 Subtask 1 Subtask 2 +U[1] +U[2] +I[…] 1=>1 2=>1 Subtask 2 +I[…]
- 19. Changelog Insights – Append-only 19 CREATE TABLE Transaction (tid BIGINT, amount INT); CREATE TABLE Payment (tid BIGINT, method STRING); CREATE TABLE Result (tid BIGINT, …); // accepts all changes INSERT INTO Result SELECT * FROM Transactions T JOIN Payments P ON T.tid = P.tid; Sink(table=[Result], changelogMode=[NONE]) +- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I]) :- Exchange(changelogMode=[I]) : +- TableSourceScan(table=[[Transaction]], changelogMode=[I]) +- Exchange(changelogMode=[I]) +- TableSourceScan(table=[[Payment]], changelogMode=[I])
- 20. Changelog Insights – Updating 20 CREATE TABLE Transaction (tid BIGINT, amount INT); CREATE TABLE Payment (tid BIGINT, method STRING); CREATE TABLE Result (tid BIGINT, …); INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid; Sink(table=[Result], changelogMode=[NONE]) +- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D]) :- Exchange(changelogMode=[I]) : +- TableSourceScan(table=[[Transaction]], changelogMode=[I]) +- Exchange(changelogMode=[I]) +- TableSourceScan(table=[[Payment]], changelogMode=[I])
- 21. Changelog Insights – Updating with PK 21 CREATE TABLE Transaction (tid BIGINT, amount INT); CREATE TABLE Payment (tid BIGINT, method STRING); CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED); INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid; Sink(table=[Result], changelogMode=[NONE], upsertMaterialize=[true]) +- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D]) :- Exchange(changelogMode=[I]) : +- TableSourceScan(table=[[Transaction]], changelogMode=[I]) +- Exchange(changelogMode=[I]) +- TableSourceScan(table=[[Payment]], changelogMode=[I])
- 22. Changelog Insights – Updating with PK 22 CREATE TABLE Transaction (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED); CREATE TABLE Payment (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED); CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED); INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid; Sink(table=[Result], changelogMode=[NONE]) +- Join(leftInputSpec=[UniqueKey], rightInputSpec=[UniqueKey], changelogMode=[I,UA,D]) :- Exchange(changelogMode=[I]) : +- TableSourceScan(table=[[Transaction]], changelogMode=[I]) +- Exchange(changelogMode=[I]) +- TableSourceScan(table=[[Payment]], changelogMode=[I])
- 23. Mode Transitions 23 Append-only Retracting Updating through operation if operator/sink requires it ChangelogNormalize if sink requires it UpsertMaterialize
- 24. Mode Transitions – Characteristics 24 Append-only ● Event-time column backed by watermarks ● Highly state efficient due to notion of completeness ● Usually no event-time column ● State usage needs to be kept in mind ● Pure materialized view maintenance Retracting Updating aka "TABLE" aka "STREAM" aka ?
- 25. Demo 29
- 26. Summary TLDR ● Flink's SQL engine is a powerful changelog processor ● Flexible tool for integrating systems with different semantics There is more… ● CDC connector ecosystem à 2.6k Github stars https://flink-packages.org/packages/cdc-connectors ● Table Store unified storage engine for dynamic tables à native changelog support à deep integration into Flink SQL "like a DB" https://flink.apache.org/news/2022/05/11/release-table-store-0.1.0.html 30