Apache Flink's Table & SQL API - unified APIs for batch and stream processing
3 likes946 views
The document discusses Apache Flink's unified API for batch and stream processing, emphasizing the importance of a relational API to simplify stream processing. It presents the Table API and SQL as tools for querying data in a consistent manner across both batch and streaming data. Additionally, it highlights the functionality of dynamic tables and the ongoing contributions for enhancing Flink's capabilities.
1 of 31
Downloaded 16 times
![Table API Example
9
val sensorData: DataStream[(String, Long, Double)] = ???
// convert DataSet into Table
val sensorTable: Table = sensorData
.toTable(tableEnv, 'location, ’time, 'tempF)
// define query on Table
val avgTempCTable: Table = sensorTable
.window(Tumble over 1.day on 'rowtime as 'w)
.groupBy('location, ’w)
.select('w.start as 'day, 'location,
(('tempF.avg - 32) * 0.556) as 'avgTempC)
.where('location like "room%")](https://image.slidesharecdn.com/timo-walther-tableapiandsql-170306091814/85/Apache-Flink-s-Table-SQL-API-unified-APIs-for-batch-and-stream-processing-9-320.jpg)
![SQL Example
10
val sensorData: DataStream[(String, Long, Double)] = ???
// register DataStream
tableEnv.registerDataStream(
"sensorData", sensorData, 'location, ’time, 'tempF)
// query registered Table
val avgTempCTable: Table = tableEnv
.sql("""
SELECT FLOOR(rowtime() TO DAY) AS day, location,
AVG((tempF - 32) * 0.556) AS avgTempC
FROM sensorData
WHERE location LIKE 'room%'
GROUP BY location, FLOOR(rowtime() TO DAY) """)](https://image.slidesharecdn.com/timo-walther-tableapiandsql-170306091814/85/Apache-Flink-s-Table-SQL-API-unified-APIs-for-batch-and-stream-processing-10-320.jpg)
![Architecture
2 APIs [SQL, Table API]
*
2 backends [DataStream, DataSet]
=
4 different translation paths?
11](https://image.slidesharecdn.com/timo-walther-tableapiandsql-170306091814/85/Apache-Flink-s-Table-SQL-API-unified-APIs-for-batch-and-stream-processing-11-320.jpg)
![Querying Dynamic Tables
§ Dynamic tables change over time
• A[t]: Table A at time t
§ Dynamic tables are queried with regular SQL
• Result of a query changes as input table changes
• q(A[t]): Evaluate query q on table A at time t
§ Query result is continuously updated as t progresses
• Similar to maintaining a materialized view
• t is current event time
22](https://image.slidesharecdn.com/timo-walther-tableapiandsql-170306091814/85/Apache-Flink-s-Table-SQL-API-unified-APIs-for-batch-and-stream-processing-22-320.jpg)
Ad
Recommended
Stream Analytics with SQL on Apache Flink
Stream Analytics with SQL on Apache FlinkFabian Hueske Fabian Hueske's presentation at the Strata Data Conference discusses stream analytics with SQL on Apache Flink, emphasizing its capabilities for stateful stream processing and real-time data integration. The session covers the advantages of using Flink's relational APIs for batch and streaming data, including dynamic tables and continuous queries. Additionally, Hueske highlights the robust features and production use cases of Flink in major companies like Alibaba and Uber.
Stephan Ewen - Stream Processing as a Foundational Paradigm and Apache Flink'...
Stephan Ewen - Stream Processing as a Foundational Paradigm and Apache Flink'...Ververica The document discusses Apache Flink's approach to stream processing, emphasizing its capability for continuous data handling and elimination of trade-offs commonly faced in stateful applications. It outlines key features such as exactly-once semantics, event time processing, and a fluent API for building stateful applications while maintaining high throughput and low latency. Additionally, it highlights Flink's architecture and performance metrics, showcasing its daily processing capabilities across various use cases.
Stream Analytics with SQL on Apache Flink - Fabian Hueske
Stream Analytics with SQL on Apache Flink - Fabian HueskeEvention The document discusses stream analytics using SQL on Apache Flink, highlighting its capabilities for real-time data processing with low latency and high throughput. It emphasizes the need for SQL support in stream processing to make it more accessible, while addressing challenges such as state management and dynamic table queries. Flink is actively developing features like dynamic tables and additional window operators to enhance its SQL and table API functionality.
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...Flink Forward This document discusses Flink's Table and SQL APIs, which provide a unified way to write batch and streaming queries. It motivates the need for a relational API by explaining that while Flink's DataStream API is powerful, it requires more technical skills. The Table and SQL APIs allow users to focus on business logic by writing declarative queries. It describes how the APIs work, including translating queries to logical and execution plans and supporting batch, streaming and windowed queries. Finally, it outlines the current capabilities and opportunities for contributors to help expand Flink's relational features.
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...Flink Forward Pravega is a new storage abstraction designed for continuous and infinite data, supporting low-latency appends and high-throughput reads. It enables dynamic scaling of stream storage and processing, coordinating with data arrival rates to achieve exactly once processing semantics across multiple applications. Unlike traditional storage systems like Kafka, Pravega ensures durability, order, and efficient parallelism while facilitating the integration with stream processing frameworks such as Apache Flink.
Fabian Hueske - Stream Analytics with SQL on Apache Flink
Fabian Hueske - Stream Analytics with SQL on Apache FlinkVerverica Fabian Hueske presented on stream analytics using SQL on Apache Flink. Flink provides a scalable platform for stream processing that is fast, accurate, and reliable. Its relational APIs allow querying both batch and streaming data using standard SQL or a LINQ-style Table API. Queries on streaming data produce continuously updating results. Windows can be used to compute aggregates over tumbling time intervals. The dynamic tables representing streaming data can be converted to output streams encoding updates as insertions and deletions. While not all queries can be supported, techniques like limiting state size allow bounding computational resources. Use cases like continuous ETL, dashboards, and event-driven architectures were discussed.
Streaming SQL to unify batch and stream processing: Theory and practice with ...
Streaming SQL to unify batch and stream processing: Theory and practice with ...Fabian Hueske The document discusses the use of Apache Flink for unifying batch and stream processing, emphasizing its scalability and fault tolerance. It highlights the SQL capabilities of Flink, allowing for consistent querying of both bounded and unbounded data streams, which enhances usability and portability. Additionally, it outlines Uber's implementation of Flink, demonstrating its efficiency in real-time data processing and analytics for various applications.
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...Flink Forward The document discusses improvements in Apache Flink for managing large state in streaming programs, emphasizing stateful and stateless processing strategies, and the nuances of internal vs. external state management. It also details the mechanics of asynchronous checkpointing and incremental snapshotting using RocksDB, highlighting the need for efficient state handling to maintain performance during event processing. Overall, it outlines a systematic approach to implement incremental checkpoints for better scalability and recovery in stream processing applications.
Why and how to leverage the power and simplicity of SQL on Apache Flink
Why and how to leverage the power and simplicity of SQL on Apache FlinkFabian Hueske The document presents the advantages and functionalities of Apache Flink, focusing on SQL's integration for stream and batch processing. It highlights Flink's ability to handle real-time data streams and the importance of unifying streaming and batch query capabilities using ANSI SQL syntax. The presentation discusses use cases, ongoing developments within the Flink community, and invites feedback on new features like the SQL query service.
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...Flink Forward The document discusses the implementation of a multi-SaaS management platform by BetterCloud using stream processing technologies like Flink and Kafka to handle hundreds of millions of messages daily. It highlights foundational learnings about event stream processing and rules engines, along with the technical architecture involving control and event streams for operational intelligence and alert detection. Additionally, it addresses challenges like historical data processing and potential areas for improvement in performance and state management.
Optimizing Your Cloud Applications in RightScale
Optimizing Your Cloud Applications in RightScaleRightScale This document discusses optimizing cloud applications using RightScale. It covers monitoring applications using tools like Collectd and New Relic RPM. It also discusses optimizing database performance on cloud infrastructure by scaling instances vertically and horizontally, using the right indices, and ensuring the working set fits in memory. The document provides an agenda and overview of these optimization techniques.
Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream ...
Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream ...Ververica The document discusses a system for real-time stream processing using Flink to join data streams of scientific accounts and publications, highlighting challenges like fault tolerance and accurate joining of data from distributed sources. It outlines a prototype implementation that facilitates the integration of user accounts with their respective publications while addressing issues related to change data capture and data synchronization. The document emphasizes the importance of maintaining up-to-date and accurate join results in a scalable infrastructure supporting millions of publications.
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...Flink Forward Uber Engineering discusses the challenges and solutions for near real-time analytics and event processing, emphasizing the use of Apache Kafka and Flink for data management and analytics. The presentation outlines various use cases such as growth metrics, fraud detection, and online analytical processing, all powered by sophisticated SQL and streaming technologies. Looking ahead, Uber aims to explore further advancements in real-time machine learning and auto-scaling within their data ecosystem.
Akka Streams
Akka StreamsDiego Pacheco This document discusses Akka Streams, which is a library for building reactive stream processing applications. It defines key concepts like streams, processing stages, back pressure, and flow control. It also covers topics like stream composition, materialization, using actors in streams, and includes code examples. The document is presented by Diego Pacheco, a principal software architect at Akka Streams.
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward The document discusses the use of Apache Flink for managing high-frequency time series data collected from wearable patches, emphasizing its capability to create a predictive health model. It explains Flink's state management features, particularly managed keyed state, and how to access queryable state outside the Flink runtime. Various use cases illustrate the transition from batch processing to stream processing within the healthcare data management context.
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large ScaleVerverica The document discusses the implementation and challenges of running Apache Flink at large scales, highlighting use cases such as stream ingestion and event-driven applications. Key issues addressed include dependency conflicts, checkpointing, state management, and performance optimization strategies. The talk emphasizes Flink's evolving architecture and the importance of understanding its underlying mechanisms for robust large-scale application deployment.
From Apache Flink® 1.3 to 1.4
From Apache Flink® 1.3 to 1.4Till Rohrmann The document provides an overview of the evolution and features of Apache Flink, highlighting updates from version 1.3 to the upcoming 1.5+, including improvements in APIs, state management, and checkpointing mechanisms. It emphasizes the platform's capabilities for event-driven I/O, integration with Kafka for exactly-once processing, and operational robustness. Finally, it invites readers to explore additional resources for deeper insights into Flink's development and innovations.
Online index rebuild automation
Online index rebuild automationCarlos Sierra This document discusses the automation of online index rebuilding in a multitenant database environment, aiming to optimize performance and reduce wasted space in indexes. A case study highlights the implementation of a custom infrastructure across numerous databases and the importance of index management strategies tailored to specific query patterns. The automation process leads to significant space savings and performance improvements, particularly for short-latency queries, while emphasizing the need for continuous monitoring and adjustments in the rebuilding process.
Stream processing - Apache flink
Stream processing - Apache flinkRenato Guimaraes Apache Flink is a distributed processing engine designed for stateful computations over both unbounded and bounded data streams, featuring low-latency, high throughput, and fault tolerance. It supports various applications such as fraud detection, real-time analytics, and data pipelines, employing several APIs like SQL, Table API, and DataStream API for stream processing. Flink's architecture includes job managers and task managers for efficient resource management and offers mechanisms like savepoints for fault tolerance and state management.
Dive into Spark Streaming
Dive into Spark StreamingGerard Maas The document discusses Spark Streaming and its execution model. It describes how Spark Streaming receives data streams, divides them into micro-batches, and processes the micro-batches using Spark. It also covers transformations and actions that can be performed on DStreams, window operations, and deployment options for Spark Streaming including local, standalone, and cluster modes.
The Power of the Log
The Power of the LogBen Stopford The document discusses the use of log-structured merge (LSM) trees in modern databases, emphasizing their efficiency in handling writes by utilizing append-only data structures and sequential operations. It also highlights techniques to optimize reads, such as caching and bloom filters, and reframes database problems in a log-centric way for better system performance. Key examples and comparisons with existing technologies are provided to illustrate the advantages of adopting a log-centric approach.
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...Lightbend This webinar discusses building streaming and fast data applications with technologies like Spark, Mesos, Akka, Cassandra and Kafka. It covers how microservices and fast data architectures are converging due to similar design problems and data becoming the dominant problem. The webinar also introduces Lightbend's Fast Data Platform for building streaming data systems and microservices with best practices, sample applications and machine learning-based monitoring and management.
Akka-chan's Survival Guide for the Streaming World
Akka-chan's Survival Guide for the Streaming WorldKonrad Malawski The document outlines the evolution and significance of reactive streams, particularly in the context of Akka and JDK 9, highlighting their role in building concurrent, distributed applications. It discusses the development timelines, key features, and comparisons between various streaming technologies and libraries. The author emphasizes the importance of reactive streams for handling back-pressure and asynchronous processing in modern data architectures.
[OracleCode SF] In memory analytics with apache spark and hazelcast
[OracleCode SF] In memory analytics with apache spark and hazelcastViktor Gamov The document discusses the integration of Apache Spark with Hazelcast's in-memory data grid, highlighting the speed advantages of Spark and its fault-tolerance with resilient distributed datasets (RDDs). It provides code examples for configuration and uses cases, emphasizing parallel processing and operational benefits. Additionally, it mentions limitations regarding data updates while reading from Spark, suggesting potential issues like cursor inaccuracies.
Stream all the things
Stream all the thingsDean Wampler This document provides an overview of streaming architectures and compares different streaming engines. It describes a common streaming architecture that ingests data from sources like logs and sockets into Kafka for storage and organization. It then discusses several streaming engines like Apache Beam, Apache Flink, and Akka Streams that can process the data from Kafka. Beam provides the most sophisticated streaming semantics to handle scenarios like late data arrival, while Flink offers low latency processing at large scales and can run Beam data flows. Akka Streams is best suited for complex event processing on individual events.
Streamsets and spark
Streamsets and sparkHari Shreedharan StreamSets can process data using Apache Spark in three ways:
1) The Spark Evaluator stage allows user-provided Spark code to run on each batch of records in a pipeline and return results or errors.
2) A Cluster Pipeline can leverage Apache Spark's Direct Kafka DStream to partition data from Kafka across worker pipelines on a cluster.
3) A Spark Executor can kick off a Spark application when an event is received, allowing tasks like model updating to run on streaming data using Spark.
Apache Beam
Apache Beam Adil Oulghard Le document présente Apache Beam, un modèle de programmation unifié pour le traitement de données infinies et non ordonnées, permettant d'écrire des pipelines portables en Java et Python. Il décrit les concepts clés tels que le fenêtrage, les transformations et les exécuteurs, ainsi que des exemples pratiques comme le pipeline de comptage de mots. En conclusion, Apache Beam offre flexibilité et modularité pour gérer divers scénarios de traitement de données, y compris le streaming et le traitement batch.
Apache Spark and Oracle Stream Analytics
Apache Spark and Oracle Stream AnalyticsPrabhu Thukkaram This document discusses Oracle Stream Analytics, which provides complex event processing capabilities for Apache Spark Streaming. It leverages Oracle's Continuous Query Engine for event-by-event processing and Apache Spark for distributed computing and fault tolerance. Key features highlighted include stateful and continuous query processing, flexible temporal windows, pattern detection, spatial analysis, and integrated business rules. It is described as reducing application development time by handling state management and fault tolerance, while also scaling linearly with Spark and providing automatic recovery from failures.
Kafka & Couchbase Integration Patterns
Kafka & Couchbase Integration PatternsManuel Hurtado The document provides an overview of Kafka & Couchbase integration patterns. It introduces Couchbase and Kafka, describes how Kafka Connect enables real-time data pipelines between data systems, and how the Couchbase Kafka connector integrates Couchbase with Kafka pipelines. Use cases for the connector include using Couchbase as a data source or sink within Kafka streams. The document concludes with demos of Couchbase as a source and sink using the connector.
Kudu Forrester Webinar
Kudu Forrester WebinarCloudera, Inc. The document covers a Cloudera webinar series focused on Apache Kudu and its capabilities for real-time data analytics across various business use cases. It addresses the complexities of traditional lambda architectures, the technical architecture of Kudu, and the importance of real-time data pipelines for enhancing operational efficiencies and decision-making. The series also includes insights from Forrester on market trends and strategies for deploying scalable real-time data platforms.
More Related Content
What's hot (11)
Why and how to leverage the power and simplicity of SQL on Apache Flink
Why and how to leverage the power and simplicity of SQL on Apache FlinkFabian Hueske The document presents the advantages and functionalities of Apache Flink, focusing on SQL's integration for stream and batch processing. It highlights Flink's ability to handle real-time data streams and the importance of unifying streaming and batch query capabilities using ANSI SQL syntax. The presentation discusses use cases, ongoing developments within the Flink community, and invites feedback on new features like the SQL query service.
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...Flink Forward The document discusses the implementation of a multi-SaaS management platform by BetterCloud using stream processing technologies like Flink and Kafka to handle hundreds of millions of messages daily. It highlights foundational learnings about event stream processing and rules engines, along with the technical architecture involving control and event streams for operational intelligence and alert detection. Additionally, it addresses challenges like historical data processing and potential areas for improvement in performance and state management.
Optimizing Your Cloud Applications in RightScale
Optimizing Your Cloud Applications in RightScaleRightScale This document discusses optimizing cloud applications using RightScale. It covers monitoring applications using tools like Collectd and New Relic RPM. It also discusses optimizing database performance on cloud infrastructure by scaling instances vertically and horizontally, using the right indices, and ensuring the working set fits in memory. The document provides an agenda and overview of these optimization techniques.
Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream ...
Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream ...Ververica The document discusses a system for real-time stream processing using Flink to join data streams of scientific accounts and publications, highlighting challenges like fault tolerance and accurate joining of data from distributed sources. It outlines a prototype implementation that facilitates the integration of user accounts with their respective publications while addressing issues related to change data capture and data synchronization. The document emphasizes the importance of maintaining up-to-date and accurate join results in a scalable infrastructure supporting millions of publications.
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...
Flink Forward SF 2017: Chinmay Soman - Real Time Analytics in the real World ...Flink Forward Uber Engineering discusses the challenges and solutions for near real-time analytics and event processing, emphasizing the use of Apache Kafka and Flink for data management and analytics. The presentation outlines various use cases such as growth metrics, fraud detection, and online analytical processing, all powered by sophisticated SQL and streaming technologies. Looking ahead, Uber aims to explore further advancements in real-time machine learning and auto-scaling within their data ecosystem.
Akka Streams
Akka StreamsDiego Pacheco This document discusses Akka Streams, which is a library for building reactive stream processing applications. It defines key concepts like streams, processing stages, back pressure, and flow control. It also covers topics like stream composition, materialization, using actors in streams, and includes code examples. The document is presented by Diego Pacheco, a principal software architect at Akka Streams.
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward The document discusses the use of Apache Flink for managing high-frequency time series data collected from wearable patches, emphasizing its capability to create a predictive health model. It explains Flink's state management features, particularly managed keyed state, and how to access queryable state outside the Flink runtime. Various use cases illustrate the transition from batch processing to stream processing within the healthcare data management context.
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large ScaleVerverica The document discusses the implementation and challenges of running Apache Flink at large scales, highlighting use cases such as stream ingestion and event-driven applications. Key issues addressed include dependency conflicts, checkpointing, state management, and performance optimization strategies. The talk emphasizes Flink's evolving architecture and the importance of understanding its underlying mechanisms for robust large-scale application deployment.
From Apache Flink® 1.3 to 1.4
From Apache Flink® 1.3 to 1.4Till Rohrmann The document provides an overview of the evolution and features of Apache Flink, highlighting updates from version 1.3 to the upcoming 1.5+, including improvements in APIs, state management, and checkpointing mechanisms. It emphasizes the platform's capabilities for event-driven I/O, integration with Kafka for exactly-once processing, and operational robustness. Finally, it invites readers to explore additional resources for deeper insights into Flink's development and innovations.
Online index rebuild automation
Online index rebuild automationCarlos Sierra This document discusses the automation of online index rebuilding in a multitenant database environment, aiming to optimize performance and reduce wasted space in indexes. A case study highlights the implementation of a custom infrastructure across numerous databases and the importance of index management strategies tailored to specific query patterns. The automation process leads to significant space savings and performance improvements, particularly for short-latency queries, while emphasizing the need for continuous monitoring and adjustments in the rebuilding process.
Stream processing - Apache flink
Stream processing - Apache flinkRenato Guimaraes Apache Flink is a distributed processing engine designed for stateful computations over both unbounded and bounded data streams, featuring low-latency, high throughput, and fault tolerance. It supports various applications such as fraud detection, real-time analytics, and data pipelines, employing several APIs like SQL, Table API, and DataStream API for stream processing. Flink's architecture includes job managers and task managers for efficient resource management and offers mechanisms like savepoints for fault tolerance and state management.
Viewers also liked (20)
Dive into Spark Streaming
Dive into Spark StreamingGerard Maas The document discusses Spark Streaming and its execution model. It describes how Spark Streaming receives data streams, divides them into micro-batches, and processes the micro-batches using Spark. It also covers transformations and actions that can be performed on DStreams, window operations, and deployment options for Spark Streaming including local, standalone, and cluster modes.
The Power of the Log
The Power of the LogBen Stopford The document discusses the use of log-structured merge (LSM) trees in modern databases, emphasizing their efficiency in handling writes by utilizing append-only data structures and sequential operations. It also highlights techniques to optimize reads, such as caching and bloom filters, and reframes database problems in a log-centric way for better system performance. Key examples and comparisons with existing technologies are provided to illustrate the advantages of adopting a log-centric approach.
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...Lightbend This webinar discusses building streaming and fast data applications with technologies like Spark, Mesos, Akka, Cassandra and Kafka. It covers how microservices and fast data architectures are converging due to similar design problems and data becoming the dominant problem. The webinar also introduces Lightbend's Fast Data Platform for building streaming data systems and microservices with best practices, sample applications and machine learning-based monitoring and management.
Akka-chan's Survival Guide for the Streaming World
Akka-chan's Survival Guide for the Streaming WorldKonrad Malawski The document outlines the evolution and significance of reactive streams, particularly in the context of Akka and JDK 9, highlighting their role in building concurrent, distributed applications. It discusses the development timelines, key features, and comparisons between various streaming technologies and libraries. The author emphasizes the importance of reactive streams for handling back-pressure and asynchronous processing in modern data architectures.
[OracleCode SF] In memory analytics with apache spark and hazelcast
[OracleCode SF] In memory analytics with apache spark and hazelcastViktor Gamov The document discusses the integration of Apache Spark with Hazelcast's in-memory data grid, highlighting the speed advantages of Spark and its fault-tolerance with resilient distributed datasets (RDDs). It provides code examples for configuration and uses cases, emphasizing parallel processing and operational benefits. Additionally, it mentions limitations regarding data updates while reading from Spark, suggesting potential issues like cursor inaccuracies.
Stream all the things
Stream all the thingsDean Wampler This document provides an overview of streaming architectures and compares different streaming engines. It describes a common streaming architecture that ingests data from sources like logs and sockets into Kafka for storage and organization. It then discusses several streaming engines like Apache Beam, Apache Flink, and Akka Streams that can process the data from Kafka. Beam provides the most sophisticated streaming semantics to handle scenarios like late data arrival, while Flink offers low latency processing at large scales and can run Beam data flows. Akka Streams is best suited for complex event processing on individual events.
Streamsets and spark
Streamsets and sparkHari Shreedharan StreamSets can process data using Apache Spark in three ways:
1) The Spark Evaluator stage allows user-provided Spark code to run on each batch of records in a pipeline and return results or errors.
2) A Cluster Pipeline can leverage Apache Spark's Direct Kafka DStream to partition data from Kafka across worker pipelines on a cluster.
3) A Spark Executor can kick off a Spark application when an event is received, allowing tasks like model updating to run on streaming data using Spark.
Apache Beam
Apache Beam Adil Oulghard Le document présente Apache Beam, un modèle de programmation unifié pour le traitement de données infinies et non ordonnées, permettant d'écrire des pipelines portables en Java et Python. Il décrit les concepts clés tels que le fenêtrage, les transformations et les exécuteurs, ainsi que des exemples pratiques comme le pipeline de comptage de mots. En conclusion, Apache Beam offre flexibilité et modularité pour gérer divers scénarios de traitement de données, y compris le streaming et le traitement batch.
Apache Spark and Oracle Stream Analytics
Apache Spark and Oracle Stream AnalyticsPrabhu Thukkaram This document discusses Oracle Stream Analytics, which provides complex event processing capabilities for Apache Spark Streaming. It leverages Oracle's Continuous Query Engine for event-by-event processing and Apache Spark for distributed computing and fault tolerance. Key features highlighted include stateful and continuous query processing, flexible temporal windows, pattern detection, spatial analysis, and integrated business rules. It is described as reducing application development time by handling state management and fault tolerance, while also scaling linearly with Spark and providing automatic recovery from failures.
Kafka & Couchbase Integration Patterns
Kafka & Couchbase Integration PatternsManuel Hurtado The document provides an overview of Kafka & Couchbase integration patterns. It introduces Couchbase and Kafka, describes how Kafka Connect enables real-time data pipelines between data systems, and how the Couchbase Kafka connector integrates Couchbase with Kafka pipelines. Use cases for the connector include using Couchbase as a data source or sink within Kafka streams. The document concludes with demos of Couchbase as a source and sink using the connector.
Kudu Forrester Webinar
Kudu Forrester WebinarCloudera, Inc. The document covers a Cloudera webinar series focused on Apache Kudu and its capabilities for real-time data analytics across various business use cases. It addresses the complexities of traditional lambda architectures, the technical architecture of Kudu, and the importance of real-time data pipelines for enhancing operational efficiencies and decision-making. The series also includes insights from Forrester on market trends and strategies for deploying scalable real-time data platforms.
Introduction to data flow management using apache nifi
Introduction to data flow management using apache nifiAnshuman Ghosh The document is an introduction to dataflow management using Apache NiFi, covering its history, features, architecture, and core components. It discusses how NiFi serves as an integrated platform for real-time data management and automation, highlighting common dataflow challenges and providing live demos and testing methodologies. The presentation aims to familiarize users with building dataflow processors, deploying dataflows, and future plans for NiFi enhancements.
Apache Flink Crash Course by Slim Baltagi and Srini Palthepu
Apache Flink Crash Course by Slim Baltagi and Srini PalthepuSlim Baltagi The document is a crash course on Apache Flink presented at a meetup in Chicago, which covers setup, configuration, and usage of Flink for batch and stream processing. It provides practical steps for installing Flink locally, on Docker, in a standalone cluster, or in the cloud, as well as using its tools and APIs. The agenda includes how to set up an IDE, write and debug applications, and utilize different deployment options.
Complex Event Processing with Esper
Complex Event Processing with EsperTed Won This document discusses Complex Event Processing (CEP) using Esper. It defines CEP as detecting patterns among events. Esper is an open source CEP engine that provides an SQL-like Event Processing Language (EPL) to define queries over event streams. The document outlines Esper's architecture, features like filtering, windows, aggregation, and joins. It provides examples of EPL queries for topics detection, continuous queries, and pattern matching.
Data Pipelines with Apache Kafka
Data Pipelines with Apache KafkaBen Stopford Kafka is a distributed streaming platform that allows for publishing and subscribing to streams of records. It provides low latency and high throughput processing of streams of records. Key features include replication across clusters for fault tolerance, partitioning for scalability, and ordering of records. Common uses of Kafka include building event driven microservices, stream processing, and serving as the data backbone for real-time analytics applications.
Step-by-Step Introduction to Apache Flink
Step-by-Step Introduction to Apache Flink Slim Baltagi This document provides a comprehensive guide on setting up, configuring, and running Apache Flink, covering environments such as local machines, virtual machines, Docker, standalone clusters, and cloud platforms. It also details the use of Flink tools including the command-line interface, web submission client, and interactive shells. Additionally, resources for further learning and specific configurations for integrating Flink with various data storage systems are included.
Apache Flink: Real-World Use Cases for Streaming Analytics
Apache Flink: Real-World Use Cases for Streaming AnalyticsSlim Baltagi The document discusses Apache Flink, emphasizing its transition from batch to streaming analytics, which is driven by the need for real-time data processing in various industries. It highlights key differentiators of Flink, such as low latency, high throughput, and its ability to ensure accurate results even during failures. Additionally, it explores real-world use cases across sectors like finance, healthcare, and retail, illustrating the growing adoption of streaming analytics for competitive advantage.
February 2017 HUG: Exactly-once end-to-end processing with Apache Apex
February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network The document discusses the Apache Apex platform's features, focusing on its capabilities for in-memory, distributed stream processing with an emphasis on exactly-once processing guarantees. It highlights the importance of fault tolerance, checkpointing for state recovery, and various strategies for ensuring that data is not duplicated or lost during failures when interacting with external systems such as databases and message queues. Key mechanisms and existing implementations from the Malhar library are also detailed for achieving these guarantees.
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at UberWSO2 The document outlines Uber's scalable real-time complex event processing (CEP) architecture, which utilizes Apache Samza and SiddhiQL for efficient event handling and processing. It discusses key features, such as state management, fault tolerance, and various types of actions for real-time applications, including fraud detection and marketing campaigns. Challenges like out-of-order event handling, large checkpointing states, and upgrading jobs are also addressed, highlighting Uber's commitment to maintaining reliable and efficient event processing systems.
Streaming all the things with akka streams
Streaming all the things with akka streams Johan Andrén This document provides an overview and introduction to Akka Streams and Reactive Streams. Some key points:
- Reactive Streams is a standard for asynchronous stream processing with non-blocking back pressure to prevent issues like out of memory errors.
- Akka Streams is a toolkit for building powerful concurrent and distributed applications simply using a Reactive Streams-compliant API. It includes sources, sinks, flows and other stages for stream processing.
- Examples show how to create simple stream graphs that process data asynchronously using Akka Streams APIs in both Java and Scala in just a few lines of code. More complex examples demonstrate features like parallelization.
- The community Alpakka
Ad
Similar to Apache Flink's Table & SQL API - unified APIs for batch and stream processing (20)
Webinar: Flink SQL in Action - Fabian Hueske
Webinar: Flink SQL in Action - Fabian HueskeVerverica Apache Flink is a powerful streaming platform that supports stateful computations over both real-time and historical data, enabling fast, scalable, and fault-tolerant analytics. Its unified APIs allow for seamless processing of batch and streaming data using ANSI SQL syntax, making it easier for users to perform complex queries on both data types. The platform is widely used in production by major companies and supports a variety of applications, including data pipelines, real-time analytics, and interactive query capabilities.
Fabian Hueske - Taking a look under the hood of Apache Flink’s relational APIs
Fabian Hueske - Taking a look under the hood of Apache Flink’s relational APIsFlink Forward The document discusses the challenges and benefits of Apache Flink's relational APIs, focusing on their ability to simplify stream and batch data processing. It highlights the use of a Table API and SQL for query writing, along with the architecture for translating and optimizing queries through Apache Calcite. The current state and future outlook for Flink's streaming capabilities are also presented, emphasizing accessibility and performance improvements.
Taking a look under the hood of Apache Flink's relational APIs.
Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske The document discusses Apache Flink's relational APIs, focusing on the challenges and advantages of using these APIs for both streaming and batch data processing. It highlights the integration of SQL and a table API within Flink, and how queries are optimized through Apache Calcite to ensure efficient execution. The current state of Flink's capabilities and future outlook for improvements in streaming and batch SQL support are also outlined.
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"Flink Forward This document discusses Flink SQL and its capabilities for stream and batch processing. Flink SQL provides a unified SQL interface that allows the same queries to run over both bounded batch data and unbounded streaming data. It achieves this through dynamic tables that are continuously updated as streaming data arrives. The document outlines Flink SQL's features, how queries are translated and executed, and examples of how SQL can be used to build data pipelines, real-time analytics, and live dashboards. It also introduces the new SQL client for interacting with Flink without writing code.
Flink Forward San Francisco 2018: Fabian Hueske & Timo Walther - "Why and how...
Flink Forward San Francisco 2018: Fabian Hueske & Timo Walther - "Why and how...Flink Forward The document discusses the integration and usage of SQL with Apache Flink for stream and batch processing, highlighting its ability to handle both bounded and unbounded data seamlessly. It details features like dynamic tables, materialized views, and continuous queries, along with practical applications in real-time analytics and dashboards. The summary also notes the ongoing development of user interfaces and query services within the Flink community to enhance usability.
Why and how to leverage the simplicity and power of SQL on Flink
Why and how to leverage the simplicity and power of SQL on FlinkDataWorks Summit The document discusses the advantages of using SQL on Apache Flink for both batch and streaming data processing, emphasizing the unified APIs that support dynamic tables and continuous queries. It highlights the capabilities of Flink for real-time analytics, data pipelines, and the maintenance of materialized views, while also addressing user interface improvements and plans for a SQL query service. Additionally, it notes that Flink is used in production by companies like Alibaba and Uber, encouraging community involvement and feedback.
Flink SQL & TableAPI in Large Scale Production at Alibaba
Flink SQL & TableAPI in Large Scale Production at AlibabaDataWorks Summit The document discusses the implementation and impact of Flink SQL and Table API in large-scale production at Alibaba, focusing on the development of Blink, Alibaba's version of Flink, designed to enhance performance for big data applications. It highlights improvements in efficiency and user experience by unifying batch and stream processing, and showcases successful examples of its application in real-time data analysis, particularly during high-traffic events like China Singles' Day. The document also details contributions to the Flink community and enhancements like user-defined functions and aggregates that support Alibaba's extensive operational needs.
What's new in 1.9.0 blink planner - Kurt Young, Alibaba
What's new in 1.9.0 blink planner - Kurt Young, AlibabaFlink Forward Kurt Young, a staff engineer at Alibaba, presents the latest developments in the Blink Planner for Apache Flink. The session covers the architecture, new features, improvements in streaming and batch processing, and future plans including the finalization of the new type system and full TPC-DS support. The Blink Planner aims to provide a unified query processing platform for both batch and streaming data operations.
Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...
Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...Flink Forward This document discusses the capabilities and features of Apache Flink's SQL and Table API for stream analytics. It covers the transition from static batch data processing to continuous stream processing, elaborating on concepts like dynamic tables, continuous queries, and materialized views. The document highlights the potential applications of these technologies in building real-time data systems and emphasizes the importance of efficient execution and automatic optimization.
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...Flink Forward This document summarizes recent improvements to Flink SQL and Table API by Blink, Alibaba's distribution of Flink. Key improvements include support for stream-stream joins, user-defined functions, table functions and aggregate functions, retractable streams, and over/group aggregates. Blink aims to make Flink work well at large scale for Alibaba's search and recommendation systems. Many of the improvements will be included in upcoming Flink releases.
Stream Analytics with SQL on Apache Flink
Stream Analytics with SQL on Apache FlinkFabian Hueske The document discusses stream analytics using SQL on Apache Flink, highlighting Flink's capabilities for low-latency stream processing and exactly-once state guarantees. It emphasizes the growing interest in SQL support for streaming applications while addressing the challenges of adapting SQL for continuous data streams. The document explores the concept of dynamic tables in Flink, allowing for real-time querying and updates, and outlines ongoing development efforts for improving streaming SQL features.
Fabian Hueske - Stream Analytics with SQL on Apache Flink
Fabian Hueske - Stream Analytics with SQL on Apache FlinkVerverica The document discusses stream analytics using SQL on Apache Flink, emphasizing its capabilities for low latency, high throughput, and exactly-once state guarantees. It introduces the concept of dynamic tables, which change over time and can be queried with standard SQL, offering a unified API for both batch and streaming data. Additionally, it highlights the current state of SQL support in Flink, ongoing development efforts, and the potential use cases for SQL in stream processing.
CDC Stream Processing with Apache Flink
CDC Stream Processing with Apache FlinkTimo Walther The document provides an overview of Apache Flink, highlighting its capabilities for stream processing and the unique aspects that distinguish it in the field. It discusses the various APIs available for users to interact with Flink, focusing on the stream-table duality and the innovative use of changelogs. The speaker also emphasizes the flexibility of Flink's SQL engine for integrating diverse systems and mentions additional resources available for users.
Towards sql for streams
Towards sql for streamsRadu Tudoran SQL can be used to query both streaming and batch data. Apache Flink and Apache Calcite enable SQL queries on streaming data. Flink uses its Table API and integrates with Calcite to translate SQL queries into dataflow programs. This allows standard SQL to be used for both traditional batch analytics on finite datasets and stream analytics producing continuous results from infinite data streams. Queries are executed continuously by applying operators within windows to subsets of streaming data.
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords Stephan Ewen The document discusses Apache Flink as a stream processor that efficiently handles both real-time and historical data with features like low latency, high throughput, and exactly-once semantics for fault tolerance. It highlights the architecture, data processing capabilities, and the concept of queryable state, which allows applications to access real-time and historical results. The conclusion emphasizes that Apache Flink's sophisticated state management improves performance and mitigates bottlenecks in data processing systems.
The Stream Processor as a Database Apache Flink
The Stream Processor as a Database Apache FlinkDataWorks Summit/Hadoop Summit Apache Flink enables stream processing on continuously produced data through its DataStream and DataSet APIs. It allows for streaming and batch processing as first class citizens. Flink programs are composed of sources that ingest data, transformations on those data streams, and sinks that output the results. Queryable state in Flink allows for querying the system state without writing to an external database, improving performance over traditional architectures that rely on writing intermediate results to external key-value stores. Flink's use of lightweight snapshots for fault tolerance and its log-based approach to persistence allows queryable state to have high throughput and low latency.
Flink's SQL Engine: Let's Open the Engine Room!
Flink's SQL Engine: Let's Open the Engine Room!HostedbyConfluent The document presents an overview of Apache Flink's SQL engine, highlighting its capabilities for stream processing using dynamic tables instead of traditional databases. It explains key concepts such as timestamp management, changelog modes, and SQL execution planning, demonstrating how SQL statements translate into physical execution plans and stream transformations. Various examples illustrate workflows, including joins and watermarks, showcasing how users can implement these features in their applications.
Why apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksSlim Baltagi This document provides an overview and agenda for a presentation on Apache Flink. It begins with an introduction to Apache Flink and how it fits into the big data ecosystem. It then explains why Flink is considered the "4th generation" of big data analytics frameworks. Finally, it outlines next steps for those interested in Flink, such as learning more or contributing to the project. The presentation covers topics such as Flink's APIs, libraries, architecture, programming model and integration with other tools.
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch ProcessingHostedbyConfluent Flink 2.0 focuses on unified stream and batch processing capabilities with enhancements in language and API support, including fault tolerance, improved SQL functions, and dynamic table management. Future developments will revolve around mixed unification, a unified SQL platform, streaming warehouses, and engine evolution for better performance and usability. The timeline indicates Flink 1.19's release in March 2024, followed by subsequent versions with incremental improvements.
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022HostedbyConfluent The document is a presentation by Timo Walther on Apache Flink, focusing on its capabilities for stream processing and the use of Flink SQL to manage data as dynamic tables. It describes the structure of Flink, including its APIs, data processing models, and concepts like changelog entry updates, stream-table duality, and operational modes. The presentation highlights Flink's usefulness for integrating various data systems and has links to additional resources and projects related to Flink.
Ad
Recently uploaded (20)
Transmission Media. (Computer Networks)
Transmission Media. (Computer Networks)S Pranav (Deepu) INTRODUCTION:TRANSMISSION MEDIA
• A transmission media in data communication is a physical path between the sender and
the receiver and it is the channel through which data can be sent from one location to
another. Data can be represented through signals by computers and other sorts of
telecommunication devices. These are transmitted from one device to another in the
form of electromagnetic signals. These Electromagnetic signals can move from one
sender to another receiver through a vacuum, air, or other transmission media.
Electromagnetic energy mainly includes radio waves, visible light, UV light, and gamma
ra
Open Source Software Development Methods
Open Source Software Development MethodsVICTOR MAESTRE RAMIREZ Open Source Software Development Methods
AI-Powered Compliance Solutions for Global Regulations | Certivo
AI-Powered Compliance Solutions for Global Regulations | Certivocertivoai Certivo offers AI-powered compliance solutions designed to help businesses in the USA, EU, and UK simplify complex regulatory demands. From environmental and product compliance to safety, quality, and sustainability, our platform automates supplier documentation, manages certifications, and integrates with ERP/PLM systems. Ensure seamless RoHS, REACH, PFAS, and Prop 65 compliance through predictive insights and multilingual support. Turn compliance into a competitive edge with Certivo’s intelligent, scalable, and audit-ready platform.
How the US Navy Approaches DevSecOps with Raise 2.0
How the US Navy Approaches DevSecOps with Raise 2.0Anchore Join us as Anchore's solutions architect reveals how the U.S. Navy successfully approaches the shift left philosophy to DevSecOps with the RAISE 2.0 Implementation Guide to support its Cyber Ready initiative. This session will showcase practical strategies for defense application teams to pivot from a time-intensive compliance checklist and mindset to continuous cyber-readiness with real-time visibility.
Learn how to break down organizational silos through RAISE 2.0 principles and build efficient, secure pipeline automation that produces the critical security artifacts needed for Authorization to Operate (ATO) approval across military environments.
wAIred_RabobankIgniteSession_12062025.pptx
wAIred_RabobankIgniteSession_12062025.pptxSimonedeGijt In today's world, artificial intelligence (AI) is transforming the way we learn.
This talk will explore how we can use AI tools to enhance our learning experiences, by looking at some (recent) research that has been done on the matter.
But as we embrace these new technologies, we must also ask ourselves:
Are we becoming less capable of thinking for ourselves?
Do these tools make us smarter, or do they risk dulling our critical thinking skills?
This talk will encourage us to think critically about the role of AI in our education. Together, we will discover how to use AI to support our learning journey while still developing our ability to think critically.
Plooma is a writing platform to plan, write, and shape books your way
Plooma is a writing platform to plan, write, and shape books your wayPlooma Plooma is your all in one writing companion, designed to support authors at every twist and turn of the book creation journey. Whether you're sketching out your story's blueprint, breathing life into characters, or crafting chapters, Plooma provides a seamless space to organize all your ideas and materials without the overwhelm. Its intuitive interface makes building rich narratives and immersive worlds feel effortless.
Packed with powerful story and character organization tools, Plooma lets you track character development and manage world building details with ease. When it’s time to write, the distraction-free mode offers a clean, minimal environment to help you dive deep and write consistently. Plus, built-in editing tools catch grammar slips and style quirks in real-time, polishing your story so you don’t have to juggle multiple apps.
What really sets Plooma apart is its smart AI assistant - analyzing chapters for continuity, helping you generate character portraits, and flagging inconsistencies to keep your story tight and cohesive. This clever support saves you time and builds confidence, especially during those complex, detail packed projects.
Getting started is simple: outline your story’s structure and key characters with Plooma’s user-friendly planning tools, then write your chapters in the focused editor, using analytics to shape your words. Throughout your journey, Plooma’s AI offers helpful feedback and suggestions, guiding you toward a polished, well-crafted book ready to share with the world.
With Plooma by your side, you get a powerful toolkit that simplifies the creative process, boosts your productivity, and elevates your writing - making the path from idea to finished book smoother, more fun, and totally doable.
Get Started here: https://www.plooma.ink/
Wondershare PDFelement Pro 11.4.20.3548 Crack Free Download
Wondershare PDFelement Pro 11.4.20.3548 Crack Free DownloadPuppy jhon ➡ 🌍📱👉COPY & PASTE LINK👉👉👉 ➤ ➤➤ https://drfiles.net/
Wondershare PDFelement Professional is professional software that can edit PDF files. This digital tool can manipulate elements in PDF documents.
Making significant Software Architecture decisions
Making significant Software Architecture decisionsBert Jan Schrijver Presented at the iSAQB Software Architecture Community NL meetup on 12-6-2025
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptxMaharshi Mallela Movie recommendation system is a software application or algorithm designed to suggest movies to users based on their preferences, viewing history, or other relevant factors. The primary goal of such a system is to enhance user experience by providing personalized and relevant movie suggestions.
Software Engineering Process, Notation & Tools Introduction - Part 4
Software Engineering Process, Notation & Tools Introduction - Part 4Gaurav Sharma Software Engineering Process, Notation & Tools Introduction
Generative Artificial Intelligence and its Applications
Generative Artificial Intelligence and its ApplicationsSandeepKS52 The exploration of generative AI begins with an overview of its fundamental concepts, highlighting how these technologies create new content and ideas by learning from existing data. Following this, the focus shifts to the processes involved in training and fine-tuning models, which are essential for enhancing their performance and ensuring they meet specific needs. Finally, the importance of responsible AI practices is emphasized, addressing ethical considerations and the impact of AI on society, which are crucial for developing systems that are not only effective but also beneficial and fair.
AI and Deep Learning with NVIDIA Technologies
AI and Deep Learning with NVIDIA TechnologiesSandeepKS52 Artificial intelligence and deep learning are transforming various fields by enabling machines to learn from data and make decisions. Understanding how to prepare data effectively is crucial, as it lays the foundation for training models that can recognize patterns and improve over time. Once models are trained, the focus shifts to deployment, where these intelligent systems are integrated into real-world applications, allowing them to perform tasks and provide insights based on new information. This exploration of AI encompasses the entire process from initial concepts to practical implementation, highlighting the importance of each stage in creating effective and reliable AI solutions.
Step by step guide to install Flutter and Dart
Step by step guide to install Flutter and DartS Pranav (Deepu) Flutter is basically Google’s portable user
interface (UI) toolkit, used to build and
develop eye-catching, natively-built
applications for mobile, desktop, and web,
from a single codebase. Flutter is free, open-
sourced, and compatible with existing code. It
is utilized by companies and developers
around the world, due to its user-friendly
interface and fairly simple, yet to-the-point
commands.
GDG Douglas - Google AI Agents: Your Next Intern?
GDG Douglas - Google AI Agents: Your Next Intern?felipeceotto Presentation done at the GDG Douglas event for June 2025.
A first look at Google's new Agent Development Kit.
Agent Development Kit is a new open-source framework from Google designed to simplify the full stack end-to-end development of agents and multi-agent systems.
Software Testing & it’s types (DevOps)
Software Testing & it’s types (DevOps)S Pranav (Deepu) NTRODUCTION TO SOFTWARE TESTING
• Definition:
• Software testing is the process of evaluating and
verifying that a software application or system meets
specified requirements and functions correctly.
• Purpose:
• Identify defects and bugs in the software.
• Ensure the software meets quality standards.
• Validate that the software performs as intended in
various scenarios.
• Importance:
• Reduces risks associated with software failures.
• Improves user satisfaction and trust in the product.
• Enhances the overall reliability and performance of
the software
Porting Qt 5 QML Modules to Qt 6 Webinar
Porting Qt 5 QML Modules to Qt 6 WebinarICS Have you upgraded your application from Qt 5 to Qt 6? If so, your QML modules might still be stuck in the old Qt 5 style—technically compatible, but far from optimal. Qt 6 introduces a modernized approach to QML modules that offers better integration with CMake, enhanced maintainability, and significant productivity gains.
In this webinar, we’ll walk you through the benefits of adopting Qt 6 style QML modules and show you how to make the transition. You'll learn how to leverage the new module system to reduce boilerplate, simplify builds, and modernize your application architecture. Whether you're planning a full migration or just exploring what's new, this session will help you get the most out of your move to Qt 6.
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
- 1. 1 Timo Walther Apache Flink PMC @twalthr With slides from Fabian Hueske Flink Meetup @ Amsterdam, March 2nd, 2017 Table & SQL API unified APIs for batch and stream processing
- 2. 2 Original creators of Apache Flink® Providers of the dA Platform, a supported Flink distribution
- 3. Motivation 3
- 4. DataStream API is not for Everyone 4 § Writing DataStream programs is not easy • Stream processing technology spreads rapidly § Requires Knowledge & Skill • Stream processing concepts (time, state, windows, ...) • Programming experience (Java / Scala) § Program logic goes into UDFs • great for expressiveness • bad for optimization - need for manual tuning
- 5. Why not a Relational API? 5 § Relational APIs are declarative • User says what is needed • System decides how to compute it § Users do not specify implementation § Queries are efficiently executed § “Everybody” knows SQL!
- 6. Goals § Flink is a platform for distributed stream and batch data processing § Relational APIs as a unifying layer • Queries on batch tables terminate and produce a finite result • Queries on streaming tables run continuously and produce result stream § Same syntax & semantics for both queries 6
- 7. Table API & SQL 7
- 8. Table API & SQL § Flink features two relational APIs • Table API: LINQ-style API for Java & Scala (since Flink 0.9.0) • SQL: Standard SQL (since Flink 1.1.0) § Equivalent feature set (at the moment) • Table API and SQL can be mixed § Both are tightly integrated with Flink’s core APIs • DataStream • DataSet 8
- 9. Table API Example 9 val sensorData: DataStream[(String, Long, Double)] = ??? // convert DataSet into Table val sensorTable: Table = sensorData .toTable(tableEnv, 'location, ’time, 'tempF) // define query on Table val avgTempCTable: Table = sensorTable .window(Tumble over 1.day on 'rowtime as 'w) .groupBy('location, ’w) .select('w.start as 'day, 'location, (('tempF.avg - 32) * 0.556) as 'avgTempC) .where('location like "room%")
- 10. SQL Example 10 val sensorData: DataStream[(String, Long, Double)] = ??? // register DataStream tableEnv.registerDataStream( "sensorData", sensorData, 'location, ’time, 'tempF) // query registered Table val avgTempCTable: Table = tableEnv .sql(""" SELECT FLOOR(rowtime() TO DAY) AS day, location, AVG((tempF - 32) * 0.556) AS avgTempC FROM sensorData WHERE location LIKE 'room%' GROUP BY location, FLOOR(rowtime() TO DAY) """)
- 11. Architecture 2 APIs [SQL, Table API] * 2 backends [DataStream, DataSet] = 4 different translation paths? 11
- 12. Architecture 12
- 13. Architecture § Table API and SQL queries are translated into common logical plan representation. § Logical plans are translated and optimized depending on execution backend. § Plans are transformed into DataSet or DataStream programs. 13
- 14. Translation to Logical Plan 14 sensorTable .window(Tumble over 1.day on 'rowtime as 'w) .groupBy('location, ’w) .select('w.start as 'day, 'location, (('tempF.avg - 32) * 0.556) as 'avgTempC) .where('location like "room%")
- 15. Translation to Optimized Plan 15
- 16. Translation to Flink Program 16
- 17. Current State (in master) § Batch SQL & Table API support • Selection, Projection, Sort, Inner & Outer Joins, Set operations • Windows for Slide, Tumble, Session § Streaming Table API support • Selection, Projection, Union • Windows for Slide, Tumble, Session § Streaming SQL • Selection, Projection, Union, Tumble, but … 17
- 18. Use Cases for Streaming SQL § Continuous ETL & Data Import § Live Dashboards & Reports § Ad-hoc Analytics & Exploration 18
- 20. Dynamic Tables § Dynamic tables change over time § Dynamic tables are treated like static batch tables • Dynamic tables are queried with standard SQL • A query returns another dynamic table § Stream ←→ Dynamic Table conversions without information loss • “Stream / Table Duality” 20
- 21. Stream to Dynamic Tables § Append: § Replace by key: 21
- 22. Querying Dynamic Tables § Dynamic tables change over time • A[t]: Table A at time t § Dynamic tables are queried with regular SQL • Result of a query changes as input table changes • q(A[t]): Evaluate query q on table A at time t § Query result is continuously updated as t progresses • Similar to maintaining a materialized view • t is current event time 22
- 24. Querying Dynamic Tables § Can we run any query on Dynamic Tables? No! § State may not grow infinitely as more data arrives • Set clean-up timeout or key constraints. § Input may only trigger partial re-computation § Queries with possibly unbounded state or computation are rejected 24
- 25. Dynamic Tables to Stream § Update: 25
- 26. Dynamic Tables to Stream § Add/Retract: 26
- 27. Result computation & refinement 27
- 28. Contributions welcome! § Huge interest and many contributors • Adding more window operators • Introducing dynamic tables § And there is a lot more to do • New operators and features for streaming and batch • Performance improvements • Tooling and integration § Try it out, give feedback, and start contributing! 28
- 29. 29 One day of hands-on Flink training One day of conference Tickets are on sale Please visit our website: http://sf.flink-forward.org Follow us on Twitter: @FlinkForward