SlideShare a Scribd company logo

Streaming Data from Scylla to Kafka

ScyllaDB, profile picture
ScyllaDB

The document discusses the integration of Scylla with Kafka for creating an event streaming platform, focusing on the implementation of a Scylla source connector built on Debezium. It details steps for setting up a Scylla table with Change Data Capture (CDC), configuring the Kafka connector, and testing the setup for data replication. The document emphasizes the exchange of data between Scylla clusters using Kafka and showcases the demonstration with example configurations and expected outputs.

Technology
Related topics:
Apache Kafka • NoSQL Database Insights
1 of 18
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka
Streaming Data from Scylla to Kafka

More Related Content

PDF
Building Out Your Kafka Developer CDC Ecosystem
confluent
 
PDF
KSQL in Practice (Almog Gavra, Confluent) Kafka Summit London 2019
confluent
 
PDF
ksqlDB: A Stream-Relational Database System
confluent
 
PDF
Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020
confluent
 
PPTX
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Joe Stein
 
PPTX
Top Ten Kafka® Configs
confluent
 
PDF
ksqlDB - Stream Processing simplified!
Guido Schmutz
 
PDF
Streaming ETL - from RDBMS to Dashboard with KSQL
Bjoern Rost
 
Building Out Your Kafka Developer CDC Ecosystem
confluent
 
KSQL in Practice (Almog Gavra, Confluent) Kafka Summit London 2019
confluent
 
ksqlDB: A Stream-Relational Database System
confluent
 
Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020
confluent
 
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Joe Stein
 
Top Ten Kafka® Configs
confluent
 
ksqlDB - Stream Processing simplified!
Guido Schmutz
 
Streaming ETL - from RDBMS to Dashboard with KSQL
Bjoern Rost
 

What's hot (20)

PDF
War Stories: DIY Kafka
confluent
 
PDF
KSQL: Streaming SQL for Kafka
confluent
 
PDF
KSQL - Stream Processing simplified!
Guido Schmutz
 
PDF
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Lightbend
 
PDF
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
confluent
 
PDF
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
confluent
 
PDF
Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020
confluent
 
PDF
What is the State of my Kafka Streams Application? Unleashing Metrics. | Neil...
HostedbyConfluent
 
PDF
Performance Tuning RocksDB for Kafka Streams’ State Stores
confluent
 
PDF
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
PDF
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
confluent
 
PDF
Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture
confluent
 
PPTX
Kick your database_to_the_curb_reston_08_27_19
confluent
 
PDF
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
Ververica
 
PDF
Apache kafka meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_2...
confluent
 
PDF
Spark day 2017 - Spark on Kubernetes
Yousun Jeong
 
PDF
Performance Analysis and Optimizations for Kafka Streams Applications
Guozhang Wang
 
PPTX
Take Kafka-on-Pulsar to Production at Internet Scale: Improvements Made for P...
StreamNative
 
PDF
Apache Kafka® Security Overview
confluent
 
PDF
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
HostedbyConfluent
 
War Stories: DIY Kafka
confluent
 
KSQL: Streaming SQL for Kafka
confluent
 
KSQL - Stream Processing simplified!
Guido Schmutz
 
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Lightbend
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
confluent
 
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
confluent
 
Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020
confluent
 
What is the State of my Kafka Streams Application? Unleashing Metrics. | Neil...
HostedbyConfluent
 
Performance Tuning RocksDB for Kafka Streams’ State Stores
confluent
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
confluent
 
Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture
confluent
 
Kick your database_to_the_curb_reston_08_27_19
confluent
 
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
Ververica
 
Apache kafka meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_2...
confluent
 
Spark day 2017 - Spark on Kubernetes
Yousun Jeong
 
Performance Analysis and Optimizations for Kafka Streams Applications
Guozhang Wang
 
Take Kafka-on-Pulsar to Production at Internet Scale: Improvements Made for P...
StreamNative
 
Apache Kafka® Security Overview
confluent
 
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
HostedbyConfluent
 
Ad

Similar to Streaming Data from Scylla to Kafka (20)

PPTX
Connecting kafka message systems with scylla
Maheedhar Gunturu
 
PDF
London Apache Kafka Meetup (Jan 2017)
Landoop Ltd
 
PDF
Beyond the brokers - A tour of the Kafka ecosystem
Damien Gasparina
 
PDF
Beyond the Brokers: A Tour of the Kafka Ecosystem
confluent
 
PDF
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
PDF
Beyond the brokers - Un tour de l'écosystème Kafka
Florent Ramiere
 
PPTX
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
 
PPTX
Kafka Tutorial - introduction to the Kafka streaming platform
Jean-Paul Azar
 
PDF
Building Event Streaming Architectures on Scylla and Kafka
ScyllaDB
 
PPTX
Kafka Tutorial: Streaming Data Architecture
Jean-Paul Azar
 
PPTX
Event streaming webinar feb 2020
Maheedhar Gunturu
 
PDF
Feeding Cassandra with Spark-Streaming and Kafka
DataStax Academy
 
PDF
kafka-tutorial-cloudruable-v2.pdf
PriyamTomar1
 
PDF
Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL
ScyllaDB
 
PDF
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
Athens Big Data
 
PPTX
Event Streaming Architectures with Confluent and ScyllaDB
ScyllaDB
 
PDF
Devoxx university - Kafka de haut en bas
Florent Ramiere
 
PPTX
Kafka Tutorial, Kafka ecosystem with clustering examples
Jean-Paul Azar
 
PPTX
kafka for db as postgres
PivotalOpenSourceHub
 
PDF
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
confluent
 
Connecting kafka message systems with scylla
Maheedhar Gunturu
 
London Apache Kafka Meetup (Jan 2017)
Landoop Ltd
 
Beyond the brokers - A tour of the Kafka ecosystem
Damien Gasparina
 
Beyond the Brokers: A Tour of the Kafka Ecosystem
confluent
 
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
Beyond the brokers - Un tour de l'écosystème Kafka
Florent Ramiere
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
 
Kafka Tutorial - introduction to the Kafka streaming platform
Jean-Paul Azar
 
Building Event Streaming Architectures on Scylla and Kafka
ScyllaDB
 
Kafka Tutorial: Streaming Data Architecture
Jean-Paul Azar
 
Event streaming webinar feb 2020
Maheedhar Gunturu
 
Feeding Cassandra with Spark-Streaming and Kafka
DataStax Academy
 
kafka-tutorial-cloudruable-v2.pdf
PriyamTomar1
 
Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL
ScyllaDB
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
Athens Big Data
 
Event Streaming Architectures with Confluent and ScyllaDB
ScyllaDB
 
Devoxx university - Kafka de haut en bas
Florent Ramiere
 
Kafka Tutorial, Kafka ecosystem with clustering examples
Jean-Paul Azar
 
kafka for db as postgres
PivotalOpenSourceHub
 
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
confluent
 
Ad

More from ScyllaDB (20)

PDF
Understanding The True Cost of DynamoDB Webinar
ScyllaDB
 
PDF
Database Benchmarking for Performance Masterclass: Session 2 - Data Modeling ...
ScyllaDB
 
PDF
Database Benchmarking for Performance Masterclass: Session 1 - Benchmarking F...
ScyllaDB
 
PDF
New Ways to Reduce Database Costs with ScyllaDB
ScyllaDB
 
PDF
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
ScyllaDB
 
PDF
Powering a Billion Dreams: Scaling Meesho’s E-commerce Revolution with Scylla...
ScyllaDB
 
PDF
Leading a High-Stakes Database Migration
ScyllaDB
 
PDF
Achieving Extreme Scale with ScyllaDB: Tips & Tradeoffs
ScyllaDB
 
PDF
Securely Serving Millions of Boot Artifacts a Day by João Pedro Lima & Matt ...
ScyllaDB
 
PDF
How Agoda Scaled 50x Throughput with ScyllaDB by Worakarn Isaratham
ScyllaDB
 
PDF
How Yieldmo Cut Database Costs and Cloud Dependencies Fast by Todd Coleman
ScyllaDB
 
PDF
ScyllaDB: 10 Years and Beyond by Dor Laor
ScyllaDB
 
PDF
Reduce Your Cloud Spend with ScyllaDB by Tzach Livyatan
ScyllaDB
 
PDF
Migrating 50TB Data From a Home-Grown Database to ScyllaDB, Fast by Terence Liu
ScyllaDB
 
PDF
Vector Search with ScyllaDB by Szymon Wasik
ScyllaDB
 
PDF
Workload Prioritization: How to Balance Multiple Workloads in a Cluster by Fe...
ScyllaDB
 
PDF
Two Leading Approaches to Data Virtualization, and Which Scales Better? by Da...
ScyllaDB
 
PDF
Scaling a Beast: Lessons from 400x Growth in a High-Stakes Financial System b...
ScyllaDB
 
PDF
Object Storage in ScyllaDB by Ran Regev, ScyllaDB
ScyllaDB
 
PDF
Lessons Learned from Building a Serverless Notifications System by Srushith R...
ScyllaDB
 
Understanding The True Cost of DynamoDB Webinar
ScyllaDB
 
Database Benchmarking for Performance Masterclass: Session 2 - Data Modeling ...
ScyllaDB
 
Database Benchmarking for Performance Masterclass: Session 1 - Benchmarking F...
ScyllaDB
 
New Ways to Reduce Database Costs with ScyllaDB
ScyllaDB
 
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
ScyllaDB
 
Powering a Billion Dreams: Scaling Meesho’s E-commerce Revolution with Scylla...
ScyllaDB
 
Leading a High-Stakes Database Migration
ScyllaDB
 
Achieving Extreme Scale with ScyllaDB: Tips & Tradeoffs
ScyllaDB
 
Securely Serving Millions of Boot Artifacts a Day by João Pedro Lima & Matt ...
ScyllaDB
 
How Agoda Scaled 50x Throughput with ScyllaDB by Worakarn Isaratham
ScyllaDB
 
How Yieldmo Cut Database Costs and Cloud Dependencies Fast by Todd Coleman
ScyllaDB
 
ScyllaDB: 10 Years and Beyond by Dor Laor
ScyllaDB
 
Reduce Your Cloud Spend with ScyllaDB by Tzach Livyatan
ScyllaDB
 
Migrating 50TB Data From a Home-Grown Database to ScyllaDB, Fast by Terence Liu
ScyllaDB
 
Vector Search with ScyllaDB by Szymon Wasik
ScyllaDB
 
Workload Prioritization: How to Balance Multiple Workloads in a Cluster by Fe...
ScyllaDB
 
Two Leading Approaches to Data Virtualization, and Which Scales Better? by Da...
ScyllaDB
 
Scaling a Beast: Lessons from 400x Growth in a High-Stakes Financial System b...
ScyllaDB
 
Object Storage in ScyllaDB by Ran Regev, ScyllaDB
ScyllaDB
 
Lessons Learned from Building a Serverless Notifications System by Srushith R...
ScyllaDB
 

Recently uploaded (20)

PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PPTX
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PPTX
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPTX
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
A Presentation on Artificial Intelligence
memembruh336
 
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Approach and Philosophy of On baking technology
30anthuong17
 
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 

Editor's Notes

  • #13: On my Scylla cluster I created a keyspace and a CDC-enabled table with cqlsh:
  • #14: Then I started a connector with the following configuration. Notable things: tasks.max=10 - the connector is able to automatically split the work evenly between 10 tasks, even when replicating a single table; transforms.unwrap - I used a standard Debezium transformation, which removes some metadata to make it easier to read.
  • #15: After starting the connector, the changes will be visible on the MyCluster_ks_t topic. To test the connector, I inserted a few rows with cqlsh:
  • #16: After starting the connector, the changes will be visible on the MyCluster_ks_t topic. To test the connector, I inserted a few rows with cqlsh:
  • #17: The connector correctly replicated those operations to the topic: (I configured the connector with JSON serializer with schema):