Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
0 likes2,190 views
The document discusses event hubs as a core component of modern data architecture, emphasizing their capabilities such as scalability, durability, and message retention. It highlights Apache Kafka as the most popular event hub and outlines its features, as well as alternatives to Kafka. The content also explores the integration of event hubs with various data sources and processing frameworks to enable real-time data analytics and microservices architecture.
![Event Hub
Stream
Analytics
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Shop
Floor
Gateway
Using Stream Analytics
• Stream-to-Stream Joins
• Stream-to-Table Joins
• Time Windowed State Management
• Event Pattern Detection
• Machine Learning Model Execution
(Inference)
[1]](https://image.slidesharecdn.com/event-hub-in-modern-data-architecture-v1-200226221521/85/Event-Hub-i-e-Kafka-in-Modern-Data-Analytics-Architecture-17-320.jpg)
![Event Hub
Stream
Analytics
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Shop
Floor
Gateway
Using Streaming Visualization
• ksqlDB pull queries or Kafka Streams
Interactive Queries allow to query state of
stream processor
[2]](https://image.slidesharecdn.com/event-hub-in-modern-data-architecture-v1-200226221521/85/Event-Hub-i-e-Kafka-in-Modern-Data-Analytics-Architecture-21-320.jpg)
![Event Hub
Stream
Analytics
Legacy
App
Machine
IIoT
Stream Data
Integration
Legacy Data Sources
CDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
Micro
service
Micro
service
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Shop
Floor
Gateway
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
Event-Driven, highly decoupled Apps
(aka. Microservices)
• 2nd microservice consumes events from 1st microservice
• Bootstrap new microservices from event history
• System-wide CQRS
[3]](https://image.slidesharecdn.com/event-hub-in-modern-data-architecture-v1-200226221521/85/Event-Hub-i-e-Kafka-in-Modern-Data-Analytics-Architecture-34-320.jpg)
![Event Hub
Stream
Analytics
Legacy
App
Machine
IIoT
Stream Data
Integration
Legacy Data Sources
CDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
Micro
service
Micro
service
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Shop
Floor
Gateway
Batch Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Data Lake /
DWH
Batch
Visualize
Batch
Analytics
Micro-Batch
Visualize
Bi-Directional Legacy Systems
Integration
• 2nd microservice consumes events from 1st microservice
• Bootstrap new microservices from event history
• System-wide CQRS
[4]AQ](https://image.slidesharecdn.com/event-hub-in-modern-data-architecture-v1-200226221521/85/Event-Hub-i-e-Kafka-in-Modern-Data-Analytics-Architecture-35-320.jpg)
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
- 1. BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://[email protected]@gschmutz Event Hub in Modern Data Architecture Guido Schmutz Data Analytics Summit 2020 – Santa Clara
- 2. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 23 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 184th edition
- 3. Agenda 1. What exactly is an Event Hub? 2. Kafka – the most popular Event Hub 3. Event Hub - core building block of a modern data architecture 4. Event Hub – Kafka Alternatives? Cloud Services? 5. Summary
- 4. What exactly is an Event Hub?
- 5. Event Hub Event Hub – as a starting point
- 6. Event Hub – Key Capabilities 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients
- 7. Kafka – the most popular Event Hub
- 8. Kafka as an Event Hub Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1 10 12 3 5 6 7 14 8 9 11 12 10 Producer3Producer3 ConsumerConsumer 3 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
- 9. • storage Kafka as an Event Hub • Horizontally scalable, guaranteed order of messages 3 10 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
- 10. • Consumer receives messages via polling • no single-point-of-failure, high availability Kafka as an Event Hub 6 5 7 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
- 11. Event Hub – Key Capabilities supported by Kafka 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients
- 12. Event Hub - core building block of a modern data architecture
- 13. Event Hub Vehicle Environ mental Streaming Data Sources Shop Floor Event Hub – as a starting point
- 14. Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Using Stream Data Integration
- 15. Stream Data Integration – Kafka Connect / StreamSets • declarative style, simple data flows • framework is part of Apache Kafka • Many connectors available • Single Message Transforms (SMT) • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing (micro- batching) • custom sources, sinks, processors
- 16. Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Edge Computing and Stream Data Integration • MQTT as a gateway to Kafka
- 17. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Stream Analytics • Stream-to-Stream Joins • Stream-to-Table Joins • Time Windowed State Management • Event Pattern Detection • Machine Learning Model Execution (Inference) [1]
- 18. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Stream Analytics • Push results back to new topic so other interested parties can use it too!
- 19. Stream Analytics - Kafka Streams • Programmatic API, “just” a Java library • Native streaming • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once • Stream Processing with zero coding using SQL-like language (now supporting push and pull queries) • built on top of Kafka Streams • interactive (CLI) and headless (cmd file) trucking_ driver Kafka Broker Java Application Kafka Streams ksqlDB trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
- 20. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Stream Data Integration to callback to Data Source (to Actuator)
- 21. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Streaming Visualization • ksqlDB pull queries or Kafka Streams Interactive Queries allow to query state of stream processor [2]
- 22. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Using Streaming Visualization
- 23. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Analytics Using Event Hub as a “Data Lake”
- 24. Event Hub as a “Data Lake” - Kafka Storage Local Storage Tiered Storage (with Confluent Enterprise) Broker 1 Broker 2 Broker 3 Broker 1 Broker 2 Broker 3 Object Storage hothot & cold cold
- 25. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics Ingest Data into Data Lake
- 26. Event Hub Stream Analytics Legacy App Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics Legacy System integration?
- 27. Event Hub Stream Analytics Legacy App Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics (Batch-Time) Legacy Systems Integration
- 28. Event Hub Stream Analytics Legacy App Stream Data IntegrationCDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics (Right-Time) Legacy Systems Integration
- 29. Event Hub Stream Analytics Legacy App Stream Data IntegrationCDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics (Right-Time) Legacy Systems Integration • Stream-to-Table join
- 30. Kafka Log Compaction Log Before Compaction Log After Compaction Tail Head
- 31. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Data Lake / DWH Batch Visualize Batch Analytics (Right-Time) Legacy Systems Integration
- 32. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Persistent Storage Integration
- 33. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event-Driven Apps (aka. Microservices) • Microservice participates as both a consumer and producer of events
- 34. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event-Driven, highly decoupled Apps (aka. Microservices) • 2nd microservice consumes events from 1st microservice • Bootstrap new microservices from event history • System-wide CQRS [3]
- 35. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Bi-Directional Legacy Systems Integration • 2nd microservice consumes events from 1st microservice • Bootstrap new microservices from event history • System-wide CQRS [4]AQ
- 36. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Serverless/Function as a Service (FaaS)
- 37. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information!
- 38. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data Integration Legacy Data Sources CDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Shop Floor Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information! Log as a first-class citizen! Turning the database Inside out!
- 39. Event Hub – Kafka Alternatives? Cloud Services?
- 40. • Cloud Services • Cloud Services with Kafka API • Kafka Cloud Services Event Hub - Kafka Alternatives? Cloud Services? • traditional Message Brokers (with a lot of limitations regarding Event Hub capabilities) • Apache Pulsar • Solace • Pravega (Dell Streaming Platform) • Oracle AQ (Kafka API coming) AQ
- 41. Summary
- 42. Ref Architecture Service Event Stream Bulk Data Flow Bulk Source Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Modern Data Platform Event Stream Event Stream
- 43. Reference 1. Stream Processing Concepts and Frameworks 2. Streaming Visualization 3. Building event-driven (Micro)Services with Apache Kafka 4. Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka