Apache Kafka as Event Streaming Platform for Microservice Architectures
21 likes3,564 views
This document provides an overview of Apache Kafka as a key technology for event-driven architectures in modern microservice applications, emphasizing its scalability and real-time streaming capabilities. It outlines the transformation from traditional data processing methods to a unified event streaming platform, showcasing various use cases and benefits such as improved customer experience and operational efficiency. The document also discusses Kafka's features, including fault tolerance, data persistence, and its integration with other tools and frameworks.
![Confluents Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)](https://image.slidesharecdn.com/201905kafkaasstreamingplatformkaiwaehner-190425051206/85/Apache-Kafka-as-Event-Streaming-Platform-for-Microservice-Architectures-19-320.jpg)
Apache Kafka as Event Streaming Platform for Microservice Architectures
- 1. 1 Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform for Microservice Architectures Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. 2 A need for integration in every enterprise Search Sensors / IoT RDBMS Monitoring NoSQLReal-time Analytics Data Warehouse Apps Microservices Big Data Integration
- 3. 3 Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency The World has Changed Mobile Cloud Microservices Internet of Things Machine Learning
- 4. 4 Before: many ad hoc pipelines Search Security Fraud Detection Application User Tracking Operational Logs Operational Metrics Big Data App Data Warehouse Mainframes NoSQL Relational DB Databases Storage Interfaces Monitoring App Databases Storage Interfaces
- 5. 5 After: streaming platform with Kafka Search Security Fraud Detection Application User Tracking Operational Logs Operational MetricsMainframes Relational DB Big Data App Monitoring App Data Warehouse Event Streaming Platform NoSQL
- 6. Events What is an event?
- 7. Events
- 8. 8 Events A Sale An Invoice A Trade A Customer Experience
- 9. 9 Where are they? Events haven’t had a proper home in infrastructure or in code. They are implicit. Here!
- 10. 10 Haven’t we seen all this before?
- 11. 11 What’s different this time around? (Published in 2009) (Published in 2004)
- 14. A Streaming Platform is the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and consumers Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservice s DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Stream processing appsStream processing apps Stream processing apps
- 16. The beginning of a new Era https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case. This is why Kafka was created!
- 17. 17 ● Global-scale ● Real-time ● Persistent Storage ● Stream Processing Apache Kafka: The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
- 18. Apache Kafka at Scale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 19. Confluents Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 20. Confluent Partner Briefing 20 Example: An Airbnb Booking Event Booked event happens { rentalId:4124, rentalPrice: 58, userId: 5893381 …. } Rental availability Rental pricing Recommended experiences Account history Account Updates Store Updates Report Updates User engagement Localized supply Topic: rentalOrders
- 21. A Modern, Distributed Platform for Data Streams. Messaging + Storage + Processing!
- 22. Apache Kafka is made up of distributed, immutable, append-only commit logs
- 23. Apache Kafka - A Distributed Commit Log Writers Kafka cluster Readers
- 24. Scalability of a filesystem • Hundreds of MB/s throughput • Many TB per server • Commodity hardware
- 25. Guarantees of a Database • Strict ordering • Persistence
- 26. Distributed by design • Replication • Fault Tolerance • Partitioning • Elastic Scaling
- 29. P Producing to Kafka Time C2 C3C1
- 30. Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 31. Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams
- 32. Kafka Connect is an integration framework on top of Kafka‘s Core
- 33. Kafka’s Streams API: Build real-time applications for your core business Kafka’s Streams API • To build real-time applications for your core business • Easiest way to process data in Apache Kafka • Apps are standard Java applications that run on client machines • Powerful yet easy-to-use library, part of Apache Kafka • https://github.com/apache/kafka/tree/trunk/streams Streams API Your App Kafka Cluster
- 34. Example: complete app, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
- 35. Confluent Platform Operations and Security Development & Stream Processing Support,Services,Training&Partners Apache Kafka Security plugins | Role-Based Access Control Control Center | Replicator | Auto Data Balancer | Operator Connectors Clients | REST Proxy MQTT Proxy | Schema Registry KSQL Connect Continuous Commit Log Streams Complete Event Streaming Platform Mission-critical Reliability Freedom of ChoiceDatacenter Public Cloud Confluent Cloud Self-Managed Software Fully-Managed Service Confluent Delivers a Mission-Critical Event Streaming Platform
- 36. KSQL – A Streaming SQL Engine for Apache Kafka
- 37. 3939 Confluent Control Center (C3) Monitors all pipelines end-to-end • Lost Messages? • Duplicates? • Latency Issues? • What is the problem? • Where is the problem? • Etc.
- 38. 4040 KSQLKafka Streams Event Streaming with Confluent’s Event Streaming Platform Splunk Security Fraud Detection Application User Tracking Operational Logs Operational MetricsMainframes Oracle DB Hadoop Business App Monitoring App Confluent Control Center Kafka Mongo DB Cassandra Kafka Connect Schema Registry Rest Proxy
- 39. 41C O N F I D E N T I A L Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design for your Event Steaming Platform Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable!
- 41. 4343 Best-of-breed Platforms, Partners and Services for Multi-cloud Streams Private Cloud Deploy on bare-metal, VMs, containers or Kubernetes in your datacenter with Confluent Platform and Confluent Operator Public Cloud Implement self-managed in the public cloud or adopt a fully managed service with Confluent Cloud Hybrid Cloud Build a persistent bridge between datacenter and cloud with Confluent Replicator Confluent Replicator VM SELF MANAGED FULLY MANAGED
- 42. 44 Confluent’s Streaming Maturity Model - where are you? Value Maturity (Investment & time) 2 Enterprise Streaming Pilot / Early Production Pub + Sub Store Process 5 Central Nervous System 1 Developer Interest Pre-Streaming 4 Global Streaming 3 SLA Ready, Integrated Streaming Projects Platform
- 43. 45Highly Scalable Microservices with Apache Kafka + Mesos Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn Questions? Feedback? Please contact me!