Building Event-Driven (Micro)Services with Apache Kafka
4 likes2,616 views
The document details the architecture and concepts involved in building event-driven microservices using the Kafka ecosystem. It discusses the transition from traditional service architectures to microservices, emphasizing the importance of service autonomy, decoupled communications, and the advantages of Kafka as an event broker. The content also covers data integration, historical analytics, and the use of streaming data within microservices, providing a comprehensive overview of best practices and architectural guidelines.
Building Event-Driven (Micro)Services with Apache Kafka
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Building event-driven Microservices with Kafka Ecosystem Guido Schmutz London, 30.5.2018 @gschmutz guidoschmutz.wordpress.com
- 2. Guido Schmutz Working at Trivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: [email protected] Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Building event-driven Microservices with Kafka Ecosystem
- 3. Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. What about streaming sources? 5. What about integrating legacy applications? 6. What about (historical) data analytics? 7. Why Kafka for Event-Driven Microservices? 8. Summary Building event-driven Microservices with Kafka Ecosystem
- 4. Building event-driven Microservices with Kafka Ecosystem Where do we come from?
- 5. Shop Rich UI Shop Backend Application Traditional Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database Building event-driven Microservices with Kafka Ecosystem sync request/response
- 6. Shop UI App Business Activity Service SOA Approach Building event-driven Microservices with Kafka Ecosystem Contract-first Web Services Technical layers offer their own interfaces Reuse on each level Lower layer often wraps legacy code Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic GUI Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP
- 7. Shop UI App Business Activity Service Virtualized SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Building event-driven Microservices with Kafka Ecosystem
- 8. Building event-driven Microservices with Kafka Ecosystem What are Microservices?
- 9. What are Microservices? Building event-driven Microservices with Kafka Ecosystem Tightly Scoped behind clear interfaces Responsible for managing their own data (not necessarily the infrastructure) Should be highly decoupled Independently deployable, self-contained and autonomous SOA done right ?!
- 10. Microservice Approach Building event-driven Microservices with Kafka Ecosystem Tightly Scoped Responsible for managing their own data Highly decoupled Independently deployable, self-contained and autonomous { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST
- 11. Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Building event-driven Microservices with Kafka Ecosystem
- 12. Synchronous World of Request-Response leads to tight, point-to-point couplings Building event-driven Microservices with Kafka Ecosystem problem in lower end of chain have a ripple effect on other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State
- 13. Building event-driven Microservices with Kafka Ecosystem Why not Event-Driven?
- 14. 3 mechanisms through which services interact Request-Driven (Imperative) Event Driven (Functional) Service Logic State Event Consume OrderEvent{iPad} Command ”Order IPad” boolean order(IPad) Event Publish OrderValidatedEvent{iPad} Query ”Retrieve my Orders List<Orders> getAllOrders(for) Building event-driven Microservices with Kafka Ecosystem
- 15. Event-Driven (Async) Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Building event-driven Microservices with Kafka Ecosystem Event Store sync request/response async request/response async, event pub/sub
- 16. What about streaming sources? Building event-driven Microservices with Kafka Ecosystem
- 17. How to work with streaming data sources Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Event Store Location Social Click stream Sensor Data Mobile Apps Weather Data Event Stream Building event-driven Microservices with Kafka Ecosystem
- 18. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Streaming Analytics Architecture BI Tools SQL Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Analytics Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Store Service Building event-driven Microservices with Kafka Ecosystem
- 19. What about integrating legacy applications? Building event-driven Microservices with Kafka Ecosystem
- 20. Data Store Integrate existing systems through CDC Customer Event Store Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Stream Processing Results Stream Processor Reference / Models Dashboard Capture changes directly on database Change Data Capture (CDC) => think like a global database trigger Transform existing systems to event producer Building event-driven Microservices with Kafka Ecosystem
- 21. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Integrate existing systems through CDC BI Tools SQL Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Processor Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Store Service Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture Building event-driven Microservices with Kafka Ecosystem
- 22. What about (historical) data analytics? Building event-driven Microservices with Kafka Ecosystem
- 23. Streaming & (Big) Data Analytics Architecture Event Stream Hadoop Clusterd Hadoop Cluster Big Data Cluster Data Flow Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Processing Cluster Stream Processor State { } API Event Stream Event Stream SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps SQL Export SearchEvent Store Service Location Social Click stream Sensor Data Mobile Apps Weather Data Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import Building event-driven Microservices with Kafka Ecosystem
- 24. Why Kafka for Event-Driven Microservices? Building event-driven Microservices with Kafka Ecosystem
- 25. Apache Kafka – A Streaming Platform Building event-driven Microservices with Kafka Ecosystem High-Level Architecture Distributed Log at the Core Scale-Out Architecture Logs do not (necessarily) forget
- 26. Apache Kafka – Schema Registry Building event-driven Microservices with Kafka Ecosystem Producer Kafka Broker Consumer Schema Registry Kafka Kafka avro schema avroavro avro schema
- 27. Apache Kafka – scalable message processing and more! Building event-driven Microservices with Kafka Ecosystem Source Connector trucking_ driver Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka RESTful RESTful
- 28. Legacy Microservice Change Data Capture (CDC) Building event-driven Microservices with Kafka Ecosystem RDBMS cdc-source trucking_ driver Customer Topic elasticsearc h-sink NoSQL
- 29. Demo Building event-driven Microservices with Kafka Ecosystem Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic REST REST Kafka Customer
- 30. Hold Data for Long-Term – Data Retention Producer 1 Broker 1 Broker 2 Broker 3 1. Never 2. Time based (TTL) log.retention.{ms | minutes | hours} 3. Size based log.retention.bytes 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 --create --topic customers --replication-factor 1 --partitions 1 --config cleanup.policy=compact Building event-driven Microservices with Kafka Ecosystem
- 31. Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) 2015-10-02 11,Peter,Muster,3010,Berne 2016-10-04 12,Paul,Steffen,8001,Zurich 2016-12-02 21,Lisa,Meier,3043,Ittigen 2017-05-03 11,Peter,Muster,3015,Berne 2017-05-03 21,Lisa,Steffen,8001,Zurich 2017-07-03 11,Peter,Muster,3052,Zollikofen Building event-driven Microservices with Kafka Ecosystem 2015-10-02 11,Peter,Muster,3010,Berne 2016-10-04 12,Paul,Steffen,8001,Zurich 2016-12-02 21,Lisa,Meier,3043,Ittigen 2017-05-03 11,Peter,Muster,3015,Berne 2017-05-03 21,Lisa,Steffen,8001,Zurich 2017-07-03 11,Peter,Muster,3052,Zollikofen
- 32. Keep Topics in Compacted Form 0 1 2 3 4 5 6 7 8 9 10 11 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value 3 4 6 8 9 10 K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction Building event-driven Microservices with Kafka Ecosystem V1 V2 V3 V4 V5 V6 V7 V8 V9V10 V11 K1 K3 K4 K5K2 K6
- 33. Keep Topics both in Original and Compacted Form Building event-driven Microservices with Kafka Ecosystem Kafka Broker A B C B A E K A E A C K B E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K Producer Consume and Produce OR TX
- 34. Enrich Stream with Static Data with Kafka Streams Building event-driven Microservices with Kafka Ecosystem Order Topic Consume and Produce Customer Table Customer Topic Custome r Handler Join Customer Local State Order & Customer Topic
- 35. Summary Building event-driven Microservices with Kafka Ecosystem
- 36. Hadoop Clusterd Hadoop Cluster Big Data Summary Billing & Ordering CRM / Profile Marketing Campaigns SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Event Hub Data Flow Data Flow Change Data Capture Parallel Processing Storage Storage RawRefined Results SQL Export Microservice State { } API Stream Processor State { } API Event Stream Event Stream Search Service Location Social Click stream Sensor Data Mobile Apps Weather Data Stream Processing Microservices
- 37. Summary Building event-driven Microservices with Kafka Ecosystem • service autonomy is key in a Microservices Architecture! • not all communication need to be synchronous => separate into • commands • events • queries • Kafka is well suited as an event broker / event store • brings many more interesting features beyond just “message passing”
- 38. References Building event-driven Microservices with Kafka Ecosystem Microservices Blog Series, Ben Stopford, Confluent: • https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: • https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con • https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: • http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: • https://www.youtube.com/watch?v=B1-gS0oEtYc
- 39. Technology on its own won't help you. You need to know how to use it properly. Building event-driven Microservices with Kafka Ecosystem