Building Event-Driven (Micro) Services with Apache Kafka
4 likes1,411 views
The document discusses building event-driven microservices using Apache Kafka. It describes how microservices can interact asynchronously through events published to an event hub like Apache Kafka. This allows for loose coupling between services and the ability to integrate new services by consuming past events from the event log. The document also discusses how Apache Kafka can be used for change data capture from legacy systems, streaming data integration, and providing a unified platform for real-time event processing and historical analytics.
Building Event-Driven (Micro) Services with Apache Kafka
- 1. gschmutz Building event-driven (Micro)Services with Apache Kafka SEACON 2019 – 24.5.2019 Guido Schmutz ([email protected]) gschmutz http://guidoschmutz.wordpress.com
- 2. gschmutz Agenda 1. Where do we come from? 2. What about Microservices? 3. Can we do better? 4. How does Apache Kafka help? 5. What about Streaming Data Sources? 6. What about integrating Legacy Applications? 7. What about (historical) data analytics? 8. Summary
- 3. gschmutz Guido Schmutz Working at Trivadis for more than 22 years Oracle Groundbreaker Ambassador & Oracle ACE Director Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: [email protected] Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 158th edition
- 4. gschmutz Where do we come from?
- 5. gschmutz Shop Rich UI Shop Backend Application “Layered Architecture” Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database sync request/response
- 6. gschmutz Shop UI App Business Activity Service SOA Approach Contract-first Web Services Technical layers offer their own interfaces Reuse on each level Lower layer often wraps legacy code Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic GUI Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP
- 7. gschmutz Business Activity Service Virtualized SOA Approach The raise of the Enterprise Service Bus (ESB) Search BAS Customer DAO Order DAO Order BAS Business Entity Service Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Shop UI App Shop UI UI Logic GUI Shop Web App Shop UI UI Logic GUI X
- 8. gschmutz Shop UI App Business Activity Service Orchestrated & Virtualized SOA Approach – Sync / Async The raise of orchestration engines (BPEL & BPMN) Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus X Orchestration
- 10. gschmutz Customer Microservice Microservice Approach Tightly Scoped behind interfaces Highly decoupled Independently deployable Bounded Context/Aggregate (DDD) Responsible for their data (does not mean they need their own DB!) Smart Endpoints and Dump Pipes just SOA done right ? What about capabilities the “smart pipes” provided? { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST
- 11. gschmutz Synchronous Request-Response lead to tight, point-to- point couplings problem in lower end of chain have a ripple effect on other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State RESTRESTRESTREST REST REST REST
- 12. gschmutz Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway X
- 13. gschmutz Microservice Approach with Side Car (i.e. K8s & Istio) Side-car can provide: • retry • load-balancing • circuit breaker, throttling • security • … Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Side Car Side Car Side Car REST RESTRESTREST
- 14. gschmutz Microservice Approach with Side Car (i.e. K8s & Istio) • Side-car and it’s advanced routing capabilities can be used to switch to new service version Service 2Service 1 { } API Logic { } API Logic StateState Service 3 – v1 { } API Logic State Side Car Side Car Service 3 – v2 { } API Logic State RESTREST REST REST
- 15. gschmutz Side Car provides lot of value …. but • we still have to change the “data owner” service …. if a new service requires the same information Service 1 { } API Logic State Service 2 { } API Logic State Service 4 Logic State Service 3 { } API Logic State { } API New Service Logic State { } API REST REST REST REST REST
- 16. gschmutz Can we do even(t) better?
- 17. gschmutz Events Distribute to all handlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result 3 mechanisms through which services can interact Commands Route to single handler Use consistent hashing Provide Result Adapted from Axon IQ
- 18. gschmutz Stock Microservice Event-Driven (Async) Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic REST REST REST REST API Gateway Event Hub sync request/response async request/response async, event pub/sub Customer Mat View Event Hub • Pub / Sub messaging • Topics schema-less • Message coupling between services • Domain Events from DDD • This is not event sourcing!
- 19. gschmutz How does Apache Kafka help?
- 20. gschmutz Apache Kafka – highly scalable message broker Kafka Cluster Consumer Consumer Consumer Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer Producer Producer kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Event Hub
- 21. gschmutz Log Compaction – specially useful for Master Data 0 1 2 3 4 5 6 7 8 9 10 11 11 12 11 14 13 12 11 15 15 12 16 12 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 Offset Key Value 3 4 6 8 9 10 14 13 11 15 12 16 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction V1 V2 V3 V7 V5 V6 V4 V8 V9 V10 V11 11 13 14 15 12 16 Id: 11 Name: Peter Muster Street: Münstergasse 5 City: 3013 Bern Category: B Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: B Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: A Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: A 11 12 V12
- 22. gschmutz Example Customer Microservice { } Customer API CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Schema Registry Schema Customer (compacted) Implementation: https://github.com/gschmutz/event-driven-microservices-demo
- 23. gschmutz Example @Configuration public class KafkaConfig { private String bootstrapServers; private String schemaRegistryURL; @Bean public Map<String, Object> producerConfigs() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL); return props; } @Bean public ProducerFactory<String, Customer> producerFactory() { .. } @Bean public KafkaTemplate<String, Customer> kafkaTemplate() { return new KafkaTemplate<>(producerFactory()); } @Component public class CustomerEventProducer { @Autowired private KafkaTemplate<String, Person> kafkaTemplate; @Value("${kafka.topic.customer}") String kafkaTopic; public void produce(Customer customer) { kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer); } }
- 24. gschmutz Adding a new service by bootstrapping from Event Hub Customer Search Microservice { } Customer API CustomerCustomer Logic REST Schema Registry Schema Consume from Offset 0 Customer Microservice { } Customer API CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Customer (compacted) Implementation: https://github.com/gschmutz/event-driven-microservices-demo
- 25. gschmutz What about Streaming Data Sources?
- 26. gschmutz How to work with streaming data sources Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Event Hub Location Social Click stream Sensor Data Mobile Apps Weather Data Event Stream
- 27. gschmutz Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Streaming Processing & Microservices Architecture BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Analytics Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service
- 28. gschmutz Apache Kafka – scalable message processing and more! Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka
- 30. gschmutz Kafka Streams • Programmatic API, “just” a Java library • Native streaming • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once KTable<Integer, Customer> customers = builder.stream(”customer"); KStream<Integer, Order> orders = builder.stream(”order"); KStream<Integer, String> joined = orders.leftJoin(customers, …); joined.to(”orderEnriched"); trucking_ driver Kafka Broker Java Application Kafka Streams
- 31. gschmutz KSQL • Stream Processing with zero coding using SQL-like language • part of Confluent Platform (community edition) • built on top of Kafka Streams • interactive (CLI) and headless (command file) CREATE STREAM customer_s WITH (kafka_topic='customer', value_format='AVRO'); SELECT * FROM customer_s WHERE address->country = 'Switzerland'; ... trucking_ driver Kafka Broker KSQL Engine Kafka Streams KSQL CLI Commands
- 32. gschmutz What about integrating Legacy Applications?
- 33. gschmutz Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Integrate existing systems through CDC BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Processor Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture
- 34. gschmutz Data Store Integrate existing systems through CDC Capture changes directly on database Change Data Capture (CDC) => think like a database-wide trigger Make existing legacy systems an event producer Customer Event Hub Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Stream Processing Results Stream Processor Reference / Models Dashboard
- 35. gschmutz Legacy Microservice Change Data Capture (CDC) with Kafka Broker and Kafka Connect RDBMS cdc-source trucking_ driver Kafka Broker elasticsearch- sink NoSQL Customer Topic
- 36. gschmutz Microservice Change Data Capture (CDC) with Kafka Broker and Kafka Connect cdc-source elasticsearch- sink NoSQL Enhance Kafka Broker Customer Topic CustomerEnhanced Topic State Legacy RDBMS
- 37. gschmutz What about (historical) data analytics?
- 38. gschmutz Streaming & (Big) Data Analytics Architecture Event Stream Hadoop Clusterd Hadoop Cluster Big Data Cluster D ata Flow Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Processing Cluster Stream Processor State { } API Event Stream Event Stream SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps SQL Export SearchEvent Hub Service Location Social Click stream Sensor Data Mobile Apps Weather Data Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import
- 39. gschmutz Summary
- 40. gschmutz Hadoop Clusterd Hadoop Cluster Big Data Summary Billing & Ordering CRM / Profile Marketing Campaigns SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Event Hub D ata Flow D ata Flow Change Data Capture Parallel Processing Storage Storage RawRefined Results SQL Export Microservice State { } API Stream Processor State { } API Event Stream Event Stream Search Service Location Social Click stream Sensor Data Mobile Apps Weather Data Stream Processing Microservices
- 41. gschmutz Summary • not all communication need to be synchronous => distinguish into commands, events, queries • Events should use a schema with support for backward and forward compatibility • Kafka handles event streaming very well • brings many more interesting features beyond just “message passing”, i.e. Log compaction • Kafka broker is not a full-fledged Event Store • For Event Sourcing additional capabilities are needed (Kafka Streams, Axon, …) • See also: Kafka as an Event Store: is it good enough? (slides, video)
- 42. gschmutz Further Information • Kafka as an Event Store: is it good enough?, Guido Schmutz, Trivadis: https://www.slideshare.net/gschmutz/kafka- as-an-event-store-is-it-good-enough • Microservices Blog Series, Ben Stopford, Confluent: https://www.confluent.io/blog/tag/microservices • Apache Kafka for Microservices: A Confluent Online Talk Series: https://www.confluent.io/landing- page/microservices-online-talk-series/ • Schemas, Contracts, and Compatibility, Gwen Shapira, Confluent: https://www.confluent.io/blog/schemas- contracts-compatibility • Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann: https://www.confluent.io/blog/put-several-event-types-kafka-topic/ • Turning the database inside-out with Apache Samza, Martin Kleppmann: https://www.confluent.io/blog/turning-the- database-inside-out-with-apache-samza/ • Immutability Changes Everything, Pat Helland, Salesforce: http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
- 43. gschmutz Technology on its own won't help you. You need to know how to use it properly. gschmutz
- 44. gschmutz Event Sourcing persists the state of an aggregate as a sequence of state-changing events Each event describes a state change that occurred to the aggregate in the past new event is appended to the list of events an aggregate’s current state is reconstructed by replaying the events => a.k.a ”rehydration” Rehydration also needed for queries Event Store ServiceApp UI UI Logic Command API & Handler Event Handler(s) Service Subscribe publish publish apply (append) REST Data Storage trigger replycommand command 1 2 3 4 5 5
- 45. gschmutz Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern Combines best of Event Sourcing and CQRS Project events published by Event Store into Read Model (Materialized Views) Write Model and Read Model might only support eventual consistency AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project 1 Event Store publish apply (append) trigger reply 2 3 4 5 5 6
- 46. gschmutz Event Store Capabilities 1. Append Events efficiently 2. Read aggregate’s events in order 3. Full Sequential Read (over all aggregates) 4. Consistent writes 5. Event versioning 6. Subscribable event stream 7. Correction events (O) 8. Ingestion & event time, bi-temporal (O) 9. Adhoc-Query on event store (O) 10. Snapshot Optimization (O) 11. High-Availability and Reliability (O)
- 47. gschmutz Kafka as an Event Store # Capability Kafka Broker 1 Append events efficiently yes 2 Read aggregate’s events in order not efficiently 3 Full sequential Read yes 4 Consistent Writes no 5 Event Versioning yes (if Avro is used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header 9 Snapshot Optimization (O) no 10 Ad-Hoc Query on Events (O) no 11 High-Availability and Reliability (O) yes