Building Event Driven (Micro)services with Apache Kafka
6 likes2,577 views
The document outlines a comprehensive approach to building event-driven microservices utilizing Apache Kafka. It discusses the evolution from traditional layered architectures to microservices, emphasizing decoupling, independence, and event-driven interactions. The presentation also covers data streaming, integration of legacy systems, and the analytics of historical data, while highlighting Kafka's scalability and robust features for message processing.
![Apache Avro
• http://avro.apache.org/docs/current/
• Apache Avro™ is a compact, fast,
binary data serialization system
invented by the makers of Hadoop
• Avro relies on schemas.
• When data is read, the schema used
when writing needs to be present
• container file for storing persistent
data
• Works both with code generation and
in a dynamic manner
{
"type" : "record",
"namespace" : "com.trivadis.avro.customer.v1",
"name" : ”Customer",
"description" : "the representation of a customr",
"fields" : [
{ "name": "id", "type": ”string" },
{ "name": "firstName", "type": "string" },
{ "name": "lastName", "type": "string" },
{ "name" : "title", "type" : {
"type" : "enum", "name" : "TitleEnum",
"symbols" : ["Unknown", "Mr", "Mrs", "Ms"]
}
},
{ "name": "email", "type": ["null","string"] },
{ "name": "dateOfBirth", "type": {
"type": "int", "logicalType": "date" } },
{ "name" : "addresses", ... }
]
}](https://image.slidesharecdn.com/buildingevent-drivenmicroserviceswithapachekafka-v1-191220084722/85/Building-Event-Driven-Micro-services-with-Apache-Kafka-24-320.jpg)
Building Event Driven (Micro)services with Apache Kafka
- 1. BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz.wordpress.com@gschmutz Building event-driven (Micro)Services with Apache Kafka Guido Schmutz OiO-Hauskonferenz – 19.12.2019
- 2. Agenda 1. Where do we come from? 2. What about Microservices? 3. Can we do better? 4. How does Apache Kafka help? 5. What about Streaming Data Sources? 6. What about integrating Legacy Applications? 7. What about (historical) data analytics? 8. Summary
- 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 23 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 178th edition
- 4. Where do we come from?
- 5. Shop Rich UI Shop Backend Application “Layered Architecture” Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database sync request/response
- 6. Shop UI App Business Activity Service SOA Approach • Contract- first Web Services • Technical layers offer their own interfaces • Reuse on each level • Lower layer often wraps legacy code Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic GUI Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP
- 7. Business Activity Service Virtualized SOA Approach • The raise of the Enterprise Service Bus (ESB) Search BAS Customer DAO Order DAO Order BAS Business Entity Service Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Shop UI App Shop UI UI Logic GUI Shop Web App Shop UI UI Logic GUI X
- 8. Shop UI App Business Activity Service Orchestrated & Virtualized SOA Approach – Sync/Async • The raise of orchestration engines • (BPEL & BPMN) Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus X Orchestration
- 10. Customer Microservice Microservice Approach • Tightly Scoped behind interfaces • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data (does not mean they need their own DB!) • Smart Endpoints and Dump Pipes • just SOA done right ? What about capabilities the “smart pipes” provided? { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST
- 11. Service 1 Service 2 Service 3 Service 4 Smart Endpoints and Dumb Pipes! - no longer a complex Integration in the middle! Integration X Service Bus Service Orchestration Service 1 Service 2 Service 3 Service 4 X X X X
- 12. Synchronous Request-Response lead to tight, point- to-point couplings problem in lower end of chain have a ripple effect on other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State RESTRESTRESTREST REST REST REST
- 13. Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway X
- 14. Microservice Approach with Side Car (i.e. K8s & Istio) like AOP but on infrastructure can be framework or proxy (side car) based Service Mesh provides: • retry, load-balancing, circuit breaker, throttling, security, tracing, … Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Side Car Side Car Side Car REST RESTRESTREST
- 15. Microservice Approach with Side Car (i.e. K8s & Istio) • Side-car and it’s advanced routing capabilities can be used to switch to new service version Service 2Service 1 { } API Logic { } API Logic StateState Service 3 – v1 { } API Logic State Side Car Side Car Service 3 – v2 { } API Logic State RESTREST REST REST
- 16. Side Car provides lot of value …. but • we still have to change the “data owner” service …. if a new service requires the same information Service 1 { } API Logic State Service 2 { } API Logic State Service 4 Logic State Service 3 { } API Logic State { } API New Service Logic State { } API REST REST REST REST REST
- 17. Can we do even(t) better?
- 18. Events Distribute to all handlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result Three mechanisms through which services can interact Commands Route to single handler Use consistent hashing Provide Result Adapted from Axon IQ
- 19. Stock Microservice Event-Driven (Async) Microservice Event Hub • Pub / Sub messaging • Message transparent to topic • Message coupling • Domain Events from DDD • This is event-driven interaction but not event sourcing! Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic REST REST REST REST API Gateway Event Hub sync request/response async request/response async, event pub/sub Customer Mat View
- 20. Customer Microservice Stock Microservice Event-Driven (Async) Microservice Consumer Microservice is responsible for adapting to subscribed event, if necessary does not have to use same technologies Anti Corruption Layer (ACL) { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic { } Stock API Stock Stock Logic REST REST REST REST Customer Mat View Event Hub Consume X Consume X Consume X Consume X
- 21. How does Apache Kafka help?
- 22. Apache Kafka – highly scalable message broker Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3 Event Hub
- 23. Example Customer Microservice { } Customer API CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Schema Registry Schema Customer (compacted) Implementation: https://github.com/gschmutz/event-driven-microservices-demo in functional terms this is an implementation of CQRS pattern => System-Wide CQRS
- 24. Apache Avro • http://avro.apache.org/docs/current/ • Apache Avro™ is a compact, fast, binary data serialization system invented by the makers of Hadoop • Avro relies on schemas. • When data is read, the schema used when writing needs to be present • container file for storing persistent data • Works both with code generation and in a dynamic manner { "type" : "record", "namespace" : "com.trivadis.avro.customer.v1", "name" : ”Customer", "description" : "the representation of a customr", "fields" : [ { "name": "id", "type": ”string" }, { "name": "firstName", "type": "string" }, { "name": "lastName", "type": "string" }, { "name" : "title", "type" : { "type" : "enum", "name" : "TitleEnum", "symbols" : ["Unknown", "Mr", "Mrs", "Ms"] } }, { "name": "email", "type": ["null","string"] }, { "name": "dateOfBirth", "type": { "type": "int", "logicalType": "date" } }, { "name" : "addresses", ... } ] }
- 25. Avro and Confluent Schema Registry
- 26. Example @Configuration public class KafkaConfig { private String bootstrapServers; private String schemaRegistryURL; @Bean public Map<String, Object> producerConfigs() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL); return props; } @Bean public ProducerFactory<String, Customer> producerFactory() { .. } @Bean public KafkaTemplate<String, Customer> kafkaTemplate() { return new KafkaTemplate<>(producerFactory()); } @Component public class CustomerEventProducer { @Autowired private KafkaTemplate<String, Person> kafkaTemplate; @Value("${kafka.topic.customer}") String kafkaTopic; public void produce(Customer customer) { kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer); } }
- 27. Adding a new service by bootstrapping from Event Hub Customer Search Microservice { } Customer API CustomerCustomer Logic REST Schema Registry Schema Consume from Offset 0 Customer Microservice { } Customer API CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Customer (compacted) Implementation: https://github.com/gschmutz/event-driven-microservices-demo • assuming Event Hub keeps message log as long as needed, new services can be bootstrapped with information from other services just out of the Event Hub
- 28. Log Compaction – specially useful for Master Data 0 1 2 3 4 5 6 7 8 9 10 11 11 12 11 14 13 12 11 15 15 12 16 12 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 Offset Key Value 3 4 6 8 9 10 14 13 11 15 12 16 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction V1 V2 V3 V7 V5 V6 V4 V8 V9 V10 V11 11 13 14 15 12 16 Id: 11 Name: Peter Muster Street: Münstergasse 5 City: 3013 Bern Category: B Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: B Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: A Id: 11 Name: Peter Muster Street: Chaumonweg 108 City: 3098 Spiegel Category: A 11 12 V12
- 29. Event-Driven (Async) Microservice • Event Hub and Service Mesh • not yet feasible, as Event Hub protocols are not supported by todays service mesh • could be used for versioning, security, tracing, monitoring … sync request/response async request/response async, event pub/sub Side Car Side Car Side Car Stock Microservice Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic { } Stock API Stock Stock Logic REST REST REST REST Customer Mat View Side Car Event Hub
- 30. What about Streaming Data Sources?
- 31. How to work with streaming data sources Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Event Hub Location Social Click stream Sensor Data Mobile Apps Weather Data Event Stream
- 32. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Streaming Processing & Microservices Architecture BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Analytics Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service
- 33. Apache Kafka – scalable message processing and more! Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka ksqlDB
- 35. Kafka Streams • Programmatic API, “just” a Java library • Native streaming • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once KTable<Integer, Customer> customers = builder.stream(”customer"); KStream<Integer, Order> orders = builder.stream(”order"); KStream<Integer, String> joined = orders.leftJoin(customers, …); joined.to(”orderEnriched"); trucking_ driver Kafka Broker Java Application Kafka Streams
- 36. ksqlDB • Stream Processing with zero coding using SQL- like language (now supporting push and pull queries) • part of Confluent Platform (community edition) • built on top of Kafka Streams • interactive (CLI) and headless (command file) CREATE STREAM customer_s WITH (kafka_topic='customer', value_format='AVRO'); SELECT * FROM customer_s WHERE address->country = 'Switzerland'; ... trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull ksqlDB
- 37. What about integrating Legacy Applications?
- 38. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Integrate existing systems through CDC BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Processor Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture
- 39. Data Store Integrate existing systems through CDC • Capture changes directly on database • Change Data Capture (CDC) => think like a database-wide trigger • Make existing legacy systems an event producer Customer Event Hub Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Stream Processing Results Stream Processor Reference / Models Dashboard
- 40. Legacy Microservice Change Data Capture (CDC) with Kafka Broker and Kafka Connect RDBMS cdc-source trucking_ driver Kafka Broker elasticsearch- sink NoSQL Customer Topic
- 41. Microservice Change Data Capture (CDC) with Kafka Broker and Kafka Connect cdc-source elasticsearch- sink NoSQL Enhance Kafka Broker Customer Topic CustomerEnhanced Topic State Legacy RDBMS
- 42. What about (historical) data analytics?
- 43. Streaming & (Big) Data Analytics Architecture Event Stream Hadoop Clusterd Hadoop Cluster Big Data Cluster D ata Flow Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Processing Cluster Stream Processor State { } API Event Stream Event Stream SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps SQL Export SearchEvent Hub Service Location Social Click stream Sensor Data Mobile Apps Weather Data Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import
- 44. Summary
- 45. Hadoop Clusterd Hadoop Cluster Big Data Summary Billing & Ordering CRM / Profile Marketing Campaigns SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Event Hub D ata Flow D ata Flow Change Data Capture Parallel Processing Storage Storage RawRefined Results SQL Export Microservice State { } API Stream Processor State { } API Event Stream Event Stream Search Service Location Social Click stream Sensor Data Mobile Apps Weather Data Stream Processing Microservices
- 46. Summary • not all communication need to be synchronous => distinguish into commands, events, queries • Events should use a schema with support for backward and forward compatibility • Kafka handles event streaming very well • brings many more interesting features beyond just “message passing”, i.e. Log compaction • Kafka broker is not a full-fledged Event Store • For Event Sourcing additional capabilities are needed (Kafka Streams, Axon, …) • See also: Kafka as an Event Store: is it good enough? (slides, video)
- 47. Further Information • Kafka as an Event Store: is it good enough?, Guido Schmutz, Trivadis: https://www.slideshare.net/gschmutz/kafka-as-an-event-store-is-it-good-enough • Microservices Blog Series, Ben Stopford, Confluent: https://www.confluent.io/blog/tag/microservices • Apache Kafka for Microservices: A Confluent Online Talk Series: https://www.confluent.io/landing- page/microservices-online-talk-series/ • Schemas, Contracts, and Compatibility, Gwen Shapira, Confluent: https://www.confluent.io/blog/schemas-contracts-compatibility • Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann: https://www.confluent.io/blog/put-several-event-types-kafka-topic/ • Turning the database inside-out with Apache Samza, Martin Kleppmann: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ • Immutability Changes Everything, Pat Helland, Salesforce: http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf