AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message Queue
Download as pptx, pdf7 likes3,478 views
The document provides an introduction to Apache Kafka, highlighting its role as a scalable and reliable event bus and message queue system. It includes details about Kafka's architecture, features, components, and historical development, alongside various use cases such as messaging, stream processing, and integration with cloud services. The document also elaborates on hands-on demonstration parts and advanced Kafka concepts like producers, consumers, and partitions, aimed at understanding Kafka in practical scenarios.
![HISTORY
• ..- 2010 – creation at Linkedin
• It was designed to provide a high-performance, scalable messaging system which could handle multiple
consumers, many types of data [at high volumes and peaks], and provide for the availability & persistence
of clean, structured data […] in real time.
• 2011 – open source under the Apache Incubator
• October 2012 – top project under Apache Software Foundation
• 2014 – several orginal Kafka engineers founded Confluent
• 2016
• Introduction of Kafka Connect (0.9)
• Introduction of Kafka Streams (0.10)
• Octobermost recent stable release 0.10.1
• Kafka is used by many large corporations:
• Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift Science
• And embraced by many software vendors & cloud providers](https://image.slidesharecdn.com/kafkasig09feb2017-170209154930/85/AMIS-SIG-Introducing-Apache-Kafka-Scalable-reliable-Event-Bus-Message-Queue-17-320.jpg)
![KAFKA STREAMS
• Real Time Event [Stream] Processing integrated into Kafka
• Aggregations & Top-N
• Time Windows
• Continuous Queries
• Latest State (event sourcing)
• Turn Stream (of changes) into Table
(of most recent or current state)
• Part of the state can be quite old
• A Kafka Streams client will have state
in memory
• Always to be recreated from topic partition
log files
• Note: Kafka Streams is relatively new
• Only support for Java clients](https://image.slidesharecdn.com/kafkasig09feb2017-170209154930/85/AMIS-SIG-Introducing-Apache-Kafka-Scalable-reliable-Event-Bus-Message-Queue-25-320.jpg)
![PRODUCING MESSAGES
• The producer sets the partition for each message
• Note: it should talk to the broker who is leader for that partition
• Messages can be produced one-by-one or in batches
• Batches balance latency vs throughput
• A batch can contain messages for different topics & partitions
• Messages can be compressed
• Producers can configure required
acknowledgement level (from broker)
• No (waiting for leader to complete)
• Wait for leader to commit [to file log]
• Wait for all replicas to complete
• Note: messages are serialized to byte array
as the wire format
Producers
Topic
Broker
tcp](https://image.slidesharecdn.com/kafkasig09feb2017-170209154930/85/AMIS-SIG-Introducing-Apache-Kafka-Scalable-reliable-Event-Bus-Message-Queue-32-320.jpg)
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message Queue
- 1. INTRODUCING APACHE KAFKA – SCALABLE, RELIABLE EVENT BUS & ESSAGE QUEUE Maarten Smeets & Lucas Jellema 09 February 2017, Nieuwegein M
- 2. AGENDA INTRODUCTION & OVERVIEW DEMO HANDSON PART 1 - PRODUCING AND CONSUMING MESSAGES (PUB/SUB) DINNER KAFKA: SOME HISTORY, A PEEK UNDER THE HOOD, ROLE IN ARCHITECTURE AND USE CASES KAFKA AND ORACLE HANDSON PART 2 – MORE COMPLEX SCENARIOS AND SOME BACKGROUND & ADMIN
- 4. SENDING MESSAGES TO CONSUMERS • Dependency on producer at design time and at run time • Deal with multiple consumers? • Synchronous (blocking) waits • (how to) Cross technology realms • (how to) Cross host, location, clouds • Availability of consumers • Message delivery guarantees • Scaling, high (peak) volumes
- 5. Producers Consumers MESSAGING – TO DECOUPLE PUB AND SUB
- 6. MESSAGING AS WE KNOW IT • JMS, Oracle Advanced Queuing, IBM MQ, MS MQ, RabbitMQ, MQTT, XMPP, WebSockets, … • Challenges • Costs • Scalability (size and speed) • (lack of) Distribution (and therefore availability) • Complexity of infrastructure • Message delivery guarantees • Lack of technology openness • Deal with temporarily offline consumers • Retain history
- 9. KAFKA TERMINOLOGY • Topic • Message • == ByteArray • Broker • Producer • Consumer Producer Consumer Topic Broker Key Value Time Message
- 11. CONSUMING • Messages are available to consumers only when they have been committed • Kafka does not push • Unlike JMS • Read does not destroy • Unlike JMS Topic • (some) History available • Offline consumers can catch up • Consumers can re-consume from the past • Delivery Guarantees • Ordering maintained • At-least-once (per consumer) by default; at-most-once and exactly-once can be implemented
- 13. WHAT’S SO SPECIAL? • Durable • Scalable • High volume • High speed • Available • Distributed • Open • Quick start • Free (no license costs)
- 15. AGENDA INTRODUCTION & OVERVIEW DEMO HANDSON PART 1 - PRODUCING AND CONSUMING MESSAGES (PUB/SUB) DINNER KAFKA: SOME HISTORY, A PEEK UNDER THE HOOD, ROLE IN ARCHITECTURE AND USE CASES KAFKA AND ORACLE HANDSON PART 2 – MORE COMPLEX SCENARIOS AND SOME BACKGROUND & ADMIN
- 16. AGENDA INTRODUCTION & OVERVIEW DEMO HANDSON PART 1 - PRODUCING AND CONSUMING MESSAGES (PUB/SUB) DINNER KAFKA: SOME HISTORY, A PEEK UNDER THE HOOD, ROLE IN ARCHITECTURE AND USE CASES KAFKA AND ORACLE HANDSON PART 2 – MORE COMPLEX SCENARIOS AND SOME BACKGROUND & ADMIN
- 17. HISTORY • ..- 2010 – creation at Linkedin • It was designed to provide a high-performance, scalable messaging system which could handle multiple consumers, many types of data [at high volumes and peaks], and provide for the availability & persistence of clean, structured data […] in real time. • 2011 – open source under the Apache Incubator • October 2012 – top project under Apache Software Foundation • 2014 – several orginal Kafka engineers founded Confluent • 2016 • Introduction of Kafka Connect (0.9) • Introduction of Kafka Streams (0.10) • Octobermost recent stable release 0.10.1 • Kafka is used by many large corporations: • Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift Science • And embraced by many software vendors & cloud providers
- 18. USE CASES • Messaging & Queuing • Handle fast data (IoT, social media, web clicks, infra metrics, …) • Receive and save – low latency, high volume • Log aggregation • Event Sourcing and Commit Log • Stream processing • Single enterprise event backbone • Connect business processes, applications, microservices
- 19. PLAYS NICE WITH & ARCHITECTURE
- 20. SOME NUMBERS
- 21. KAFKA INCARNATIONS • Kafka Docker Images • Confluent (Spotify, Wurstmeister) • Cloud: • CloudKarafka • IBM BlueMix Message Hub • AWS supports Kafka (but tries to propose Amazon Kinesis Streams) • Google runs Kafka (though tries to push Google Pub/Sub) • Bitnami VMs for many cloud providers such as Azure, GCP, AWS, OPC • Kafka Connectors in many platforms • Azure IoT Hub, Google Pub/Sub, Mule AnyPoint Connector, … • Oracle ….
- 22. KAFKA ECO SYSTEM • Confluent • OpenSource: Native Clients, Camus (link to Hadoop), REST Proxy, Schema Registry • Enterprise: Kafka Ops Dashboard/Control Center, Auto Data Balancing, MultiData Center Replication , • Community • Connectors • Client libraries • …
- 23. KAFKA CONNECT • Kafka Connect is a framework for connectors (aka adapters) that provide bridges for • Producing from specific technologies to Kafka • Consuming from Kafka to specific technologies • For example: • JDBC • Hadoop
- 24. KAFKA CONNECT – CONNECTORS
- 25. KAFKA STREAMS • Real Time Event [Stream] Processing integrated into Kafka • Aggregations & Top-N • Time Windows • Continuous Queries • Latest State (event sourcing) • Turn Stream (of changes) into Table (of most recent or current state) • Part of the state can be quite old • A Kafka Streams client will have state in memory • Always to be recreated from topic partition log files • Note: Kafka Streams is relatively new • Only support for Java clients
- 27. EXAMPLE OF KAFKA STREAMS Topic SelectKey AggregateByKey Join Topic Map (Xform) Publish CountryMessage Continent Name Population Size Set Continent as key Update Top 3 biggest countries As JSON Size in Square Miles, % of entire continent Total area for each continent Topic: Top3CountrySizePerContinent
- 28. countries2.csv Topic Broker Producer SelectKey AggregateByKey Map (Xform) Publish Set Continent as key Update Top 3 biggest countries Topic: Top3CountrySizePerContinent
- 29. EXAMPLE OF KAFKA STREAMS Topic SelectKey AggregateByKey Publish to Topic Topic: Top3CountrySizePerContinent CountryMessage Continent Name Population Size Set Continent as key Update Top 3 biggest countries As JSON Print
- 31. PARTITIONS • Topics are configured with a number of partitions • Storage, serialization, replication, availability, order guarantee are all at partition level • Each partition is an ordered, immutable sequence of records that is continually appended to • Producer can specify the destination partition to write to • Alternatively the partition is determined from the message key or simply by load balancing • Multiple partitions can be written to at the same time
- 32. PRODUCING MESSAGES • The producer sets the partition for each message • Note: it should talk to the broker who is leader for that partition • Messages can be produced one-by-one or in batches • Batches balance latency vs throughput • A batch can contain messages for different topics & partitions • Messages can be compressed • Producers can configure required acknowledgement level (from broker) • No (waiting for leader to complete) • Wait for leader to commit [to file log] • Wait for all replicas to complete • Note: messages are serialized to byte array as the wire format Producers Topic Broker tcp
- 33. CONSUMING • A consumer pulls from a Topic • Consuming can be done in parallel to producing • And many consumers can consume at the same time • Each consumer has a Message Offset per partition • That can be different across consumers • That can be adjusted at any time • Delivery Guarantees • At least once (per consumer) by default; adjust offset when all messages have been processed • At-most-once and exactly-once can be implemented (for example: maintain offset in the same transaction that processes the messages) • Message Retention • Time Based (at least for … time) • Size Based (log files can be no larger than … MB/GB/TB) • Key based aka Log Compaction (retain at least the latest message for each primary key value) Consumers Topic tcp
- 34. CONSUMER GROUPS FOR PARALLEL MESSAGE PROCESSING • Multiple consumers can be in the same Consumer Group • They collaborate on processing messages from a Topic (horizontal scalability) • Each Consumer in the Group receives messages from a different partition • Messages are delivered to only one consumer in the group • Consumers outside the Consumer Group can pull from the same Topic & Partition • And process the same messages Consumers Topic tcp
- 35. CLUSTER – RELIABLE, SCALABLE • A cluster consists of multiple brokers, possibly on multiple server nodes • Each node runs • Apache ZooKeeper to keep track • One or more Kafka Brokers • Each with their own set of storage logs • Each partition lives on one or more brokers (and sets of logs) • Defined through topic replication factor • One is the leader, the others are follower replicas • Clients communicate about a partition with the broker that contains the leader replica for that partition • Changes are committed by the leader, then replicated across the followers Broker Topic Partition Partition Broker Topic Partition Partition Broker Topic Partition Partition Broker Topic Partition Partition
- 36. CLUSTER – RELIABLE, SCALABLE (2) • ZooKeeper has list of all brokers and a list of all topics and partitions (with leader and ISR) • Leader has list of all alive followers (in-synch replicas or ISR) • Follower-replicas consume messages from the leader to synchronize • Similar to normal message consumers • Note: message producers requesting full acknowledgement will get ack once all follower replicates have consumed the message • N-1 replicas can fail without loss of messages Broker Topic Partition Partition Broker Topic Partition Partition Broker Topic Partition Partition Broker Topic Partition Partition
- 37. AGENDA INTRODUCTION & OVERVIEW DEMO HANDSON PART 1 - PRODUCING AND CONSUMING MESSAGES (PUB/SUB) DINNER KAFKA: SOME HISTORY, A PEEK UNDER THE HOOD, ROLE IN ARCHITECTURE AND USE CASES KAFKA AND ORACLE HANDSON PART 2 – MORE COMPLEX SCENARIOS AND SOME BACKGROUND & ADMIN
- 38. ORACLE AND KAFKA • On premises • Service Bus Kafka transport (demo!) • Stream Analytics Kafka Adapter (demo!) • GoldenGate for Big Data handler for Kafka • Data Integrator (coming soon) • Cloud • Elastic Big Data & Streaming platform • Event Hub (coming soon)
- 39. GOLDENGATE FOR BIG DATA
- 40. GOLDENGATE FOR BIG DATA
- 41. DATA INTEGRATOR
- 42. ELASTIC BIG DATA & STREAMING PLATFORM
- 43. EVENT HUB
- 44. EVENT HUB
- 45. EVENT HUB
- 46. AGENDA INTRODUCTION & OVERVIEW DEMO HANDSON PART 1 - PRODUCING AND CONSUMING MESSAGES (PUB/SUB) DINNER KAFKA: SOME HISTORY, A PEEK UNDER THE HOOD, ROLE IN ARCHITECTURE AND USE CASES KAFKA AND ORACLE HANDSON PART 2 – MORE COMPLEX SCENARIOS AND SOME BACKGROUND & ADMIN
- 47. HANDS ON PART 2 • Continue part 1 • Java and/or Node consuming/producing • Some Admin & advanced stuff • Partitions • Multiple producers, multiple consumers • New consumer, go back in time • Expiration of messages • Multi-broker, Cluster configuration, ZooKeeper
- 48. • Resources: https://github.com/MaartenSmeets/kafka-workshop • Blog: technology.amis.nl On Oracle, Cloud, SQL, PL/SQL, Java, JavaScript, Continuous Delivery, SOA, BPM & more • Email: [email protected] , [email protected] • : @MaartenSmeetsNL , @lucasjellema • : smeetsm , lucas-jellema • : www.amis.nl, [email protected] +31 306016000 Edisonbaan 15, Nieuwegein
Editor's Notes
- #22: http://stackoverflow.com/questions/35861501/kafka-in-docker-not-working Docker images from Confluent: https://hub.docker.com/r/confluent/kafka/
- #23: http://docs.confluent.io/2.0.0/platform.html
- #24: http://docs.confluent.io/2.0.0/platform.html https://www.confluent.io/blog/apache-kafka-getting-started/