Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschool, Spring 2022)

Classificatie: vertrouwelijk
Microservices in
real life – with
Apache Kafka,
Node & Dapr.io
Part 2
March 2022
Lucas Jellema
Architect & CTO AMIS | Conclusion

Microservices in real life – with Node & Dapr.io
Founded in 1991 by students
from University of Twente –
Aircraft Maintenance Information
System (AMIS)
80 colleagues,
located in
Nieuwegein
the core of what we do:
working with Data.
partnering with peers
and companies in
several countries
Lucas Jellema (2002)
Cloud Solution Architect & CTO
lucas.jellema@amis.nl | technology.amis.nl | @lucasjellema | lucas-jellema

Data Engineering, Data Analytics
(& Data Science)
Data &
Application
Integration
web
applications
Internet of Things
cloud, DevOps, PaaS,
streaming, microservices,
Software Studio, database,
software engineering
Oracle, Microsoft Azure,
open source, Java, SQL,
NodeJS, Python,Kafka
React/Angular

Classificatie: Public
Publiek
Overview
• Part One (last week)
• Microservices recap
• Dapr.io – personal assistant for applications & distributed application runtime
• Handson with Dapr.io
• Quick Intro to programming in Node[.js]
• Handson Microservice implementation with Node and Dapr.io
• Part Two (today)
• Introduction of Apache Kafka (“Twitter for systems”)
• Asynchronous interactions through Apache Kafka – concepts, terminology
• Handson with Apache Kafka and Node
• Real world use cases and scenarios with Apache Kafka
• Handson Multi-microservice set up with Apache Kafka, Node and Dapr.io

Publiek
Assumed is
• A development environment with VS Code, Docker and Docker Composer,
the ability (permissions) to install software
• Knowledge of
• HTTP, REST, JSON
• Containers, Docker (and Kubernetes)
• Java or C#
• SQL and a database (MySQL or PostgreSQL or SQL Server)
• perhaps Message Broker/Event Queue (RabbitMQ?), Cache (Redis?)
• Cloud fundamentals
• Microservices concepts

Publiek
It would be so nice if I could
publish my ideas and actions,
accessible near instantly for
everyone who is interested
Heck, I do not even know these people
and they may not know me [personally]
– just my pearls of wisdom. And if they
are late to the party, they can also
check out the historic archives of my
eloquence
Without fretting about the numbers of
readers involved and whether they are
in the same time zone as me and online
when I publish my messages – and
which device they use

Publiek
• Decoupled communication
• 0, 1 or many followers
• Scalable number of messages (and parties)
• Reliable (mostly available, few messages lost)
• Full history
• Open: cross device, cross location
• Not Sub-second, near real-time fast
• Rate limited (#messages/minute)
• Size limited (140-280 characters)
• Format limited (text)
• Not for private interactions
• Not (really) for programmatic use

Publiek
What does the Twitter for System Driven Event Interaction
look like?
Microservices & Apache Kafka - 24 maart 2021
• Decoupled communication – organized per topic
• 0, 1 or many Consumers per Topic
• Reliable (distributed)
• Full history
• Open: libraries in many technologies & REST APIs

Publiek
What does the Twitter for System Driven Event Interaction
look like?
• Decoupled communication – organized per topic
• 0, 1 or many Consumers per Topic
• Reliable (distributed)
• Full history
• Open: libraries in many technologies & REST APIs
• Near real-time fast
• No Rate Limit
• No enforced size limit
• Anything goes (it’s all byte[])
• On premises or in cloud, private or trusted
• Very much for programmatic use

Publiek
Events
Producers
Consumers
Robust, QoS, Scalable,
Fast,
History Retention
Order preserved
Containerized/Cloud-
enabled
Open

Publiek
Messaging as we know it
• JMS, IBM MQ, MS MQ, RabbitMQ, MQTT, XMPP, WebSockets, Apache
ActiveMQ, …
• Challenges
• Costs
• Scalability (size and speed)
• (lack of) Distribution (and therefore availability)
• Complexity of infrastructure
• Message delivery guarantees (reliability)
• Lack of technology openness
• Deal with temporarily offline consumers
• Retain history

Publiek
Introducing Apache Kafka
• ..- 2010 – creation at Linkedin
• Message Bus | Event Broker
• High volume, low latency, highly reliable, cross technology
• Scalable, distributed, strict message ordering, ….
• 2011/2012 – open source under the Apache Incubator/ Top Project
• Kafka is used by many large corporations:
• Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift
Science, Zalando, The New York Times, Airbnb, Coursera, ING Bank,…
• … and embraced by many software vendors & cloud providers
• Commercial backing by and Enterprise support from Confluent

Publiek
Producers
Consumers
tcp
tcp
Client polls Event
Broker for messages

Publiek
Producers
Consumers
Topic

Publiek
Kafka terminology
• Topic
• Message
• == ByteArray
• Broker
• Producer
• Consumer
• Offset
Producer Consumer
Topic
Broker
Key
Value
Time
Message
offset (for a specific consumer)

Publiek
Producers
Consumers
Topic
Broker
Key
Value
Time

Publiek
Consuming
• Messages are available to consumers only when they have been committed
by the producer
• Kafka does not push
• Unlike JMS
• Read does not destroy
• Unlike JMS Topic
• (some | much | all) History available
• Offline consumers can catch up
• Consumers can re-consume from the past (just move offset)
• Delivery Guarantees
• Ordering maintained
• At-least-once (per consumer) by default; at-most-once and exactly-once
can be implemented

Publiek
The company bulletin board
!
!!
“@#”
!
!
!!

Publiek
The company bulletin board
!
!
!!
“@#$”
€
$
¥ ₤ ₨

Publiek
Producers
Consumers
Topic
Broker
Key
Value
Time
Message
Log

Publiek
Producers
Consumers
Topic
Broker
tcp
tcp
Cluster

Publiek
Producers
Consumers
Topic
Broker
tcp
tcp
Consumer Group
Partition
Cluster
Partition Log
Replica

Publiek
Consumers
Topic
Broker
Consumer Group
tition
Cluster
Consumer Group
• each consumer group consumes all
messages (once)
• each consumer in a group consumes
from one or more partitions (no partition
is consumed by more than one
consumer in a group)
• when more consumers in a group than
partitions in the topic, some consumers
get no messages
• when consumers disappear from a
group, their partition(s) are reassigned
to other consumers in the group

Publiek
What’s so special?
• Durable transactions
• Scalable
• High volume
• High speed
• Parallel processing
• Available
• Distributed
• Open
• Quick start
• Free (no license costs)
• “Self Fulfilling Prophecy” leading to “de facto standard”
(positive feedback loop feeding from buzz around Kafka)
• Eco system, tools/libraries/resources, cloud services

Publiek
Some Kafka Use Cases
• Connected Cars, Manufacturing, Mobility, Gaming, Betting
• Tesla: Processing and analyzing the data from their vehicles, smart grids,
and factories and integrating with the rest of the IT backend services in real-
time is a crucial piece of Tesla’s success
https://kai-waehner.medium.com/when-not-to-use-apache-kafka-a35345226a9f

Publiek
• Regulatory compliance and zero data loss are crucial.
• transactionally safe data replication

Publiek
• Royal Caribbean: Each cruise ship has a Kafka cluster running locally for
use cases such as payment processing, loyalty information, customer
recommendations, etc. Sync with shore when entering a port.

Publiek
Quick Demo – Apache Kafka through Kafka Console
• Show interaction at the most basic level
• Broker
• Topic
• partition
• Produce Message
• Consume Message
• Consumer Group
Docker
Compose
Browser
Laptop
28042
9092
localhost
kafka

Publiek
Lab: First steps with Apache Kafka
• Using Kafka Console
• Create a topic
• Publish messages to topic
• Using Kafka Console
• Consume messages from topic
• Work with Apache Kafka HQ GUI to
• inspect the Kafka Cluster
• produce additional messages
• consume message from a special Consumer Group
Docker
Compose
Browser
Laptop
28042
9092
localhost
kafka
Lab Resources: https://github.com/lucasjellema/fontys-2022-microservices-kafka-dapr

Publiek
Lab Environment
Docker
Compose
Browser
Laptop
28042
9094
9093
9092
2181
28044
localhost
kafka
Lab Resources: https://github.com/lucasjellema/fontys-2022-microservices-kafka-dapr

Publiek
Partly Clouded Lab Environment
Browser
Laptop
2181
localhost
CloudKarafka
Console
Lab Resources: -
www.cloudkarafka.com/plans.html

Publiek
Agenda -
• Part One (last week – microservices, Node, Dapr.io)
• Part Two (today)
• Introduction of Apache Kafka (“Twitter for systems”)
• Asynchronous interactions through Apache Kafka – concepts, terminology
• Handson with Apache Kafka and Node
• Real world use cases and scenarios with Apache Kafka
• Handson Multi-microservice set up with Apache Kafka, Node and Dapr.io

Publiek
Programming against Kafka: Client Libraries
• Client libraries
• C/C++
• Python
• Go
• Java
• .NET
• JavaScript/Node
• Ruby
• Scala
• and more..
• Also: REST Proxy (part of open source Confluent Platform)
• Produce, Consumer and manage through REST API calls

Publiek
Programming against Kafka:
Typical Steps in a Client Producer Application
• Create a connection
• connected to one or more brokers in a cluster or to the Zookeeper node
• you need the broker-endpoints or the Zookeeper endpoint
• optionally use credentials or certificate for authentication
• Create a Producer on top of the connection
• When the Producer-Connection is available…
• [Optionally] Start a Transaction
• Produce message[s] to topic(s)
[or specific topic partition(s)]
• [Optionally] Commit the Transaction
• [Repeat Message Production]
• Disconnect the producer/close the connection
Client
Application
9094
9093
9092
2181

Publiek
Programming against Kafka:
Typical Steps in a Client Consumer Application
• Create a connection
• connected to one or more brokers in a cluster or to Zookeeper node
• you need the broker-endpoints or the Zookeeper endpoint
• optionally use credentials or certificate for authentication
• Create a Consumer or a ConsumerStream on top of the connection
• Subscribing to one or more topics
• [Optionally] Associating with a Consumer Group
• (Which leads to a link to one or topic
more partitions on the Kafka brokers)
• [Optionally] Overriding the auto-commit
• When the Consumer or Stream is available…
• Consume message[s] (from the specified subscriptions)
• [Optionally, depending on auto-commit poperty]
Commit the Transaction
• [Repeat Message Consumption]
• Disconnect the consumer/close the connection
Client
Application
9094
9093
9092
2181

Publiek
Node and Kafka
• We will use NPM module node-rdkafka to facilitate the interaction with
Apache Kafka from our Node application
• This module leverages a C/C++ library for Kafka interactions

Publiek
Node and Kafka – a simple producer

Publiek
Node and Kafka – a simple producer
Client
Application
9094
9093
9092
test-topic
1
2

Publiek
Node and Kafka – a simple consumer
9094
9093
9092
test-topic

Publiek
Node and Kafka – a simple consumer
Client
Application
9094
9093
9092
test-topic
1
2

Publiek
Lab – Programmatic interaction with Apache Kafka
• Prequisite: local Node runtime
• Step Two:
• Node application to Produce Messages to Kafka Topic
• Node application to Consume Messages from Kafka Topic
• Bonus Step Three:
• Node Web Application to Produce Messages from HTTP Request
• Node Web Application to return Messages on Topic in HTTP Response

Publiek
Produce
Consume
kjynvuby-default

Publiek
Utility company with data. Lots of it.
Interesting mix of operational, real time systems in physical environments, large scale enterprise systems for
1000s of internal professionals and millions of external consumers – with a mission to improve the world

Publiek
Utility Company looking for Performance, Scalability and
Availability in their core data sets
CRM
Meter Readings
batch
Invoicing
Billing
Marketing
Campaigns
Load on Core Systems &
effect on Performance
Availability Core Systems

Publiek
Utility Company looking for Performance, Scalability and
Availability in their core data sets
CRM
Meter Readings
Invoicing
Billing
Marketing
Campaigns
CRM
Cache DB
Meter
Readings
Cache DB

Event Sourcing

Publiek
Event Sourcing
Events Event Store
Current State
accountId:
123
amount: 10
Owner: Jane Doe

Publiek
Event Sourcing
Events Event Store
Current State
Other State Aggregate

Publiek
Event Sourcing
• Event Store is immutable – append-only log of domain state transitions
• It is the truth about data – everything else is derived
• Replay events
• to (re)construct a representation of the current state (aggregate)
• up to a specific time to recreate moments in time
• in Test environment to investigate an issue
• on a remote location to create mirror & share state across boundaries
• produce a fine grained audit trail
• Challenges
• Time required to reconstruct state
• Grain of aggregates / definition of domain events

Publiek
Retail chain – 1200 stores and one central environment
Central Environment
(on Azure)
Store A
Store B
Store C
Store D

Publiek
Retail chain – 1200 stores and one central environment
Central Environment
(on Azure)
Store A
Store B
Store C
Store D
• fast
• capable of high volume
• reliable / transactional
• cross technology
• not invasive
• future proof (and proven)
• bi-directional

Publiek
Outbound – from central to 1200 stores
Central Environment
(on Azure)
Store A
Store B
Store C
Store D

Publiek
Inbound – from 1200 stores to Central
Central Environment
(on Azure)
Store A
Store B
Store C
Store D

Publiek
Web Portal – Faster, Simpler and more Available
Lab
Systems Web Portal

Publiek
Web Portal – The Original Situation (2014)
Relational Database
Service Bus
Lab
Systems Web Portal
SOAP/XML
SQL & Stored
Procedures

Publiek
Web Portal – Faster, Simpler and more Available (2021)
Relational Database
Service Bus
Lab
Systems
Web Portal
SOAP/XML
Kafka
Connect
Azure Cloud
NoSQL
Database
Kafka
REST/JSON
Changed
Data Capture

Publiek
Apache Kafka is more than the core distributed platform
And there is more to distributed event platforms than Kafka
• Apache Kafka is surrounded by an ecosystem
• Primary supporting company: Confluent
• Extensions, Tools, Enterprise Edition
• Supporting resources (books, articles, tutorials, conferences)
• Partner companies
• Managed cloud services
• Kafka-compatible Public Cloud offerings
• Kafka-like offerings
• Competitors, imitators, …

Publiek
Ecosystem – Kafka and Friends
• Kafka Schema Registry – manage message schema definitions
• Kafka Connect – read data and change events
from many sources and/or write to many targets
• Also see Debezium
• Kafka Streams
• Kafka KSQL
• Confluent Enterprise
• Enterprise Grade Security
• Replicator
(synch across regions)
• Ops (Operations) support
• Confluent Managed Kafka Cloud Offering

Publiek
Confluent offerings

Publiek
Select Running Count
from <stream of tweet events>
select tag
, count(*) tweet_count
from tweets
Where tag = 'COVID2019' or tag = 'KAFKA'
group
by tag
<--- streaming data

Publiek
Topic
Continuous Queries on Streaming Data
• Count Events, Aggregate Payloads, Filter, Combine & Enrich…
• … and produce: new events
Topic
Topic
Processor
Topic
Processor
Table

Publiek
Alternatives for Apache Kafka’s
Distributed Streaming Event Platform
• Apache Pulsar
• Solace PubSub+ Event Portal
• Rabbit MQ
• Apache Active MQ
• Streaming Data libraries
• Apache Spark Streaming, Conduit, Faust, Apache Flink, Apache Storm,
Samza, Apache Beam, Apache Flume

Publiek
Managed Kafka (-like) Cloud Offering
• Confluent
• AWS
• SQS
• Managed Streaming for Apache Kafka (Amazon MSK)
• EventBridge, Kinesis
• Azure
• Event Hub
• EventGrid, Stream Analytics
• Oracle Cloud Infrastructure
• Streams
• Google Cloud Platform
• PubSub
• DataFlow
• … (Heroku Managed Kafka, Alibaba Cloud LogHub, IBM Event Streams,…)
?

Publiek
Slide from last week:
Pub/Sub with Dapr from Node applications
sidecar (the personal assistant)
Node SDK
for Dapr
Pub/Sub
components
publish message
components.yaml
PubSub
Brokers
pub/sub
interface
Node SDK
for Dapr
components.yaml
subscribe on topic
with handler function
message sent to handler

Publiek
Node SDK
for Dapr
Pub/Sub
components
publish message
components.yaml
PubSub
Brokers
Node SDK
for Dapr
components.yaml
subscribe on topic
pub/sub
interface
publish
message

Publiek
Node SDK
for Dapr
Pub/Sub
components
publish message
ponents.yaml
PubSub
Brokers
Node SDK
for Dapr
components.yaml
pub/sub
interface
subscribe
on topic
message sent
to handler

Publiek
Replace Redis by Apache Kafka
Node SDK
for Dapr
Pub/Sub
components
publish message
to topic
components.yaml
PubSub
Brokers
pub/sub
interface
Node SDK
for Dapr
components.yaml
subscribe on topic
9094
9093
9092
test-topic

Publiek
Labs on Kafka and Node and Dapr
• Programmatically work with Kafka from Node – with and without Dapr
• publishing and consuming messages
• Implement the CQRS pattern between two microservices
• one is master of data, the other has a stand alone replica – to be synchronized
• Implement decoupled conversation between multiple
microservices – through Kafka
Billing Engine
CRM
Handles assignment on
workflow queue to produce a
bill for a customer
Publish Event to Kafka Topic
with question details and
conversation identifier
Consume Event, handle the question and publish
a response with customer details on a second
queue – including the conversation identifier
questions
answers
workflow-
queue
Billing Run
Coordinator
Create bill
processing
instructions for
customers
1
2
3
4
5
bill

Publiek
Lab – Microservice interaction
• Asynchronous conversation between microservices
Billing Engine
CRM
Handles assignment on
workflow queue to produce
a bill for a customer
Publish Event to Kafka
Topic with question details
and conversation identifier
Consume Event, handle the question
and publish a response with customer
details on a second queue – including
the conversation identifier
questions
answers
workflow-queue
Billing Run
Coordinator
Create bill
processing
instructions for
customers
1
2
3
4
5
bill

Publiek
Summary
• => == =>
• Apache Kafka is emerging as platform of choice for message exchange in a world of
• Microservices
• Event Sourcing, CQRS and Data Source Synchronization
• Clouds
• Fast Data (IoT) and Streaming Analysis
• Real time data integration & distribution
• Getting started with Apache Kafka is not very hard at all
• The platform is open source – and has broad client support (Java, Node, …)
• Many resources are available – tutorials, blog article, demonstrations, presentation
slides and recordings of conference sessions, samples on GitHub
• Using Docker Compose it is quite easy to quickly run a Kafka Cluster
(and with CloudKarafka even easier)
• Note: managing a production grade cluster is not so easy

you are invited to come do an internship and explore job opportunities with us!
lucas.jellema@amis.nl | technology.amis.nl | @lucasjellema | lucas-jellema

Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschool, Spring 2022)

More Related Content

What's hot (20)

Similar to Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschool, Spring 2022) (20)

More from Lucas Jellema (20)

Recently uploaded (20)

Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschool, Spring 2022)

Editor's Notes