HiveMQ + Kafka: The ideal solution for IoT MQTT data integration

Copyright © by HiveMQ. All Rights Reserved.Copyright © HiveMQ. All Rights Reserved.
WELCOME TO
We will start the session shortly

Copyright © by HiveMQ. All Rights Reserved.
WELCOME
Dominik Obermaier Magi Erber
• HiveMQ CTO
• Strong background in
distributed and large scale
systems architecture
• OASIS MQTT TC Member
• Author of „The Technical
Foundations of IoT“
• Conference Speaker and
Author
• Program committee
member for German and
international IoT
conferences
• Product Manager
@HiveMQ
• Conference Speaker
• Author
• Expert for cloud native
technologies and Apache
Kafka
@dobermai
linkedin.com/in/dobermai/
@ErberMagi
linkedin.com/in/margaretha-erber/

The Internet of Things is HUGE
PEOPLE ON THE INTERNET

The Internet of Things is HUGE
DEVICES ON THE INTERNET

Technical IoT Challenges
Customers, ARR, ...
Conﬁdential and Proprietary. Copyright © by HiveMQ. All Rights Reserved.

Challenge 1 - Scalability
• Enterprise IT infrastructure is not
suitable for IoT
• Massive scalability required for
millions of devices

Challenge 2 - Instant Data Delivery required
• End customers are used to instant
user experiences like instant
messaging with WhatsApp
• Critical systems need reliable and
instant data transfer like manufacturing
systems
• Bidirectional communication required

Challenge 3 - Unreliable Networks
• Customer experience for IoT apps and devices
must be excellent even when internet connectivity
is ﬂaky
Especially for moving “devices” like cars
• Devices and apps must be easy to program and
maintain, complexity should be in the cloud not
on the device
Cloud is easier to update than physical devices

Web Technology used today is
built for the
Internet of Humans,
NOT for the
Internet of Things

We need open standards
designed for the
Internet of Things

What Is MQTT?
• Easy on the device side, pushes all implementation
complexity to the server
• Publish/Subscribe based architecture
• Created for extreme scale and instant data exchange
• (I)IoT Messaging Protocol
• Built for machines and constrained devices(binary,
data agnostic)
• Designed for reliable communication over unreliable
channels

Connected Car
IIoT /
Industry 4.0
Logistics Telecommunication
IoT Messaging
Middleware
MQTT Use Cases

MQTT Use Cases
Push Communication
Reliable Communication over
unreliable networks
Constrained Devices
Low Bandwidth and High
Latency
Industrial Message Bus

Apache Kafka
• Distributed streaming platform
• Used by over one third of Fortune 500 companies
• Most popular Apache project on GitHub
• Central messaging and distributed stream processing
application

Apache Kafka Strengths
• Optimized to stream data between systems and
applications in a scalable manner
• Scale-out with multiple partitions for a topic and
multiple nodes
• Perfect for inter-system communication
• inside trusted network
• with stable IP addresses and
connections and
• limited number of producers and
consumers

Apache Kafka

Apache Kafka
Producers Consumers
Partition 1
Partition 2
Partition 3
My Topic
Producer A
Producer B
Consumer
A
Consumer
B
Consumer
C
0
0
0
1
1
1
2

Apache Kafka for IoT -
How does it ﬁt in?

IoT Reality Challenges

Challenge 1 - Millions of Connections
● Clients are connected over the Internet
● Load Balancers are used as ﬁrst line of defense
● IP addresses of infrastructure (e.g. Kafka nodes) not
exposed to the public Internet
● Load Balancers effectively act as proxy
● Kafka Clients need to address Kafka brokers directly,
which is not possible with L4 load balancers
IoT REALITY

● IoT devices typically are segmented to use individual topics
● Individual topics very often contain data like unique device
identiﬁer
● Multiple millions of topics can be used in a single IoT scenario
● Ideal for security as it’s possible to restrict devices to only
produce and consume for speciﬁc topics
● Topics are usually dynamic
● Kafka is hard to scale to multiple thousands or
even millions of topics
IoT REALITY
Challenge 2 - Scalability and Topics

car-0000001
my-iot-devices/ger/group-1/car-0000001/speed
my-iot-devices/ger/group-1/car-0000001/location
my-iot-devices/ger/group-1/car-0000001/motor-heat
car-1045107
my-iot-devices/eu/group-3/car-1045107/speed
my-iot-devices/eu/group-3/car-1045107/location
my-iot-devices/eu/group-3/car-1045107/motor-heat
my-iot-devices/usa/group-1/car-5239284/speed
my-iot-devices/usa/group-1/car-5239284/location
my-iot-devices/usa/group-1/car-5239284/motor-heat
...
...
car-5239284
Challenge 2 - Scalability and Topics

● IoT devices are typically very constrained (computing power and
memory)
● Device programmer need very simple APIs AND full ﬂexibility when it
comes to library behavior
● Single IoT devices typically don’t require lot of throughput
● Important to limit and understand the number of TCP connections,
especially over the Internet. Very often only one TCP connection to
the backend desired
● Kafka Clients are reasonable complex by design (e.g. use multiple
TCP connections)
● Libraries optimized for throughput
● APIs for Kafka libraries are simple to use but the behavior sometimes
isn’t conﬁgurable easily (e.g. async send() method can block)
IoT REALITY
Challenge 3 - Constraint Devices

● Features like on/off notiﬁcations are often required
● Unreliable networks require lightweight keep-alive mechanisms for
producers and consumers (half-open connections)
● Device communication over the Internet requires minimal
communication overhead
● No on/off notiﬁcation mechanism
● No Keep-Alive mechanism individual TCP connections for producers
● Kafka Protocol for producers rather heavyweight over the Internet
(lots of communication)
IoT REALITY
Challenge 4 - Unreliable Network

Kafka is well suited for data
ingestion of cloud native server
applications,
but not well suited for IoT device
data connectivity

Challenges for Apache Kafka in IoT
Kafka brokers need to be addressed directly by the clients
Kafka does not support large amounts of topics
Kafka clients are reasonably complex and resource intensive compared to
client libraries for IoT protocols
Kafka clients require a stable TCP connection for best results
It’s unusual (and very often not even possible at all) to have tens of
thousands or even millions of clients connected to a single Kafka cluster
Kafka is missing some key IoT features

How to use the best of
both worlds?

A Love Story Made in Heaven
❤

HiveMQ - Enterprise MQTT Broker
● Connectivity and Messaging Platform
● Based on standard IoT protocol
(MQTT)
● Scales to more than 10 million
always-on devices
● Allow multi-cloud and Enterprise
software integration

HiveMQ - Enterprise MQTT Broker

HiveMQ and Kafka
Fast, reliable communication
Ideal for communication over
unreliable networks
Connects millions of IoT
Devices
Optimized to stream data
between systems and
applications
Communication inside trusted
networks
Connects limited number of
communication partners

HiveMQ Enterprise Extension for Kafka
• Native implementation of Kafka protocol
• End to-end persistent messaging guarantees
• Bi-directional communication
• High Scalability and resilience
• Support of Local Schema Registry (Avro, JSON)
• Support of Conﬂuent Schema Registry (Avro)
• Stream to multiple Kafka instances

Support of Different Kafka Distributions
• Multiple Kafka deployments simultaneously
• AWS MSK
• Apache Kafka (Open Source) • Conﬂuent Platform
• Conﬂuent Cloud

High Scalability and Resilience
Scales elastically with the MQTT broker
No message loss: Queues messages, if Kafka cluster is temporarily
not available
Extreme throughput. Can write hundreds of thousands of MQTT
messages per second to Kafka

Bidirectional Communication

• MQTT to Kafka Topic Mapping
• One to One Topic Mapping
• Aggregating MQTT Topics to Firehose
Kafka Topic (using topic ﬁlters)
• Kafka to MQTT Topic Mapping
• One to One Topic Mapping
• One (Many) to Many Topic Mapping
• Message transformation (using parts of
Kafka message)

Record
Kafka Key
Kafka Value
PUBLISH
MQTT Topic
MQTT Payload
Topic

Message Transformation
Record
Kafka Key
{
“client-id”: “client-1”,
“message”: “whatever needs to be sent”
}
PUBLISH
my-iot-devices/group-1/${client-id}
/{message}
Topic
my-iot-devices/group-1/client-1
“whatever needs to be sent”

Message Multicasting
Record
Kafka Key
{
“client-id”: [“client-1”, “client-2”, “client-3”],
“message”: “whatever needs to be sent”
}
PUBLISH
my-iot-devices/group-1
/${client-id}
“whatever needs to
be sent”
Topic
PUBLISH
/${client-id}
be sent”
PUBLISH
/${client-id}
be sent”
/client-1
/client-2
/client-3

Support of Schema Registries
Apache Kafka
• Kafka does not care about message formats
• No veriﬁcation of message correctness
• Systems are evolving and message formats will
change
● Support of Local Schema Registry
○ JSON and Avro messages
○ Message validation
Schema Registries
• Guarantee the correct functionality of
messaging
• Message validation
• Schema evolution

Local Schema Registry

Support of Schema Registries
Apache Kafka
• Kafka does not care about message formats
• No veriﬁcation of message correctness
• Systems are evolving and message formats will
change
● Support of Local Schema Registry
○ JSON and Avro messages
● Support of Conﬂuent Schema Registry
○ Avro messages
○ Schema evolution
Schema Registries
• Guarantee the correct functionality of
messaging
• Message validation
• Schema evolution

Conﬂuent Schema Registry

Observability of Communication

• Connect millions of IoT devices, over unreliable networks, to seamlessly send data to one or
multiple Kafka clusters
• Route IoT device data to multiple Kafka clusters allowing for a single IoT platform to support
different types of devices and forward device data to different back-end applications
• Poll information from one or multiple Kafka clusters and distribute this information to millions
of IoT devices
• Support of all MQTT 3.1.1 and MQTT 5 features
• Map millions of MQTT topics to a limited number of Kafka topics using MQTT wildcards
• Enable end to-end persistent messaging guarantees from device to Kafka so no messages are
lost
• Monitor MQTT messages written to Kafka using the HiveMQ Control Center

Demo

HiveMQ Enterprise Extension for Kafka - Demo
Contact us at kafka@hivemq.com for getting a live demonstration of the HiveMQ Enterprise Extension for Kafka
Mobile
Application
sensor-data
Stream processing
using KSQL
Other
Backend
Systems
MQTT to Kafka
Kafka to MQTT
100simulatedcars
Conﬂuent Schema Registry
Get Schema
alert

THANK YOU
For attending the webinar
We will upload the webinar on our YouTube Channel
Subscribe to our YouTube Channel: page.video/hivemq
Stay updated on upcoming webinars
Subscribe to our Newsletter: newsletter.social/hivemq
All unanswered questions will be answered on the
HiveMQ Community Forum
To the HiveMQ Community Forum:
community.hivemq.com

HiveMQ + Kafka: The ideal solution for IoT MQTT data integration

More Related Content

What's hot (20)

Similar to HiveMQ + Kafka: The ideal solution for IoT MQTT data integration (20)

Recently uploaded (20)

HiveMQ + Kafka: The ideal solution for IoT MQTT data integration