Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehner, Confluent & Dominik Obermaier, HiveMQ) Frankfurt 2019 Confluent Streaming Event
0 likes111 views
This document discusses best practices for streaming IoT data with MQTT and Apache Kafka. It begins with an overview of a use case involving a global automotive company building a connected car infrastructure. An architecture is presented showing how sensor data from cars can be ingested via MQTT into Apache Kafka and then processed using tools like Kafka Streams, TensorFlow, and ElasticSearch for analytics and alerts. A live demo is described that implements this full pipeline. The document concludes with a discussion of best practices around choosing the right tools, separation of concerns, data types, and next steps.
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehner, Confluent & Dominik Obermaier, HiveMQ) Frankfurt 2019 Confluent Streaming Event
- 1. 1 Best Practices for Streaming IoT Data with MQTT and Apache Kafka Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de Real Time Data Processing and Analytics with 100000 Connected Cars Dominik Obermaier CTO HiveMQ [email protected] www.linkedin.com/in/dobermai @dobermai www.hivemq.com
- 2. 2 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 3. 3 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 4. 4 Global Automotive Company Builds Connected Car Infrastructure Use Cases: • Connected Car Infrastructure (Cars, Traffic Lights, Cloud Services, etc.) • Real Time Analytics (Predictive Maintenance, etc.) • Continuous Services / Sales • Partner Integration (Car workshop, gas station, food market, etc.) • …
- 5. 55 Other Components Kafka Ecosystem (3) Read Data TensorFlow I/O (5) Deploy Model (2) Preprocess Data (8a) Alert Car Real Time Kafka App TensorFlow Serving gRPC Car Sensor HiveMQ MQTT Broker MQTT Connector Kafka Connect or Confluent Proxy or HiveMQ Plugin Kafka Cluster Kafka Connect Grafana Elastic Search KSQL Tensor Flow (04) Train Model Real Time Kafka Streams Application (Java / Scala) Tensor Flow Real Time Edge Computing (C / librdkafka) Tensor Flow Lite (1) Ingest Data (8b) Alert Driver (e.g. mobile App) (6a) Consume Car Data (6b) All Data (7) Potential Defect
- 6. 6 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 7. 7 Cloud Native Infrastructure Benefits • Scalable • Flexible • Agile • Elastic • Automated • Etc.
- 8. 8 • IoT-specific features for bad network / connectivity • Widely used (mostly IoT, but also web and mobile apps via MQTT over WebSocket) • Built on top of TCP/IP for constrained devices and unreliable networks • Many (open source) broker implementations • Many (open source) client libraries MQTT - Publish / subscribe messaging protocol
- 9. 9 MQTT Trade-Offs Pros • Lightweight • All programming languages supported • Built for poor connectivity / high latency scenarios (e.g. mobile networks!) • High scalability and availability * • ISO Standard • Most popular IoT protocol Cons • Only pub/sub, not stream processing • Asynchronous processing (clients can be offline for long time) • No reprocessing of events * Depending on Broker
- 10. 10 A Streaming Platform is the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservice s events SaaS data Customer experience s Streams of real time events Stream processing apps
- 11. 11 Kafka Trade-Offs (from IoT perspective) Pros • Stream processing, not just pub/sub • High throughput • Large scale • High availability • Long term storage and buffering • Reprocessing of events • Good integration to rest of the enterprise Cons • Not built for tens of thousands connections • Requires stable network and good infrastructure • No IoT-specific features like keep alive, last will or testament
- 12. 12 (De facto) Standards for Processing IoT Data A Match Made In Heaven + =
- 13. 13 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 14. 14 Advanced Demo
- 15. 15 MVP
- 16. 16 Live Demo End-to-End Integration and Data Processing for 100000 Connected Cars
- 17. 17 https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference or http://bit.ly/kafka-mqtt-ml-demo => Try it out in 30 minutes! Demo 100.000 Connected Cars (Kafka + MQTT + TensorFlow)
- 18. 18 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 19. 19 Typical Journey Value Maturity (Investment & time) 2 Enterprise Streaming Pilot / Early Production Pub + Sub Store Process 5 Central Nervous System 1 Developer Interest Pre-Streaming 4 Global Streaming 3 SLA Ready, Integrated Streaming Projects Platform
- 20. 20 Start Small, but prepare for Scalability from Beginning 1. Use cloud native and scalable components • Confluent Platform is cloud native and built for scale • HiveMQ is cloud native and built for scale 2. Don’t deep dive too much in the beginning – but understand options • HiveMQ Kafka Extension? • Confluent MQTT connectors? • Customer Integration? 3. Plan for Enterprise-readiness • Security • Monitoring • Operations tooling • Bi-directional communication
- 21. 21 Choose the right tool stack and infrastructure Understand Trade-Offs and choose the right options for deployments • Edge • On Premise • Cloud Use the best tools for the job • Confluent Platform for Event Streaming • HiveMQ for MQTT messaging and connectivity
- 22. 22 Separation of concerns 1. Devices 2. Gateway 3. Integration 4. Data Streaming 5. Consumer Apps Decouple tasks • Source integration • Data processing • Business logic • Sink integration • Analytics • …
- 23. 23 Different data for different use cases • Database, Data Lake • Search • Real time, Near Real Time, Batch • Streaming, Request-Response • CQRS, Event Sourcing • Machine Learning There is no single MASTER DATA EVENT…
- 24. 24 Agenda • Use Case • Architecture • Live Demo • Best Practices • Next steps
- 26. 26 The HiveMQ Platform – Open Source and Enterprise-grade
- 27. 27 Confluent Platform Operations and Security Development & Stream Processing Support,Services,Training& Partners Apache Kafka Security plugins | Role-Based Access Control Control Center | Replicator | Auto Data Balancer | Operator Connectors Clients | REST Proxy MQTT Proxy | Schema Registry KSQL Connect Continuous Commit Log Streams Complete Event Streaming Platform Mission-critical Reliability Freedom of Choice Datacenter Public Cloud Confluent Cloud Self-Managed Software Fully-Managed Service
- 28. 28 Confluent Cloud Cloud-Native Confluent Platform Fully-Managed Service Available on the leading public clouds with mission-critical SLAs and consumption-based pricing. Serverless Kafka characteristics: Pay-as-you-go, elastic auto-scaling, abstracting infrastructure (topics not brokers) Spend your time on your applications!
- 29. 29 Next steps… Try out the demo in 30 minutes: https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training- inference http://bit.ly/kafka-mqtt-ml-demo Check out the documentation and blog posts • HiveMQ and Apache Kafka - Streaming IoT Data and MQTT Messages: https://www.hivemq.com/blog/streaming-iot-data-and-mqtt-messages-to-apache-kafka/ • Internet of Things (IoT) and Event Streaming at Scale with Apache Kafka and MQTT: https://www.confluent.io/blog/iot-with-kafka-connect-mqtt-and-rest-proxy Contact us for questions or any other feedback: • Website, Email, Slack, Phone, … • Dominik: [email protected] , Kai: [email protected]
- 30. 30 Questions? Feedback? Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de Please contact us! Dominik Obermaier CTO HiveMQ [email protected] www.linkedin.com/in/dobermai www.hivemq.com www.twitter.com/dobermai