IoT Sensor Analytics with Kafka, ksqlDB and TensorFlow
2 likes1,477 views
The document discusses IoT data integration and processing using technologies like Apache Kafka, KSQL, TensorFlow, and MQTT, highlighting their roles in real-time analytics. It explores the architecture of IoT systems, MQTT's pros and cons, and Kafka's advantages for high-throughput stream processing. The presentation also includes a live demo of sensor analytics and options for integrating MQTT with Kafka for improved data handling and analysis.
IoT Sensor Analytics with Kafka, ksqlDB and TensorFlow
- 1. IoT Sensor Analytics with Apache Kafka, KSQL, TensorFlow and MQTT Kafka-Native End-to-End IoT Data Integration and Processing Kai Waehner Technology Evangelist Confluent [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. POLL: What are your current Use Cases for Apache Kafka?
- 3. Agenda 3 01 IoT Use Cases 06 IoT Data Processing 02 MQTT Standard 07 Live Demo: End-to-End Sensor Analytics 03 Apache Kafka Ecosystem 04 TensorFlow for IoT Scenarios 05 End-to-End IoT Integration Architecture(s)
- 5. The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Front Camera Audi Active lane assistant Speed limit indicator Adaptive light Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist
- 6. The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Ultrasonic SensorsCrash Sensors Front Camera Infrared Camera Front and Rear Radar Sensors Traffic Alerts Hazard Alerts Personalization Anomaly Detection MQTT MQTT MQTT MQTT MQTTMQTT
- 7. 7
- 8. 8
- 9. 9
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 15. POLL: Which IoT scenarios do you see in your company?
- 16. MQTT Standard
- 17. MQTT - Publish / subscribe messaging protocol • Built on top of TCP/IP for constrained devices and unreliable networks • Many (open source) broker implementations • Many client libraries • IoT-specific features for bad network / connectivity • Widely used (mostly IoT, but also web and mobile apps via MQTT over WebSockets) 17
- 19. MQTT Trade-Offs PROs ● Lightweight ● Simple API ● Built for poor connectivity / high latency scenario ● Many client connections (tens of thousands per MQTT server) 19 CONs ● “Just” queuing, not stream processing ● Can’t handle usage surges (no buffering) ● Most MQTT brokers don’t support high scalability ● Very asynchronous processing (often offline for long time) ● No good integration to the rest of the enterprise ● No reprocessing of events
- 21. Apache Kafka - The Rise of an Event Streaming Platform 21
- 22. Log & Pub/Sub 22
- 23. Apache Kafka == Distributed Commit Log with Replication 23
- 24. Apache Kafka at Scale at Tech Giants 24 > 7 trillion messages / day > 6 Petabytes / day ...you name it!
- 25. Kafka Trade-Offs (IoT Perspective) PROs ● Stream processing, not just queuing ● High throughput ● Large scale ● High availability ● Long term storage and buffering ● Reprocessing of events ● Good integration to the rest of the enterprise 26 CONs ● Not built for tens of thousands connections ● Requires stable network and good infrastructure ● No IoT-specific features like keep alive, last will or testament
- 26. (De facto) Standards for Processing IoT Data 27 ...a match made in Heaven
- 27. TensorFlow for IoT Scenarios
- 28. TensorFlow TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains. 29
- 29. The first analytic models 30 How to deploy the models in production? …real-time processing? …at scale? …24/7 zero downtime?
- 30. Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 31
- 31. Apache Kafka’s Open Source Ecosystem as Infrastructure for ML 32
- 32. Apache Kafka’s Open Ecosystem as Infrastructure for ML 33
- 35. Model Training without additional Data Store https://github.com/tensorflow/io/tree/master/tensorflow_io/kafka 36 ● Native integration between Kafka and TensorFlow ● KafkaDataSet and KafkaOutputSequence for TensorFlow ● Written in C++ (linked with librdkafka) ● Part of the graph in TensorFlow ● Direct training and inference from streaming data ● No data storage like S3 or HDFS needed
- 36. Replayability — a log never forgets! 37
- 37. Analytic Model (Autoencoder for Anomaly Detection) 38
- 38. Model Deployment #1: RPC Communication to do Model Inference 39
- 39. Model deployment #2: Model inference natively in the App 40
- 42. Architecture (High Level) – Machine Learning Perspective 43
- 43. Kafka-Native Integration Options between MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy 44
- 44. Kafka-Native Integration Options between MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy 45
- 45. Integration with Kafka Connect (Source and Sink) 46
- 46. Kafka Connect – Don‘t reinvent the wheel! 47
- 47. Kafka-Native Integration Options between MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy 48
- 48. MQTT Proxy 49
- 49. Kafka-Native Integration Options between MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy 50
- 50. REST Proxy 51
- 51. Confluent REST Proxy - Produce and Consume Messages 52
- 54. Processing Options for MQTT Data with Kafka 55
- 55. The 3 stream processing modalities with Confluent 56
- 56. The 3 stream processing modalities differ in flexibility and ease of use 57
- 57. Using external processing systems leads to complicated architectures 58
- 58. We can put it back together in a simpler way 59
- 59. Connect integration and pull queries enable end- to-end streaming in just a few SQL statements 60
- 60. Live Demo: End-to-End Sensor Analytics
- 61. KSQL and Deep Learning (Auto Encoder) for Anomaly Detection 63
- 62. Model Training with Python, KSQL, TensorFlow, Keras and Jupyter https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras 64
- 63. Model Deployment with Apache Kafka, KSQL and TensorFlow 65
- 64. Live Demo: End-to-End Sensor Analytics…
- 65. Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot 67
- 66. Demo: Predictive Analytics for 100000 Connected Cars Kafka + KSQL + MQTT + TensorFlow + Kubernetes 68 https://www.kai-waehner.de/blog/2019/11/08/live-demo- iot-100-000-connected-cars-kubernetes-kafka-mqtt- tensorflow/
- 67. POLL: What is the best choice for your IoT integration between MQTT and Kafka?
- 68. Stay in touch! Try Confluent cnfl.io/download Confluent Blog cnfl.io/blog Community cnfl.io/meetups