![MQTT Source connector
• Message Queueing Telemetry Transport messaging protocol
• Optimised for connected devices and M2M protocol
• Low bandwidth, high-latency, unreliable networks
• Plain or SSL connections
• Kafka Connect Query Language
INSERT INTO kafka_topic
SELECT * FROM /mqtt_source_topic
[WITHCONVERTER=`myclass`]
MQTT
Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions](https://image.slidesharecdn.com/kafkameetuplondonfeb2018-180227055533/85/MQTT-Kafka-InfluxDB-SQL-IoT-Harmony-tutorial-by-Stefan-Bocutiu-11-320.jpg)
MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu
- 1. Stefan Bocutiu @stepib London Apache Kafka Meetup, Feb 2018 End to end topologies on Apache Kafka An IoT use case CONNECT ANALYSE REACT
- 2. About Today Intros IoT High Level Architecture Build the Pipeline Q&A
- 3. Hi! It’s Stefan! @stepib https://github.com/stheppi CONNECT ANALYSE REACT Streaming Data Processing & Integration KAFKA - KUBERNETES
- 4. EVOLVESCALEBUILDIoT initiative challenges ★ 94% face challenges collecting and analysing IoT data ★ 26+ Billion devices by 2020, Gartner More Data, More Complexity ★ 2.500.000 trillion bytes produced daily REAL TIME DATA INGESTION + IMMEDIATE QUERIES = ACTIONABLE / TIMELY INSIGHTS IoT Data Flows
- 5. IoT data flow store datastream & process datasensor data
- 6. IoT data flow store datastream & process datasensor data E T L
- 7. InfluxDBsensor data MQTT InfluxDB Connector LENSES SQL PROCESSOR KAFKA CONNECT KAFKA CONNECT MQTT Connector KAFKA STREAMS
- 8. InfluxDBsensor data MQTT InfluxDB Connector LENSES SQL PROCESSOR KAFKA CONNECT KAFKA CONNECT MQTT Connector KAFKA STREAMS Synthetic Data With Moquette 1 Broker, data generate sensor data InfluxDB Docker
- 9. ® IoT & High Volumes, N devices & irregular intervals Real Time Analytics & Microservices Unlimited streams of data, async transfers Producers & Consumers decoupled operation Kafka Streams API Processing and analysing data in motion Multiple sources of data & long term storeage Kafka Connect API, move data with pluggable reusable & scalable connectors An open source streaming framework with messaging semantics where records are key-value pairs.
- 10. LENSES BOX For Kafka Development Single docker image with all services Growing list of examples & synthetic data generators. Stream Reactor 25+ connectors Kafka CLI autocomplete. Rest and WebSocket endpoints Redux JS Client Testing Libraries FREE! All you need to start working with Apache Kafka ecosystem
- 11. MQTT Source connector • Message Queueing Telemetry Transport messaging protocol • Optimised for connected devices and M2M protocol • Low bandwidth, high-latency, unreliable networks • Plain or SSL connections • Kafka Connect Query Language INSERT INTO kafka_topic SELECT * FROM /mqtt_source_topic [WITHCONVERTER=`myclass`] MQTT Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions
- 12. • Process data in motion with Lenses SQL • Create & Deploy Lenses SQL Processors • Native Kafka & Kubernetes Support Scalability • Visual Topology • Monitor and manage processors via LENSES Process & Analyse Data in motion
- 13. InfluxDB Sink connector • AVRO & JSON Support • Error handling ( Noop, Retry, Throw) • Field selection & field alias • Multiple inserts supported • Kafka Connect Query Language • MEASUREMENTS, KCQL: INSERTS • TAGS, KCQL: WITHTAG • TIMESTAMPS, KCQL: WITHTIMESTAMP • DURATION Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions INSERT INTO sensorMeasure SELECT * FROM sensor_data_avg WITHTAG (id, poweredBy=Lenses)
- 14. Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions InfluxDB Sink connector • Timestamp nanosecond • KCQL • No filter support • No function support • Throughput involves domain knowlege • Kafka Connect rebalances
- 15. visual data flow Visual and Interactive Topology for a 360 view of your data End-To-End Topology Management Visualise your Kafka-based data flows in real time
- 16. Wrap it up! CONNECT ANALYSE REACT Real Time, IoT pipelines in minutes! MQTT
- 17. Contact us www.landoop.com [email protected] Twitter @LandoopLtd Github github.com/landoop Gitter https://gitter.im/Landoop Slack https://launchpass.com/landoop-community Thank you!