Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows
0 likes156 views
This document discusses using Kafka to stream IoT time-series data from devices to InfluxDB for real-time analytics and insights. It introduces Lenses.io as an open-source streaming data platform that enables connecting Kafka to InfluxDB via reusable connectors without code. The document demonstrates how Lenses provides a unified view and SQL support for processing, filtering, and manipulating streaming data across infrastructure and application layers in a distributed, scalable manner on Kubernetes.
Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows
- 1. DataOps on streaming data Kafka to InfluxDb via Kubernetes Native Flows - IoT demo Chris Kotsis lenses.io
- 2. #About lenses.io Enterprise DataOps over Streaming Data
- 3. #About me
- 4. #DataOps Embrace Collaboration Eliminate Friction, turn data to Value Faster
- 5. #DataOps Personas people who ● Collect and Prepare the data ● Analyse the data ● Use the findings for business value
- 6. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025”
- 7. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025” Its Management and Analysis will become harder and continue to break traditional tools
- 8. The IoT Data Flow
- 9. How to implement a Streaming Architecture for IoT Time-Series data and Real-Time insights?
- 10. Infrastructure Layer Where the Data lives Self-service Data Access, Multi-tenancy, Security, Governance to Accessibility & Visibility for ALL
- 12. IoT & kafka High Volumes, N devices & irregular intervals Real Time Analytics & Microservices Multiple sources of data & long term storage An open source streaming framework with messaging semantics where records are key-value pairs Unlimited streams of data, async transfers Producers & Consumers decoupled operation Kafka Streams API Processing and analysing data in motion Kafka Connect API, Move data with pluggable reusable & scalable connectors
- 13. Application Layer Multiple Distributed Microservices for complex Data Flows
- 16. How Lenses integrates with the Application Layer?
- 17. Lenses SQL Processors ● Simply Filter, Enrich, Split & Bind your data ● Manipulate Live Streams of Data ● Scalable (Kubernetes Native)
- 18. Connect Kafka to InfluxDB Real Time Ingestion Distributed Fault tolerance Scalability Error Handling Monitoring & Alerting Governance & Security Easy Data Manipulation ...
- 19. Lenses Connectors: Binding external sources is a few clicks away
- 20. InfluxDB sink connector by Lenses.io Kafka to InfluxDB No code required CLI/UI/API/Monitor SQL Support Multiple inserts supported AVRO & JSON Support Supported Features: MEASUREMENTS (SQL: INSERT TO) TIMESTAMPS (SQL: WITHTIMESTAMP) TAGS (SQL: WITHTAG) DURATION
- 21. The uncontrolled “Chaos” of modern data streams
- 22. Lenses Topology, one view to bind them all
- 25. What it takes to demo this architecture
- 26. Lenses Box ● Always Free for Developers ● Single Broker setup with the ecosystem services ● 25+ Open source Kafka connectors ● Synthetic Data generators and running examples ● Lenses intuitive UI ● The powerful Lenses SQL ● Works on your Laptop, works on Cloud
- 27. Demo
- 28. Thank you and <3 Q&A