Тарас Кльоба "ETL — вже не актуальна; тривалі живі потоки із системою Apache Kafka"
Download as PPTX, PDF0 likes106 views
This document discusses the limitations of traditional Extract, Transform, Load (ETL) systems and how Apache Kafka provides a better solution for streaming data. It outlines some of the challenges with isolation levels, triggers, and change data capture in high load systems. The document then introduces Apache Kafka as a horizontally scalable and fault tolerant streaming platform that can be used to build real-time data pipelines. It demonstrates how Kafka Connect and the Kafka Streams API can be used for Extract, Transform, Load (ETL) and streaming data integration in a more reliable way compared to previous approaches.
Тарас Кльоба "ETL — вже не актуальна; тривалі живі потоки із системою Apache Kafka"
- 1. ETL is Dead. Long Live Streams with Apache Kafka. Taras Kloba, BI Team Lead/Data Architect, Intellias
- 2. Agenda • About me; • One problem in data transferring; • Ways to solve this problem; • About Apache Kafka; • Demo of reliable data sending; • Questions?
- 3. Taras Kloba • 7 years of experience with databases; • Certified Data Engineer on Google Cloud; • Certified Expert Microsoft SQL Server; • Co-organizer “SQL Saturday” in Lviv and Krakow; • Trainer, speaker, consultant; • Owner “SQL” trademark in Ukraine . SQL.ua, CEO/Founder Intellias, BI Team Lead/Data Architect Quick facts (Q62JCJRJGY77)(9DG5NZ4EVP7A) (M2HE6LPRJ6MV)
- 4. My current project: One of the biggest B2B software solution for the iGaming industry in the World. +300 GB new data every day
- 5. Previous legacy system 00:00 00:01 00:02 00:03 00:04 00:05 00:06 00:07 00:08 00:09 SELECT * FROM fact_transactions WHERE upd BETWEEN ’2018-11-03 00:00:00’ AND ’2018-11-03 00:04:00’ SELECT * FROM fact_transactions WHERE upd BETWEEN ’2018-11-03 00:04:00’ AND ’2018-11-03 00:08:00’
- 6. ?
- 7. Previous legacy system 00:00 00:01 00:02 00:03 00:04 00:05 00:06 00:07 00:08 00:09 SELECT * FROM fact_transactions WHERE upd BETWEEN ’2018-11-03 00:00:00’ AND ’2018-11-03 00:04:00’ SELECT * FROM fact_transactions WHERE upd BETWEEN ’2018-11-03 00:02:00’ AND ’2018-11-03 00:06:00’
- 8. ?
- 9. Phantom reads (classical definition)
- 10. Phantom reads (in our cases) Tnx: 1 2018-11-03 12:00:00 Tnx: 2 2018-11-03 12:01:00 Tnx: 2 commit 12:03:00 Tnx: 1 commit 12:05:00 SELECT * FROM fact_transactions WHERE upd BETWEEN ’2018-11-03 11:58:00’ AND ’2018-11-03 12:04:00’ Trans_id Upd 2 2018-11-03 12:01:00
- 11. ?
- 12. #1. Isolation levels - Serializable With a lock-based concurrency control DBMS implementation, serializability requires read and write locks (acquired on selected data) to be released at the end of the transaction. Also range-locks must be acquired when a SELECT query uses a ranged WHERE clause, especially to avoid the phantom reads phenomenon. A not best solution for high load solutions.
- 13. #2. Triggers Traditionally, the most common technique used for capturing events was to use database or application-level triggers. The reason why this technique is still very widespread is due to its simplicity and familiarness. A not best solution for high load solutions.
- 14. #3. Change Data Capture is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data. Also, Change data capture (CDC) is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources. (Wikipedia) CDC solutions occur most often in data- warehouse environments since capturing and preserving the state of data across time is one of the core functions of a data warehouse.
- 15. Apache Kafka Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
- 17. Typical data flow in companies
- 18. Streaming platform to coordinate all data flows.
- 19. Kafka Connect API (E and L in Streaming ETL) • Scalability: Leverages Kafka for scalability • Fault tolerance: Builds on Kafka’s fault tolerance model • Management and monitoring: One way of monitoring all connectors • Schemas: Offers an option for preserving schemas from source to sink
- 20. Kafka Connect. Create new connector.
- 21. Kafka’s streams API (The T in ETL) • Easiest way to do stream processing using Kafka; • True event-at-a-time stream “ processing; no microbatching; • Dataflow-style windowing based on “ event-time; handles late-arriving data
- 22. Kafka Stream API. Create new processor
- 23. Demo
- 24. Conclusion • Apache Kafka is robust • Triggers will keep your data in sync but can have significant performance overhead • Utilizing a logical replication slot can eliminate trigger overhead and transfer the computation load elsewhere • Not a panacea: still need to use good architectural patterns
- 25. Questions?
- 26. Thank you! Taras Klioba +38 093 74 876 15 [email protected]