Kafka Streams
- 1. Kafka Streams Quando streams encontram tabelas
- 2. Cristiano Altmann Arquiteto Software https://www.linkedin.com/in/crisaltmann/
- 3. Matheus Alagia Eng. de Computação https://www.linkedin.com/in/matheusalagia/ https://ubots.com.br
- 4. “Data models are perhaps the most important part of developing software, because they have such a profound effect: not only on how the software is written, but also on how we think about the problem that we are solving.” - Martin Kleppmann Designing data-intensive applications
- 5. Microservices Independent deploy Low coupling Horizontal Scalability Technological choices Business time Independent teams Technological evolution Resilient
- 6. Increasingly we build ecosystem
- 7. Microservices is really just distributed systems!
- 8. The Hardest Part About Microservices: Your Data
- 9. Common Patterns ● Shared database ● Database per service ● Event driven approaches
- 10. The beginning... Shared Database serviceXserviceY
- 13. What is Streaming Processing? “Is some kind of computation over a Data Stream. First and foremost, a data stream is an abstraction representing an unbounded dataset. Unbounded means infinite and ever growing.” Kafka: The Definitive Guide
- 14. Stream processing is a programming paradigm... Request-Response Batch ProcessingStreaming Processing Throughput Latency
- 15. Stream-Processing Concepts Time Event time Local state Log append time State Processing time External state Time Windows Slide window Tumbling window Hopping Window
- 16. Stream-Processing Concepts Time Event time Local state Log append time State Processing time External state Time Windows Slide window Tumbling window Hopping Window
- 17. Stream-Processing Concepts Time Event time Local state Log append time State Processing time External state Time Windows Slide window Tumbling window Hopping Window
- 18. Calculation of the time window Size Advance interval How long remains updatable
- 20. The world always changes, and sometimes we are interested in the events that caused those changes, whereas other times we are interested in the current state of the world…. Stream-Table Duality
- 21. Systems that allow you to transition back and forth between the two ways of looking at data are more powerful than systems that support just one. - Neha Narkhede (Kafka: The definitive guide)
- 22. Kafka vs Kafka Stream ● Distributed log ● High available ● ⅓ Fortune 500 ● APIS: ○ Producer ○ Consumer ○ Connect ○ Streams ● Part of Kafka ecosystem ● Just a lib ● Simple API ● DSL
- 25. Code
- 26. External Lookup: Stream-Table Join
- 27. Processing with Local State
- 28. But sometimes we need state…. Moving the state to a database just push the problem to another layer.
- 31. Stateless is good!! ● Services start instantly ● Thread Safe ● Scaled out linearly ● Not shared state
- 33. Stream
- 34. Table
- 35. Stream
- 37. Join ● Stream - Stream ● Stream - Table ● Table - Table
- 38. Stream - Stream (Windowed-join)
- 39. Stream - Table
- 45. Obrigado!
- 46. Perguntas?