Containerizing Distributed Pipes
1 like633 views
The document discusses the concept of containerizing distributed pipes, highlighting the use of Apache Kafka for stream processing and its integration with Docker containers. It covers various aspects of Kafka, including its architecture, message processing, and tool application stack, while also referencing the UNIX philosophy for stream processing. The summary concludes with a recap of the challenges faced in implementing Kafka streams and the need for durable storage and capacity planning.
![Unix Pipe Recap
• […]In Unix-like computer operating systems, a
pipeline is a sequence of processes chained
together by their standard streams, so that the
output of each process (stdout) feeds directly as
input (stdin) to the next one.
3](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-3-320.jpg)
![Containers
• […] Operating-system-level virtualization is a
server virtualization method in which the kernel of
an operating system allows the existence of
multiple isolated user-space instances
19](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-19-320.jpg)
![clj-kstream-cutter
Edmilson Alves 0 Edmilson Alves -LRB- born February
17 , 1976 -RRB- , is a Brazilian midfielder who currently
plays for Roasso Kumamoto in the J. League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
31
Input:
Output:](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-31-320.jpg)
![[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born, …]
Edmilson ~10 Alves ~8
clj-kstream-hh
32
Input:
Output:](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-32-320.jpg)
![(defn- heavy-hitter-processor
"Main stream processor takes a configuration and a mapper function to apply."
[conf]
(let [streamBuilder (-> (new TopologyBuilder)
(.addSource (:name conf) string_dser string_dser (into-array [(:input-topic conf)]))
(.addProcessor "HeavyHitter"
(reify ProcessorSupplier
(get [this]
(get-processor)))
(into-array [(:name conf)]))
(.addStateStore
(->> (Stores/create storeName)
(.withStringKeys)
(.withLongValues)
(.inMemory)
(.build))
(into-array ["HeavyHitter"]))
(.addSink
"Sink"
(:output-topic conf)
string_ser
string_ser
(into-array ["HeavyHitter"])))]
(.start
(KafkaStreams.
streamBuilder
(get-props conf)))))
36
clj-kstream-hh Topology](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-36-320.jpg)
![(defn ^Processor get-processor []
(reify org.apache.kafka.streams.processor.Processor
(init [this context]
(.schedule (:context @application-state) (:time-window @application-state))
(swap! hh/state assoc
:top-n 5
:number-of-hashfn 10N
:bucket-size 1000N)
(reset! hh/hitter ^(priority-map))
(reset! hh/min-sketch (make-array Integer/TYPE 10N 1000N)) …)
(process [this key value]
(debug "Process (k,v)::" key value)
(hh/sketch-value value)
(hh/add-to-hitter value) …)
(punctuate [this timestamp …)
(close [this]
(.close (:store @application-state)))))
37
clj-kstream-hh Processor](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-37-320.jpg)
![Our Distributed Tool
Application Stack in Docker
Edmilson Alves 0 Edmilson Alves -LRB- born February 17 , 1976 -RRB- , is
a Brazilian midfielder who currently plays for Roasso Kumamoto in the J.
League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
Alves ~10 Alves ~8
{“name“: “Alves“, “count“: 18, “time“: “January 26th 2017, 17:03:00.000”}
Alves 18
41](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-41-320.jpg)
![Containerising
Distributed Pipes
a story about convenience
(thanks (listening [this]))
47](https://image.slidesharecdn.com/hagentoenniesdistpipeswed02082017-170222164012/85/Containerizing-Distributed-Pipes-47-320.jpg)
Containerizing Distributed Pipes
- 1. Containerising Distributed Pipes a story about convenience Hagen Tönnies www.linkedin.com/in/hagen-toennies 1
- 2. Agenda • Introduction • A Distributed Pipe • Tool Application Stack • Recap 2
- 3. Unix Pipe Recap • […]In Unix-like computer operating systems, a pipeline is a sequence of processes chained together by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one. 3
- 4. Unix Philosophy • Write programs that do one thing and do it well. • Write programs to work together. • Write programs to handle text streams, because that is a universal interface. Peter H. Salus says about the unix philosophy 4
- 6. Apache Kafka Cluster Kafka Broker 6 Zookeeper
- 14. Kafka Streaming API 14 Low-Level High-Level • Topology Builder • Stream and Table abstractions • Custom Aggregators • Simple Transformation • Custom Processors • Simple Joins of Tables and Streams
- 15. Kafka Streaming API 15 Table Stream (change-log) alice | 1 alice | 1 charlie | 1 alice | 2 charlie | 1 („alice“ , 1) („charlie, 1) („alice, 2)
- 16. Kafka featuring Unix (split %1 "s") 16
- 17. Kafka featuring Unix (split %1 "s") kafka Message Broker Stream processing job 17
- 18. Kafka featuring Unix (sketch %1)(split %1 "s") (store %1)(agg-by-key %1) 18
- 19. Containers • […] Operating-system-level virtualization is a server virtualization method in which the kernel of an operating system allows the existence of multiple isolated user-space instances 19
- 20. Containers • docker run -t -i CONTAINER-NAME …args • java -jar …args 20
- 21. Our Distributed Pipe Application Stack in Docker ZooKeeper kafka Discovering and configuring services in your infrastructure. Distributed append log a.k.a Message Broker enables highly reliable distributed coordination. Provides a distributed full-text search engine 21
- 22. consul: image: qnib/alpn-consul networks: - network cpu_shares: 4 mem_limit: 1g environment: - DC_NAME=es ports: - “8500:8500" 22
- 23. zookeeper: image: qnib/zookeeper extends: file: base.yml ports: - "2181:2181" ZooKeeper 23
- 24. kafka-broker: image: kafka:0.10.0.1 extends: file: base.yml service: gaikai volumes: - /tmp/kafka-logs ports: - "9092:9092" kafka 24
- 25. elasticsearch: image: elasticsearch:1.7 command: "elasticsearch -Des.cluster.name=dp-es -Dnetwork.bind_host=0.0.0.0" environment: - ES_HEAP_SIZE=2g ports: - "9200:9200" - "9300:9300" 25
- 26. 26
- 28. 28 kafka kafka streams Stream Processing Recap
- 29. • clj-kstream-cutter • clj-kstream-hh • clj-kstream-string-long-window-aggregate • clj-kstream-elasticsearch-sink https://github.com/sojoner Our Distributed Tool Application Stack in Docker 29
- 31. clj-kstream-cutter Edmilson Alves 0 Edmilson Alves -LRB- born February 17 , 1976 -RRB- , is a Brazilian midfielder who currently plays for Roasso Kumamoto in the J. League Division 2 . [ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …] 31 Input: Output:
- 32. [ Edmilson, Alves, 0, Edmilson, Alves, LRB, born, …] Edmilson ~10 Alves ~8 clj-kstream-hh 32 Input: Output:
- 33. 33 Count Min (CM) sketch
- 35. And ~10 Bob ~7 Alice ~5 Foo ~3 Bar ~2 take top_n retrieve sketched value Heavy Hitter for t_1 35 Heavy Hitter
- 36. (defn- heavy-hitter-processor "Main stream processor takes a configuration and a mapper function to apply." [conf] (let [streamBuilder (-> (new TopologyBuilder) (.addSource (:name conf) string_dser string_dser (into-array [(:input-topic conf)])) (.addProcessor "HeavyHitter" (reify ProcessorSupplier (get [this] (get-processor))) (into-array [(:name conf)])) (.addStateStore (->> (Stores/create storeName) (.withStringKeys) (.withLongValues) (.inMemory) (.build)) (into-array ["HeavyHitter"])) (.addSink "Sink" (:output-topic conf) string_ser string_ser (into-array ["HeavyHitter"])))] (.start (KafkaStreams. streamBuilder (get-props conf))))) 36 clj-kstream-hh Topology
- 37. (defn ^Processor get-processor [] (reify org.apache.kafka.streams.processor.Processor (init [this context] (.schedule (:context @application-state) (:time-window @application-state)) (swap! hh/state assoc :top-n 5 :number-of-hashfn 10N :bucket-size 1000N) (reset! hh/hitter ^(priority-map)) (reset! hh/min-sketch (make-array Integer/TYPE 10N 1000N)) …) (process [this key value] (debug "Process (k,v)::" key value) (hh/sketch-value value) (hh/add-to-hitter value) …) (punctuate [this timestamp …) (close [this] (.close (:store @application-state))))) 37 clj-kstream-hh Processor
- 38. 38 clj-kstream-hh Container clj-kstream-hh: image: sojoner/clj-kstream-hh:0.1.0 hostname: clj-kstream-hh container_name: clj-kstream-hh extends: file: base.yml service: sojoner command: "--broker kafka-broker:9092 --input-topic mapped-test-json --output-topic heavy-hitters --window-size 1 --name stream-hh"
- 39. Input: Output: Alves ~8 Alves ~10 Edmilson ~5 Edmilson~3 Alves ~18 Edmilson ~8 (key, value) 39 clj-kstream-string-long- window-aggregate
- 40. clj-kstream-elasticsearch-sink Input: Output: { “name“: “Alves“, “count“: 18, “time“: “January 26th 2017, 17:03:00.000” } Alves ~18 40
- 41. Our Distributed Tool Application Stack in Docker Edmilson Alves 0 Edmilson Alves -LRB- born February 17 , 1976 -RRB- , is a Brazilian midfielder who currently plays for Roasso Kumamoto in the J. League Division 2 . [ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …] Alves ~10 Alves ~8 {“name“: “Alves“, “count“: 18, “time“: “January 26th 2017, 17:03:00.000”} Alves 18 41
- 42. Our Distributed Tool Application Stack in Docker 42
- 43. From the development setup… $ export DOCKER_HOST=tcp://my.desktop.de:2576 43
- 44. …to a datacenter setup $ export DOCKER_HOST=tcp://my.datacenter.de:2576 Build a Docker Swarm 44
- 45. Current Challenges • Kafka Streams still at least once • Persistent and durable Storage • Still need capacity planning • Testing / Debugging is still a challenge • Consistency of the state storage • Processing Time vs. Event Time • How about Amdahl’s law 45
- 46. Recap (split %1 "s") 46 as Message broker as Stream Processor (split %1 "s")
- 47. Containerising Distributed Pipes a story about convenience (thanks (listening [this])) 47
- 48. Sources 1 • http://kafka.apache.org/ • https://martin.kleppmann.com/2015/05/06/data-agility-at-strata.html • https://speakerdeck.com/ept/kafka-and-samza-distributed-stream- processing-in-practice • https://github.com/mhausenblas/dnpipes • https://en.wikipedia.org/wiki/Pipeline_%28Unix%29 • https://zookeeper.apache.org/doc/trunk/zookeeperOver.html • https://github.com/sojoner/container-stacks/tree/master/ kafkaelasticsearch 48
- 49. Sources 2 • https://kafka.apache.org/documentation/streams#streams_processor • https://kafka.apache.org/documentation/streams#streams_dsl • https://hub.docker.com/r/sojoner/clj-kstream-elasticsearch-sink/ • https://hub.docker.com/r/sojoner/clj-kstream-cutter/ • https://hub.docker.com/r/sojoner/clj-kstream-hh/ • https://hub.docker.com/r/sojoner/clj-kstream-string-long-window- aggregate/ • https://blog.acolyer.org/2016/07/21/time-adaptive-sketches-ada- sketches-for-summarizing-data-streams/ 49