Apache Apex - Hadoop Users Group
- 1. Pramod Immaneni <[email protected]> PPMC Member, Architect @DataTorrent Inc Dec 2nd, 2015 Stream Processing Architecture and Applications Apache Apex (incubating)
- 2. © 2015 DataTorrent Apex Platform Overview 2
- 3. © 2015 DataTorrent Apache Malhar Library 3
- 4. © 2015 DataTorrent Native Hadoop Integration 4 • YARN is the resource manager • HDFS used for storing any persistent state
- 5. © 2015 DataTorrent Application Programming Model 5 A Stream is a sequence of data tuples An Operator takes one or more input streams, performs computations & emits one or more output streams • Each Operator is YOUR custom business logic in java, or built-in operator from our open source library • Operator has many instances that run in parallel and each instance is single-threaded Directed Acyclic Graph (DAG) is made up of operators and streams Directed Acyclic Graph (DAG) Output StreamTuple Tuple er Operator er Operator er Operator er Operator
- 6. © 2015 DataTorrent Advanced Windowing Support 6 Application window Sliding window and tumbling window Checkpoint window No artificial latency
- 7. © 2015 DataTorrent Application Specification 7
- 8. © 2015 DataTorrent Partitioning and unification 8
- 9. © 2015 DataTorrent Advanced Partitioning 9
- 10. © 2015 DataTorrent Dynamic Partitioning 10 • Partitioning change while application is running ᵒ Change number of partitions at runtime based on stats ᵒ Determine initial number of partitions dynamically • Kafka operators scale according to number of kafka partitions ᵒ Supports re-distribution of state when number of partitions change ᵒ API for custom scaler or partitioner unifiers not shown 1a 2a 1b 2b 3 2b 1b 2c 3 2a 2d 1a 2b 1b 2c 3b 2a 2d 3a1a
- 11. © 2015 DataTorrent How tuples are partitioned 11 • Tuple hashcode and mask used to determine destination partition ᵒ Mask picks the last n bits of the hashcode of the tuple ᵒ hashcode method can be overridden • StreamCodec can be used to specify custom hashcode for tuples ᵒ Can also be used for specifying custom serialization tuple: { Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x11) Partition 00 1 01 2 10 3 11 4
- 12. © 2015 DataTorrent Custom partitioning 12 • Custom distribution of tuples ᵒ E.g.. Broadcast tuple:{ Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x00) Partition 00 1 00 2 00 3 00 4
- 13. © 2015 DataTorrent Fault Tolerance 13 • Operator state is checkpointed to a persistent store ᵒ Automatically performed by engine, no additional work needed by operator ᵒ In case of failure operators are restarted from checkpoint state ᵒ Frequency configurable per operator ᵒ Asynchronous and distributed by default ᵒ Default store is HDFS • Automatic detection and recovery of failed operators ᵒ Heartbeat mechanism • Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data • Application master state checkpointed
- 14. © 2015 DataTorrent Processing Guarantees 14 Atleast once • On recovery data will be replayed from a previous checkpoint ᵒ Messages will not be lost ᵒ Default mechanism and is suitable for most applications • Can be used in conjunction with following to ensure data is written once to store in case of fault recovery ᵒ Transactions with meta information, Rewinding output, Feedback from external entity, Idempotent operations Atmost once • On recovery the latest data is made available to operator ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient Exactly once • Operators checkpointed every window ᵒ Can be combined with transactional mechanisms to ensure end-to-end exactly once behavior
- 15. © 2015 DataTorrent Stream Locality 15 • By default operators are deployed in containers (processes) randomly on different nodes across the Hadoop cluster • Custom locality for streams ᵒ Rack local: Data does not traverse network switches ᵒ Node local: Data is passed via loopback interface and frees up network bandwidth ᵒ Container local: Messages are passed via in memory queues between operators and does not require serialization ᵒ Thread local: Messages are passed between operators in a same thread equivalent to calling a subsequent function on the message
- 16. © 2015 DataTorrent Data Processing Pipeline Example App Builder 16
- 17. © 2015 DataTorrent Monitoring Console Logical View 17
- 18. © 2015 DataTorrent Monitoring Console Physical View 18
- 19. © 2015 DataTorrent Real-Time Dashboards Real Time Visualization 19
- 20. © 2015 DataTorrent Resources 20 Apache Apex Community Page - http://apex.incubator.apache.org/
- 21. © 2015 DataTorrent We Need Your Vote (Today) 21 Introducing Apache Apex - Not Just Another Stream Processing Platform Next Gen Big Data Analytics with Apache Apex Enterprise-grade streaming under 2ms on Hadoop
- 22. End 22
- 23. © 2015 DataTorrent Extra Slides
- 24. © 2015 DataTorrent Partitioning and Scaling Out 24 • Operators can be dynamically scaled • Flexible Streams split • Parallel partitioning • MxN partitioning • Unifiers
- 25. © 2015 DataTorrent Fault Tolerance Overview 25 Stateful Fault Tolerance Processing Semantics Data Locality Supported out of the box – Application state – Application master state – No data loss Automatic recovery Lunch test Buffer server At least once At most once Exactly once Stream locality for placement of operators Rack local – Distributed deployment Node local – Data does not traverse NIC Container local – Data doesn’t need to be serialized Thread local – Operators run in same thread Data locality
- 26. © 2015 DataTorrent Machine Data Application Logical View 26
- 27. © 2015 DataTorrent Machine Data Application Physical View 27