seven-ways-to-run-flink-on-aws.pdf
- 2. whoami Software Engineer with Focus on Data-Intensive Applications Site Reliability Engineering
- 3. Stateful Computations over Data Streams Deployment YARN Mesos Kubernetes Apache Flink Apache Flink apache/flink
- 4. Stream Processor Stream Processor 1. "Functional" DSL 1. "Functional" DSL 3. Parallel Execution Graph 3. Parallel Execution Graph 2. Logical Execution Graph 2. Logical Execution Graph from("json-events").map(MyPojo::fromJson) .keyBy(MyPojo::getUserId) .timeWindow(SECONDS.of(10)) .count() .to("events-per-user")
- 6. Flink cluster Flink cluster Job Manager Task Manager Task Manager Task Manager
- 9. The recommended way The recommended way EMR EMR
- 10. https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html
- 12. EMR Architecture EMR Architecture Job Manager Task Manager Task Manager Task Manager YARN Container YARN Container YARN Container YARN Container
- 14. 👍 👍 Recommended Battle-tested 👎 👎 Need to manage Hadoop cluster Flink version lag (currently 1.8.0)
- 15. The do-it-yourself way The do-it-yourself way EC2 EC2
- 16. EC2 Architecture EC2 Architecture Job Manager Task Manager Task Manager Task Manager
- 18. 👍 👍 Fully customizable 👎 👎 Need to build everything on your own Need to manage EC2 instances
- 19. The containerized way The containerized way ECS + EC2 ECS + EC2
- 20. ECS Architecture (with EC2) ECS Architecture (with EC2) Job Manager Task Manager Task Manager Task Manager ECS Task ECS Service ECS Task ECS Task ECS Task ECS Service
- 21. 👍 👍 Useful model 👎 👎 Proprietary primitives Not common, little community support Need to manage EC2 instances
- 22. The CaaS way The CaaS way ECS + Fargate ECS + Fargate
- 23. ECS Architecture (with Fargate) ECS Architecture (with Fargate) Job Manager Task Manager Task Manager Task Manager ECS Task ECS Service ECS Task ECS Task ECS Task ECS Service
- 24. 👍 👍 Useful model No instance-level management required 👎 👎 Proprietary primitives Not common, little community support
- 25. The cloud-native way The cloud-native way EKS EKS
- 26. Kubernetes Kubernetes … is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling of applications.
- 27. EKS Architecture EKS Architecture Job Manager Task Manager Task Manager Task Manager Kubernetes pod Kubernetes pod Kubernetes pod Kubernetes pod Kubernetes deployment Kubernetes deployment
- 30. 👍 👍 Powerful primitives De-facto standard Strong ecosystem 👎 👎 Need to manage Kubernetes cluster
- 31. The hosted way The hosted way Kinesis Data Analytics for Java Kinesis Data Analytics for Java
- 32. Kinesis Data Analytics for Java Architecture Kinesis Data Analytics for Java Architecture
- 34. 👍 👍 Fully hosted State migration feature 👎 👎 Restricted to specific use case Failure scenarios Flink version lag (currently 1.6)
- 35. The serverless way The serverless way Lambda Lambda
- 38. 👍 👍 Fun experiment 👎 👎 Not a good fit
- 39. Recap Recap EMR The recommended way EC2 The do-it-yourself way ECS + EC2 The containerized way ECS + Fargate The CaaS way EKS The cloud-native way Kinesis Analytics for Java The hosted way Lambda The serverless way
- 40. What's next? What's next? State management & fault tolerance Monitoring & alerting Continuous Deployment Infrastructure as Code Tuning Checkpointing Upgrading Applications and Flink Versions Job Manager High Availability mbode/flink-prometheus-example Monitoring Flink with Prometheus (Flink Forward 2018) hashicorp/terraform Data Types & Serialization Tuning Checkpoints and Large State Improving throughput and latency with Flink's network stack (Flink Forward 2018)
- 41. [email protected] @mxpbode maximilianbo.de mbode