Extending DevOps to Big Data Applications with Kubernetes
9 likes3,468 views
The document provides an overview of integrating DevOps practices with big data applications using tools like Kubernetes and Apache Spark. It discusses the evolution of big data architectures from batch to streaming solutions and emphasizes the importance of collaboration between development and operations teams to improve software delivery. Additionally, it highlights challenges in testing and maintaining big data applications in production environments, and demonstrates the potential of using Kubernetes for deploying Spark applications in a cloud-native context.
![DOCKER
Docker has revolutionized the way we build
and ship software.
Containers can be created using a
Dockerfile (example on the right)
There are at least 10 different ways to
create Docker containers
A container is not a VM (even if it seems
really close to it)
A container hosts a single application
Containers need to be orchestrated to
deliver their full power
FROM ubuntu:16.04
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:
RUN echo "deb http://repo.mongodb.org/apt/ubuntu $(cat
RUN apt-get update && apt-get install -y mongodb-org
# Create the MongoDB data directory
RUN mkdir -p /data/db
# Expose port #27017 from the container to the host
EXPOSE 27017
# Set /usr/bin/mongod as the dockerized entry-point app
ENTRYPOINT ["/usr/bin/mongod"]](https://image.slidesharecdn.com/devops-bigdata-kubernetes-voxxed-bucharest-2017-170313172555/85/Extending-DevOps-to-Big-Data-Applications-with-Kubernetes-24-320.jpg)
Extending DevOps to Big Data Applications with Kubernetes
- 1. EXTENDING DEVOPS TO BIG DATA APPLICATIONS WITH KUBERNETES
- 2. ABOUT ME NICOLA FERRARO Software Engineer at Red Hat Working on Apache Camel, Fabric8, JBoss Fuse, Fuse Integration Services for Openshift Previously Solution Architect for Big Data systems at Engineering (IT) @ni_ferraro
- 3. SUMMARY Overview of Big Data Systems and Apache Spark DevOps, Docker, Kubernetes and Openshift Origin Demo: Oshinko, Kafka, Spring-Boot and Fabric8 tools
- 4. BIG DATA Big Data ≈ Machine Learning at Scale Big Data systems are capable of handling data with high: Volume Terabytes, petabytes, ... Log files Transactions Velocity Streaming data Micro batching Near real-time Variety Structured data Images Videos Free text Volume Velocity Variety Here > 100 machines
- 5. EVOLUTION OF BIG DATA SOFTWARE http://www.slideshare.net/nicolaferraro/a-brief-history-of-big-data-48525942 Hadoop Pig Hive (a whole zoo) Spark
- 6. EVOLUTION OF BIG DATA INFRASTRUCTURE Commodity hardware (Really ?) Specialized hardware (Appliances) The Cloud
- 7. EVOLUTION OF ARCHITECTURES: BATCH The first Big Data architecture was heavily based on Hadoop: MapReduce + HDFS. Some commercial advertising motivations: Extract meaningful information from raw data: "companies analyze around <<put-a- random-number-between-0-and-20- here>> % of the data they produce" More data = more precision for machine learning algorithms = better insights Assist the decision making process: predict the future using all the available data SELECT page, count(*) FROM logs
- 8. ARCHITECTURES: BATCH Hadoop (HDFS + MapReduce) Nodes App Server Logs + Data (Flume) Report (DWH) Client Batch Processing (Hive) MR
- 9. EVOLUTION OF ARCHITECTURES: HYBRID Second generation architectures were focused to improve user experience through machine learning: Not everything could be executed in streaming, for performance reasons Execute heavyweight steps offline in batches (generate the "view", e.g. a machine learning model) Execute lightweight steps online with streaming applications The view must be refreshed periodically +
- 10. ARCHITECTURES: HYBRID (LAMBDA) Hadoop Nodes App Server Client Batch Processing NoSQL + Messaging + Other Streaming Processing
- 11. EVOLUTION OF ARCHITECTURES: STREAMING The new generation of architectures are streaming only: Provide immediate feedback to the user No need to execute offline work Provide a personalized experience to each user Here we focus (mainly) on streaming applications ...
- 12. ARCHITECTURES: STREAMING (KAPPA) Hadoop Nodes App Server Client NoSQL + Messaging + Other Streaming Processing Here we focus (mainly) on streaming applications ...
- 13. APACHE SPARK One of the main reason of Spark success is that it allows to define, by design, with a unified model: Batch processing and streaming (MapReduce / Storm) Multi-language: core in Scala, can be also used in Java, Python and R Declarative semantics (SQL), procedural and functional programming (Hive / MapReduce / Pig) General purpose data processing and machine learning (MapReduce / Mahout) Logistic regression in Spark vs. Hadoop (it was so in 2013) And the performance are really impressive!
- 14. SPARK ARCHITECTURE Driver Cluster Manager Driver (2nd app) Workers Driver App (main) Executor Executor Single Worker Tasks: "do something on a data partition" Assign executors to the driver app Define the app workflow Do the dirty job Logical viewPhysical view
- 15. SPARK ARCHITECTURE: DATA DISTRIBUTION HDFS / S3 Spark Executo r Spark Executo r Spark Executo r r/w Spark Driver Kafka Partition Spark Executo r Spark Executo r Spark Executo r Spark Driver Kafka Partition Kafka Partition HDFS / S3 HDFS / S3 Why distribution? ... to spread the work across thousands of workers
- 16. SPARK DEMO A simple quickstart showing a batch application doing a word count. The purpose is showing which part of the software is executed in the driver and which parts are sent to the remote executors.
- 17. SPARK: WHAT CAN GO WRONG ??? Programming with Spark is awesome, but there are a lot of problems you need to solve: Most of the problems do not happen in a standalone cluster (serialization issues, classloading issues, network issues, ...) Something that works well with a small dataset hangs forever in production: you must test it earlier with a lot of data! Spark applications must be tuned: decide caching steps, decide partitioning, reduce the number of shuffles, do not "collect()" For streaming apps: Exactly-once message semantics cannot be guaranteed all the times: you have to write idempotent operations to take into account duplicate messages org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCl at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$Closur at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:1 at org.apache.spark.SparkContext.clean(SparkContext.scala:2055) at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:324) at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:323) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationSco at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationSco at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.map(RDD.scala:323)
- 18. AND SPARK IS THE MINOR PROBLEMEvery operation is executed by different software distributed in hundreds of machines. Production different from the development environment: difficult to maintain software aligned (patches and upgrades of NoSQL databases, message brokers, ...): you need to test the integration with different versions of external systems The whole system randomly doesn't work: e.g. you expect a message in a queue or you expect a result in a query, and your expectations are not satisfied Debugging is hard: it's better to fix bugs earlier, but bugs appear late in production! There are consistency problems in some NoSQL databases: the CAP theorem Too many steps from development to release, with different teams involved: development, qa, operations I can continue...
- 19. THE 2 MAIN PROBLEMS 1 The local development environment is usually too different from production (replication and versions), so you find bugs late 2 Big Data applications are difficult to test effectively, because configuring a good testing environment is a hard and long task: it's difficult to debug bugs all together How? we can apply DevOps practices with Kubernetes ... Easy Solution: Test early Test more Test the production software
- 20. SUMMARY Overview of Big Data Systems and Apache Spark DevOps, Docker, Kubernetes and Openshift Origin Demo: Oshinko, Kafka, Spring-Boot and Fabric8 tools
- 21. DEVOPS No common definition Combination of software development, qa and operations: working together to release working software frequently Applying common dev and agile techniques to operations It's a cultural change, rather than just a change in development and administration tools Common practices: Infrastructure as code Continuous integration Continuous delivery DEV OPS
- 22. WHY DEVOPS Increased communication and collaboration among: dev, ops, qa, business Fast time to market: accomodating business and customer needs quickly to have competitive advantage Embrace change, do not fear it Embrace innovation (and team engagement), through usage of new technologies Improved quality of software through automated testing DEV OPS
- 23. A DEVOPS WORKFLOW The key point of DevOps is being able to collaborate on the infrastructure the same way you collaborate on software: Common shared repository (SCM) Pull requests Peer review Automated checks (correctness, quality) Continuous integration Continuous delivery Testing the infrastructure like you test software: scalability, fault tolerance, backup, recovery options. DevOps practices become difficult to apply and uneffective when your infrastructure does not follow the software release cycle.
- 24. DOCKER Docker has revolutionized the way we build and ship software. Containers can be created using a Dockerfile (example on the right) There are at least 10 different ways to create Docker containers A container is not a VM (even if it seems really close to it) A container hosts a single application Containers need to be orchestrated to deliver their full power FROM ubuntu:16.04 RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com: RUN echo "deb http://repo.mongodb.org/apt/ubuntu $(cat RUN apt-get update && apt-get install -y mongodb-org # Create the MongoDB data directory RUN mkdir -p /data/db # Expose port #27017 from the container to the host EXPOSE 27017 # Set /usr/bin/mongod as the dockerized entry-point app ENTRYPOINT ["/usr/bin/mongod"]
- 25. DOCKER ARCHITECTURE Docker is the building block of infrastructure as code: The image is completely described by the Dockerfile Any container is stateless (although stateful data can be attached with volumes). This means: You don't ssh your server and change the configuration You rebuild a new version of the application when you need to change something "Deployment" means basically replacing a container with a different container
- 26. KUBERNETES Cloud platform, with several deployment options (including local and private cloud), to Orchestrate (Docker) containers: Born at Google Production ready (Pokemon Go!) Provides: Application composition in namespaces Virtual networks Service discovery Load balancing Auto (and manual) scaling Automatic recovery Health checking Rolling upgrades Monitoring Logging .... kubectl run --image=nginx nginx-app --port=80 oc run --image=nginx nginx-app --port=80 # or better.. # oc new-app nginx --name nginx-app2 BOTH Kuebertes and Openshift Origin are Open Source projects!
- 27. KUBERNETES ARCHITECTURE Very high level architecture of Kubernetes. Note: it resembles the Spark architecture seen in previous slides (with the Kubernetes master playing the role of the cluster manager), but we want to deploy a whole Spark application inside Kubernetes (including the driver).
- 28. KUBERNETES FOR LOCAL DEV Minikube https://github.com/kubernetes/minikube OPENSHIFT FOR LOCAL DEV Minishift "oc cluster up" https://github.com/minishift/minishift $ minikube start Starting local Kubernetes cluster... Running pre-create checks... Creating machine... Starting local Kubernetes cluster... $ kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080 $ minishift start ... $ oc cluster up -- Checking OpenShift client ... OK -- Checking Docker client ... OK ...
- 29. SUMMARY Overview of Big Data Systems and Apache Spark DevOps, Docker, Kubernetes and Openshift Origin Demo: Oshinko, Kafka, Spring-Boot and Fabric8 tools
- 30. OPENSHIFT ORIGIN DEMO A simple recommender system using collaborative filtering on Openshift origin. Deploy a: Spring-boot microservice Kafka Broker Spark Cluster Into openshift origin. Source code: https://github.com/nicolaferraro/voxxed-bigdata-spark Rating **** Rating **** Recommendations Recommendations Spark Cluster
- 31. SPARK ON KUBERNETES The problem: A Spark application needs a cluster of n worker machines to run You need 1 cluster manager node The application runs on 1 driver node We want to deploy everything inside Kubernetes or Openshift Origin Basically, every piece of the Spark architecture should become a POD. There are multiple solutions. I'm going to introduce the Oshinko project. http://radanalytics.io/ https://github.com/radanalyticsio Driver Cluster Manager Recall the Spark architecture PODs Worker Worker Worker
- 32. OSHINKO Making Apache Spark Cloud Native on Openshift Origin. W WW CM D Spark Cluster W WW CM D Spark Cluster 1 cluster per application! Deployed in the application namespace Oshinko console
- 33. DEMO TIME We will deploy the Oshinko rest server and then deploy the spark streaming application. It will connect to Kafka to retrieve ratings and return personalized (not really) recommendations. Sources: https://github.com/nicolaferraro/voxxed-bigdata-spark I used a basic implementation of the slope one algorithm ( ) for collaborative filtering. It's just an example of "real life" custom Spark application that sometimes doesn't work :) https://en.wikipedia.org/wiki/Slope_One
- 34. TESTING One of the major advantages of making a cloud native Spark application is the possibility of executing system tests on software identical to production. Production pods Optional testing tools or load generators Test driver (inside or outside the namespace) You can create virtual test environments on the fly and do assertions. Demo: using fabric8-arquillian and the fabric8 kubernetes client
- 35. TESTING: SOME SCENARIOS Some tests that you may want to run: Deploy the production infrastructure (or part of it) and run a suite of functional tests to ensure every piece works as expected to produce the result Deploy the production infrastructure and do performance tests using external or injected data Deploy the production infrastructure (or part of it) and check that every piece is healthy. Then randomly kill pods while sending a load to test the system availability (system tests) The Chaos Monkey!
- 36. FABRIC8 We have used some tools from Fabric8: Fabric8 Maven Plugin: to build Kubernetes resources for Java projects Docker Maven Plugin: used by the fabric8 maven plugin to create docker images Fabric8 Arquillian: to create system tests from artifacts using the fabric8 plugin Fabric8 Kubernetes Client: to interact with Kubernetes resources (pods, services, etc.) Fabric8 is a complete development platform for Kubernetes! Uses Jenkins Pipelines and to provide: Continuous integration Continuous delivery Out of the box! https://fabric8.io/
- 37. QUESTIONS?
- 38. Q/A What about storage and data locality? In cloud native applications, using a storage system like s3 has many advantages over HDFS (costs, availability, usage patterns in testing). Performance of HDFS are superior (6x), but you can use larger clusters for a limited amount of time to gain the same performance. More info: Can you scale the application directly from Openshift, or you always need to change the code? Sure, you can use Openshift to scale the application. Beware that, in a pure DevOps approach, changing the configuration of a system at runtime is not something you should do, especially on production. You lose the benefits of having all the software and configuration in a "executable" state, and you lose the effectiveness of system tests because tests will not run on an environment identical to production. As an alternative, you can use the auto-scaling feature (enabling it in the code). Is the Spark demo using a machine learning algorithm? The demo is using a basic collaborative filtering algorithm called "slope one" (the focus of this talk is not machine learning). Spark provides many algorithms out of the box, also for collaborative filtering. Many of them are supposed to be executed in a batch application, rather than in a streaming app. http://qr.ae/TAF4cN
- 39. THANKS Follow me on Twitter! @ni_ferraro