Making Apache Kafka Elastic with Apache Mesos
Download as pptx, pdf25 likes7,971 views
This document discusses Kafka on Mesos, which allows Kafka to elastically scale on the Mesos cluster. It provides a quick introduction to Mesos and Kafka. The Kafka on Mesos project aims to make Kafka elastic by allowing smart broker assignment and configuration changes, rolling restarts, and scaling the cluster up and down both automatically and manually. It includes a scheduler and executor that manage the Kafka brokers. The CLI and REST API allow adding, updating, removing, starting, and stopping brokers as well as rebalancing topics. An example demonstrates launching 20 brokers in seconds on Mesos. Finally, it notes that Kafka is available on DCOS for deployment and management on Mesosphere DC/OS.
1 of 23
Downloaded 80 times
Ad
Recommended
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Joe Stein Making Distributed Data Persistent Services Elastic (Without Losing All Your Data)
Mesos Overview
Roles, Attributes, Constraints, Modules, Hooks
Marathon, Kafka, Cassandra, HDFS, MySQL, YARN and more!
Get started with Developing Frameworks in Go on Apache Mesos
Get started with Developing Frameworks in Go on Apache MesosJoe Stein Apache Mesos provides a platform for building distributed systems. Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. How to use that platform and what to make of it becomes a complex task requiring not only understanding of where the system has been but also where it is going. Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework in Go, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but should also be used for custom internal R&D built software applications too.
Apache Kafka, HDFS, Accumulo and more on Mesos
Apache Kafka, HDFS, Accumulo and more on MesosJoe Stein Slides from my talk about using Mesos to run persistant services like Apache Kafka, Apache HDFS, Apache Accumulo on Apache Mesos with Marathon
Containerized Data Persistence on Mesos
Containerized Data Persistence on MesosJoe Stein Elodina is a startup focused on supporting open source software like Mesos frameworks. They have experience implementing and assisting with Kafka, Mesos, Hadoop, Cassandra, and other big data systems. The document discusses containerizing data persistence on Mesos with various data systems like Kafka, MySQL, Cassandra and HDFS. It provides overviews of file systems, databases, and Mesos concepts like roles and resources. It also summarizes efforts to run these data systems as Mesos frameworks.
Developing Frameworks for Apache Mesos
Developing Frameworks for Apache MesosJoe Stein Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Introduction To Apache Mesos
Introduction To Apache MesosJoe Stein This document provides an introduction and overview of Apache Mesos. It begins by describing Mesos' origins at companies like Google and how it enables fine-grained resource sharing in data centers. It then discusses concepts like schedulers, executors, frameworks and how Mesos allows building distributed applications and data center infrastructure. The document also covers Mesos concepts such as resources, attributes, roles, constraints and how tools like Marathon interact with Mesos. Finally, it provides examples of Mesos frameworks for technologies like Kafka and Cassandra.
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein Introduction to Apache Kafka
Brokers “as a Service”
Producers & Consumers “as a Service” with Heron
More Use Cases for Kafka
Building and Deploying Application to Apache Mesos
Building and Deploying Application to Apache MesosJoe Stein Building and Deploying Application to Apache Mesos using Marathon, Apache Aurora and Custom Frameworks
Deploying Docker Containers at Scale with Mesos and Marathon
Deploying Docker Containers at Scale with Mesos and MarathonDiscover Pinterest Connor Doyle from Mesosphere.
Deploying Docker Containers at Scale with Mesos and Marathon
The norm these days is to operate apps at web scale. But that’s out of reach for most companies. Deploying Docker containers with Mesos and Marathon makes it easier. See how they help deploy and manage Docker containers at scale and how the Mesos cluster scheduler builds highly-available, fault-tolerant web scale apps.
8a. How To Setup HBase with Docker
8a. How To Setup HBase with DockerFabio Fumarola This document discusses how to setup HBase with Docker in three configurations: single-node standalone, pseudo-distributed single-machine, and fully-distributed cluster. It describes features of HBase like consistent reads/writes, automatic sharding and failover. It provides instructions for installing HBase in a single node using Docker, including building an image and running it with ports exposed. It also covers running HBase in pseudo-distributed mode with the processes running as separate containers and interacting with the HBase shell.
Ceph-Mesos framework
Ceph-Mesos frameworkZhongyue Luo This document discusses the ceph-mesos framework, which implements a Mesos scheduler and executor for Ceph. The goal is to provide RADOS services like the RADOS gateway in a Mesos cluster. The scheduler has callback modules that interact with the Mesos master and provide a REST API and static file server. The executor launches Ceph Docker containers as tasks. The framework is still in early development and future work includes improving support for host hardware selection and networking configurations to optimize Ceph performance. A video demo of ceph-mesos is available online.
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...DataStax Academy This document discusses real-time log analysis using Mesos, Docker, Kafka, Spark, Cassandra and Solr at scale. It provides an overview of the architecture, describing how data from various sources like syslog can be ingested into Kafka via Docker producers. It then discusses consuming from Kafka to write to Cassandra in real-time and running Spark jobs on Cassandra data. The document uses these open source tools together in a reference architecture to enable real-time analytics and search capabilities on streaming data.
Introduction to Mesos
Introduction to Mesoskoboltmarky Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It was first released in 2009 at Berkeley and uses a master-slave architecture with 10,000 lines of C++ code. Mesos allows organizations to combine CPUs, memory, storage and other resources into a shared pool that can be allocated to applications known as frameworks.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Introduction of mesos persistent storage
Introduction of mesos persistent storageZhou Weitao 1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
Hadoop on-mesos
Hadoop on-mesosHenry Cai 蔡明航 The document discusses Hadoop on Mesos, beginning with a short history of distributed computing. It describes how Mesos provides an operating system for clusters that allows applications like Hadoop to run as distributed frameworks. The document outlines challenges in running Hadoop on Mesos and how these were addressed, including using Mesos schedulers and reservations. It also presents a case study of how Airbnb migrated its Hadoop infrastructure from Amazon EMR to run on Mesos, improving availability, performance, and customer satisfaction.
Friends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFSSaumitra Srivastav Nutch is an open source web crawler built on Hadoop that can be used to crawl websites at scale. It integrates directly with Solr to index crawled content. HDFS provides a scalable storage layer that Nutch and Solr can write to and read from directly. This allows building indexes for Solr using Hadoop's MapReduce framework. Morphlines allow defining ETL pipelines to extract, transform, and load content from various sources into Solr running on HDFS.
HBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with ClusterdockMichael Stack This document discusses using clusterdock, an open-source container orchestration framework, to deploy and manage Apache HBase clusters on Docker. It provides an overview of Docker and clusterdock, describes how the HBase topology uses clusterdock to build and start HBase clusters quickly, and demos running an HBase integration test on a clusterdock cluster. It also discusses plans to use clusterdock for running HBase integration tests upstream and improving the release process.
Mesos and containers
Mesos and containersJiang Yan Xu Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
8b. Column Oriented Databases Lab
8b. Column Oriented Databases LabFabio Fumarola The document discusses cloning Twitter using HBase. It describes some key features of Twitter like allowing users to post status updates, follow other users, mention users, and re-tweet posts. It then provides an overview of HBase including its features like consistency, automatic sharding and failover. It discusses how to install HBase in single node, pseudo-distributed and fully distributed modes using Docker. It also demonstrates some common HBase shell commands like creating and listing tables, putting and getting data. Finally, it discusses how to model the user, tweet, follower and following relationships in HBase.
Cassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL MeetupMichael Wynholds This presentation introduces people to Cassandra and Column Family Datastores in general. I will discuss what Cassandra is, how and when it is useful, and how it integrates with Rails. I will also go in to lessons learned during our 3-month project, and the useful patterns that emerged. The discussion will be very technical, but targeted at developers who are not familiar with, or have not done a project with Cassandra.
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...C4Media Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1L2FXLC.
Joe Stein introduces Mesos and managing data services on it, presenting use cases for replacing classic solutions (like cold storage) with new functionality based on these technology. Filmed at qconnewyork.com.
Joe Stein is the CEO of Elodina, a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. He is also the Founder and Principal Consultant of Big Data Open Source Security.
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesMydbops There are several exciting and long-awaited features released from MongoDB 4.0. He will focus on the prime features, the kind of problem it solves, and the best practices for deploying replica sets.
Web scraping with nutch solr part 2
Web scraping with nutch solr part 2Mike Frampton Part 2 of a three part presentation showing how nutch and solr may be used to crawl the web, extract data and prepare it for loading into a data warehouse.
Meetup on Apache Zookeeper
Meetup on Apache ZookeeperAnshul Patel Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.
Develop with linux containers and docker
Develop with linux containers and dockerFabio Fumarola Docker allows building and running applications inside lightweight containers. Some key benefits of Docker include:
- Portability - Dockerized applications are completely portable and can run on any infrastructure from development machines to production servers.
- Consistency - Docker ensures that application dependencies and environments are always the same, regardless of where the application is run.
- Efficiency - Docker containers are lightweight since they don't need virtualization layers like VMs. This allows for higher density and more efficient use of resources.
Large Scale Crawling with Apache Nutch and Friends
Large Scale Crawling with Apache Nutch and FriendsJulien Nioche This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Kafka Connect & Streams - the ecosystem around Kafka
Kafka Connect & Streams - the ecosystem around KafkaGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
More Related Content
What's hot (20)
Deploying Docker Containers at Scale with Mesos and Marathon
Deploying Docker Containers at Scale with Mesos and MarathonDiscover Pinterest Connor Doyle from Mesosphere.
Deploying Docker Containers at Scale with Mesos and Marathon
The norm these days is to operate apps at web scale. But that’s out of reach for most companies. Deploying Docker containers with Mesos and Marathon makes it easier. See how they help deploy and manage Docker containers at scale and how the Mesos cluster scheduler builds highly-available, fault-tolerant web scale apps.
8a. How To Setup HBase with Docker
8a. How To Setup HBase with DockerFabio Fumarola This document discusses how to setup HBase with Docker in three configurations: single-node standalone, pseudo-distributed single-machine, and fully-distributed cluster. It describes features of HBase like consistent reads/writes, automatic sharding and failover. It provides instructions for installing HBase in a single node using Docker, including building an image and running it with ports exposed. It also covers running HBase in pseudo-distributed mode with the processes running as separate containers and interacting with the HBase shell.
Ceph-Mesos framework
Ceph-Mesos frameworkZhongyue Luo This document discusses the ceph-mesos framework, which implements a Mesos scheduler and executor for Ceph. The goal is to provide RADOS services like the RADOS gateway in a Mesos cluster. The scheduler has callback modules that interact with the Mesos master and provide a REST API and static file server. The executor launches Ceph Docker containers as tasks. The framework is still in early development and future work includes improving support for host hardware selection and networking configurations to optimize Ceph performance. A video demo of ceph-mesos is available online.
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...DataStax Academy This document discusses real-time log analysis using Mesos, Docker, Kafka, Spark, Cassandra and Solr at scale. It provides an overview of the architecture, describing how data from various sources like syslog can be ingested into Kafka via Docker producers. It then discusses consuming from Kafka to write to Cassandra in real-time and running Spark jobs on Cassandra data. The document uses these open source tools together in a reference architecture to enable real-time analytics and search capabilities on streaming data.
Introduction to Mesos
Introduction to Mesoskoboltmarky Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It was first released in 2009 at Berkeley and uses a master-slave architecture with 10,000 lines of C++ code. Mesos allows organizations to combine CPUs, memory, storage and other resources into a shared pool that can be allocated to applications known as frameworks.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Introduction of mesos persistent storage
Introduction of mesos persistent storageZhou Weitao 1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
Hadoop on-mesos
Hadoop on-mesosHenry Cai 蔡明航 The document discusses Hadoop on Mesos, beginning with a short history of distributed computing. It describes how Mesos provides an operating system for clusters that allows applications like Hadoop to run as distributed frameworks. The document outlines challenges in running Hadoop on Mesos and how these were addressed, including using Mesos schedulers and reservations. It also presents a case study of how Airbnb migrated its Hadoop infrastructure from Amazon EMR to run on Mesos, improving availability, performance, and customer satisfaction.
Friends of Solr - Nutch & HDFS
Friends of Solr - Nutch & HDFSSaumitra Srivastav Nutch is an open source web crawler built on Hadoop that can be used to crawl websites at scale. It integrates directly with Solr to index crawled content. HDFS provides a scalable storage layer that Nutch and Solr can write to and read from directly. This allows building indexes for Solr using Hadoop's MapReduce framework. Morphlines allow defining ETL pipelines to extract, transform, and load content from various sources into Solr running on HDFS.
HBaseConEast2016: HBase on Docker with Clusterdock
HBaseConEast2016: HBase on Docker with ClusterdockMichael Stack This document discusses using clusterdock, an open-source container orchestration framework, to deploy and manage Apache HBase clusters on Docker. It provides an overview of Docker and clusterdock, describes how the HBase topology uses clusterdock to build and start HBase clusters quickly, and demos running an HBase integration test on a clusterdock cluster. It also discusses plans to use clusterdock for running HBase integration tests upstream and improving the release process.
Mesos and containers
Mesos and containersJiang Yan Xu Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
8b. Column Oriented Databases Lab
8b. Column Oriented Databases LabFabio Fumarola The document discusses cloning Twitter using HBase. It describes some key features of Twitter like allowing users to post status updates, follow other users, mention users, and re-tweet posts. It then provides an overview of HBase including its features like consistency, automatic sharding and failover. It discusses how to install HBase in single node, pseudo-distributed and fully distributed modes using Docker. It also demonstrates some common HBase shell commands like creating and listing tables, putting and getting data. Finally, it discusses how to model the user, tweet, follower and following relationships in HBase.
Cassandra and Rails at LA NoSQL Meetup
Cassandra and Rails at LA NoSQL MeetupMichael Wynholds This presentation introduces people to Cassandra and Column Family Datastores in general. I will discuss what Cassandra is, how and when it is useful, and how it integrates with Rails. I will also go in to lessons learned during our 3-month project, and the useful patterns that emerged. The discussion will be very technical, but targeted at developers who are not familiar with, or have not done a project with Cassandra.
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...C4Media Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1L2FXLC.
Joe Stein introduces Mesos and managing data services on it, presenting use cases for replacing classic solutions (like cold storage) with new functionality based on these technology. Filmed at qconnewyork.com.
Joe Stein is the CEO of Elodina, a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. He is also the Founder and Principal Consultant of Big Data Open Source Security.
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesMydbops There are several exciting and long-awaited features released from MongoDB 4.0. He will focus on the prime features, the kind of problem it solves, and the best practices for deploying replica sets.
Web scraping with nutch solr part 2
Web scraping with nutch solr part 2Mike Frampton Part 2 of a three part presentation showing how nutch and solr may be used to crawl the web, extract data and prepare it for loading into a data warehouse.
Meetup on Apache Zookeeper
Meetup on Apache ZookeeperAnshul Patel Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.
Develop with linux containers and docker
Develop with linux containers and dockerFabio Fumarola Docker allows building and running applications inside lightweight containers. Some key benefits of Docker include:
- Portability - Dockerized applications are completely portable and can run on any infrastructure from development machines to production servers.
- Consistency - Docker ensures that application dependencies and environments are always the same, regardless of where the application is run.
- Efficiency - Docker containers are lightweight since they don't need virtualization layers like VMs. This allows for higher density and more efficient use of resources.
Large Scale Crawling with Apache Nutch and Friends
Large Scale Crawling with Apache Nutch and FriendsJulien Nioche This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.
Similar to Making Apache Kafka Elastic with Apache Mesos (20)
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Kafka Connect & Streams - the ecosystem around Kafka
Kafka Connect & Streams - the ecosystem around KafkaGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Introduction to kafka connector
Introduction to kafka connectorKnoldus Inc. In this session, we will learn about the kafka connector and how to start the Kafka connect cluster using docker-compose.
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
Apache kafka configuration-guide
Apache kafka configuration-guideChetan Khatri Single Node Apache Kafka 0.10 Setup and Configuration on Ubuntu 14.04 documents the steps to setup a single node Apache Kafka cluster on Ubuntu 14.04. The document outlines downloading and extracting Apache Kafka 0.10, configuring Zookeeper and Kafka server properties, creating a Kafka topic, producing and consuming messages from the topic, and describing and deleting topics. Key steps include starting Zookeeper, starting the Kafka server, creating a topic, producing messages to the topic using a console producer, and consuming messages from the topic using a console consumer.
Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka core
Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka coreGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
Kafka Connect & Kafka Streams/KSQL - the ecosystem around KafkaGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Apache Kafka - Scalable Message Processing and more!
Apache Kafka - Scalable Message Processing and more!Guido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Python Kafka Integration: Developers Guide
Python Kafka Integration: Developers GuideInexture Solutions Explore Python's prowess in Kafka integration through our Developer's Guide. step-by-step instructions, code examples, and implementation.
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkTimothy Spann JConWorld: Continuous SQL with Kafka and Flink
In this talk, I will walk through how someone can setup and run continous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas and publishing data.
We will then cover consuming Kafka data, joining Kafka topics and inserting new events into Kafka topics as they arrive. This basic over view will show hands-on techniques, tips and examples of how to do this.
Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html
https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber
Apache Kafka - Strakin Technologies Pvt Ltd
Apache Kafka - Strakin Technologies Pvt LtdStrakin Technologies Pvt Ltd The document provides an overview of Apache Kafka. It discusses how LinkedIn faced the problem of collecting data from various sources in different formats. It explains that Apache Kafka, an open-source stream-processing software developed by LinkedIn, provides a unified platform for handling real-time data feeds through its distributed transaction log architecture. The document then describes Kafka's architecture, including its use of topics, producers, consumers and brokers. It also covers how to install and set up Kafka along with examples of using its Java producer and consumer APIs.
Unlocking the Power of Apache Kafka: How Kafka Listeners Facilitate Real-time...
Unlocking the Power of Apache Kafka: How Kafka Listeners Facilitate Real-time...Denodo Watch full webinar here: https://buff.ly/43PDVsz
In today's fast-paced, data-driven world, organizations need real-time data pipelines and streaming applications to make informed decisions. Apache Kafka, a distributed streaming platform, provides a powerful solution for building such applications and, at the same time, gives the ability to scale without downtime and to work with high volumes of data. At the heart of Apache Kafka lies Kafka Topics, which enable communication between clients and brokers in the Kafka cluster.
Join us for this session with Pooja Dusane, Data Engineer at Denodo where we will explore the critical role that Kafka listeners play in enabling connectivity to Kafka Topics. We'll dive deep into the technical details, discussing the key concepts of Kafka listeners, including their role in enabling real-time communication between consumers and producers. We'll also explore the various configuration options available for Kafka listeners and demonstrate how they can be customized to suit specific use cases.
Attend and Learn:
- The critical role that Kafka listeners play in enabling connectivity in Apache Kafka.
- Key concepts of Kafka listeners and how they enable real-time communication between clients and brokers.
- Configuration options available for Kafka listeners and how they can be customized to suit specific use cases.
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6Kai Wähner New Features in Confluent Platform 6.0 / Apache Kafka 2.6, including REST Proxy and API, Tiered Storage for AWS S3 and GCP GCS, Cluster Linking (On-Premise, Edge, Hybrid, Multi-Cloud), Self-Balancing Clusters), ksqlDB.
Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka
Kafka Connect & Kafka Streams/KSQL - the ecosystem around KafkaGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Stories from running Kafka on K8S.pdf
Stories from running Kafka on K8S.pdfAvinashUpadhyaya3 Slides from the talk on lessons on running Kafka on Kubernetes by Pavan Keshavamurthy and Avinash Upadhyaya of Platformatory at the Apache Kafka Mumbai July 2023 meetup.
Look at various tooling around running Apache Kafka on Kubernetes and cover best practices for running a distributed system such as Kafka on Kubernetes.
A day in the life of a log message
A day in the life of a log messageJosef Karásek Slides from a talk at Open Source Summit Europe 2018
https://events.linuxfoundation.org/events/open-source-summit-europe-2018/program/schedule/
Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...
Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent In this session we share our experience of building a real-time data pipelines at Tencent PCG - one that handles 20 trillion daily messages with 700 clusters and 100Gb/s bursting traffic from a single app. We discuss our roadmap of enhancing Kafka to break its limits in terms of scalability, robustness and cost of operation.
We first built a proxy layer that aggregates physical clusters in a way agnostic to the clients. While this architecture solves many operational problems, it requires significant development to stay future-proof. With retrospection with our customer and careful study of the ongoing work from the community, we then designed a region federation solution in the broker layer, which allows us to deploy clusters at a much larger scale than previously possible, while at the same time providing better failure recovery and operability. We discuss how we make this development compatible with KIP-500 and KIP-405, and the two KIP (693, 694) that we submitted for discussion.
SMACK Stack 1.1
SMACK Stack 1.1Joe Stein The document discusses the SMACK stack 1.1, which includes tools for streaming, Mesos, analytics, Cassandra, and Kafka. It describes how SMACK stack 1.1 adds capabilities for dynamic compute, microservices, orchestration, and microsegmentation. It also provides examples of running Storm on Mesos and using Apache Kafka for decoupling data pipelines.
Apache Kafka - A modern Stream Processing Platform
Apache Kafka - A modern Stream Processing PlatformGuido Schmutz After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
Partner Development Guide for Kafka Connect
Partner Development Guide for Kafka Connectconfluent This guide is intended to provide useful background to developers implementing Kafka Connect sources and sinks for their data stores. Visit www.confluent.io for more information.
Ad
More from Joe Stein (13)
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.
Developing with the Go client for Apache Kafka
Developing with the Go client for Apache KafkaJoe Stein This document summarizes Joe Stein's go_kafka_client GitHub repository, which provides a Kafka client library written in Go. It describes the motivation for creating a new Go Kafka client, how to use producers and consumers with the library, and distributed processing patterns like mirroring and reactive streams. The client aims to be lightweight with few dependencies while supporting real-world use cases for Kafka producers and high-level consumers.
Current and Future of Apache Kafka
Current and Future of Apache KafkaJoe Stein Apache Kafka is a distributed streaming platform. It provides a high-throughput distributed messaging system that can handle trillions of events daily. Many large companies use Kafka for application logging, metrics collection, and powering real-time analytics. The current version is 0.8.2 and upcoming versions will include a new consumer, security features, and support for transactions.
Introduction Apache Kafka
Introduction Apache KafkaJoe Stein Apache Kafka is a high-throughput distributed messaging system that can be used for building real-time data pipelines and streaming apps. It provides a publish-subscribe messaging model and is designed as a distributed commit log. Kafka allows for both push and pull models where producers push data and consumers pull data from topics which are divided into partitions to allow for parallelism.
Introduction to Apache Mesos
Introduction to Apache MesosJoe Stein Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.
Developing Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache KafkaJoe Stein Developing Realtime Data Pipelines With Apache Kafka. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein Apache Kafka is a distributed streaming platform that allows for building real-time data pipelines and streaming apps. It provides a publish-subscribe messaging system with persistence that allows for building real-time streaming applications. Producers publish data to topics which are divided into partitions. Consumers subscribe to topics and process the streaming data. The system handles scaling and data distribution to allow for high throughput and fault tolerance.
Real-time streaming and data pipelines with Apache Kafka
Real-time streaming and data pipelines with Apache KafkaJoe Stein Get up and running quickly with Apache Kafka http://kafka.apache.org/
* Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
* Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
* Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
* Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Apache Cassandra 2.0
Apache Cassandra 2.0Joe Stein Detail behind the Apache Cassandra 2.0 release and what is new in it including Lightweight Transactions (compare and swap) Eager retries, Improved compaction, Triggers (experimental) and more!
• CQL cursors
Storing Time Series Metrics With Cassandra and Composite Columns
Storing Time Series Metrics With Cassandra and Composite ColumnsJoe Stein This document discusses storing and aggregating time series metrics in Cassandra using counters and composite columns. It provides an example schema using multiple column families partitioned by time period (day, hour, minute, second). Data is inserted by incrementing counters for composite column names representing the aggregated values. Retrieval involves multiget queries on ranges of composite column names to retrieve aggregated counts for a time period.
Apache Kafka
Apache KafkaJoe Stein Apache Kafka is a distributed publish-subscribe messaging system that was originally created by LinkedIn and contributed to the Apache Software Foundation. It is written in Scala and provides a multi-language API to publish and consume streams of records. Kafka is useful for both log aggregation and real-time messaging due to its high performance, scalability, and ability to serve as both a distributed messaging system and log storage system with a single unified architecture. To use Kafka, one runs Zookeeper for coordination, Kafka brokers to form a cluster, and then publishes and consumes messages with a producer API and consumer API.
Hadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonJoe Stein This document provides an overview and tutorial on streaming jobs in Hadoop, which allow processing of data using non-Java programs like Python scripts. It includes sample code and datasets to demonstrate joining and counting data from multiple files using mappers and reducers. Tips are provided on optimizing streaming jobs, such as padding fields for sorting, handling errors, and running jobs on Hadoop versus standalone.
jstein.cassandra.nyc.2011
jstein.cassandra.nyc.2011Joe Stein Cassandra is used as the central data store for the distributed systems architecture of Medialets, a mobile rich media ad platform. Medialets collects terabytes of new user event data daily from hundreds of millions of mobile devices. Cassandra allows Medialets to store this time series data in aggregate form across multiple column families based on time periods (day, hour, minute, second). The composite column names allow flexible querying of metrics grouped by attributes like app, device, events. Skeletor, a Scala wrapper for Hector, is used to easily insert aggregated data into Cassandra.
Ad
Making Apache Kafka Elastic with Apache Mesos
- 2. Joe Stein CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. Joe Stein is also the Founder and Principal Consultant of Big Data Open Source Security http://stealth.ly and has been working for the last couple of years on implementing and assisting organizations with their Kafka, Mesos, Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems. Prior to this, Joe Stein was responsible for building out a platform that ingested and processed the analytics for 6 billion unique mobile devices.
- 3. ◉ Quick intro to Mesos ◉ Quick intro to Kafka ◉ Making Kafka elastic with Mesos Overview
- 4. Quick intro to Mesos
- 10. Data Center Operating System
- 12. Mesos
- 13. Quick intro to Kafka
- 15. Kafka decouples data pipelines
- 16. Making Kafka elastic with Mesos
- 17. Goals we set out with ● smart broker.id assignment. ● preservation of broker placement (through constraints and/or new features). ● ability to-do configuration changes. ● rolling restarts (for things like configuration changes). ● scaling the cluster up and down with automatic, programmatic and manual options. ● smart partition assignment via constraints visa vi roles, resources and attributes.
- 19. Scheduler ● Provides the operational automation for a Kafka Cluster. ● Manages the changes to the broker's configuration. ● Exposes a REST API for the CLI to use or any other client. ● Runs on Marathon for high availability. Executor ● The executor interacts with the kafka broker as an intermediary to the scheduler Scheduler & Executor
- 20. CLI & REST API ● scheduler - starts the scheduler. ● add - adds one more more brokers to the cluster. ● update - changes resources, constraints or broker properties one or more brokers. ● remove - take a broker out of the cluster. ● start - starts a broker up. ● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh help stop) ● rebalance - allows you to rebalance a cluster either by selecting the brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic. ● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
- 21. Launch 20 brokers in seconds ./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1 ./kafka-mesos.sh start 1000..1019
- 22. Kafka is available on DCOS https://mesosphere.com/product/ dcos install kafka dcos kafka help Mesosphere DCOS