How to size up an Apache Cassandra cluster (Training)

Editor's Notes

• #3: Discuss the main features and highlights of the Cassandra database. The features that are important to you will influence how you design, size, and configure your Cassandra cluster.For Rack and Datacenter awareness, mention that includes deploying Cassandra in the cloud, such as Amazon EC2, Rackspace, Google Computing Cloud, etc.
• #4: This should be self-explanatory for me as I go through this slide
• #5: Since Cassandra is linearly scalable, your cluster can be scaled as large as needed. The focus for this presentation is more on the minimum number of nodes you’d want / need, alongwith the replication setting.Based on data from a University of Toronto studyhttp://vldb.org/pvldb/vol5/p1724_tilmannrabl_vldb2012.pdf
• #6: Replication needed to achieve high availability when designing for failure.Can’t have replication factor larger than number of nodes in the cluster.
• #7: Wait, there are people that didn’t understand what a rack or datacenter is? Well then, let’s backtrack a little and define some of these terms. Starting with the smallest unit, we have the node…By process, I mean a Java virtual machine. Cassandra is written in Java, and its binary code must run in a virtual machine.Nodes can also be represented as a cloud or virtual server instance.
• #9: Datacenters can be geographically separated, but also logically separated as well. Additional use cases for data centers include disaster recovery and workload seperation.
• #11: With single node databases when you write data, you can expect to read back the same data. It’s not so easy with distributed systems though. The node that a client writes data to may not be the same node another client is trying to read the data from. In that case, a distributed system must implement a consistency model to determine when written data is saved to the relevant nodes. May want to mention BASE, make sure to clarify that eventual consistency usually occurs within milliseconds (thanks Netflix!)
• #12: Tunable consistency is a key feature of Cassandra, and the type of consistency you want to use may affect your cluster design.
• #14: Be sure to explain what happens if data returned from each node does not match.
• #15: Be sure to explain what happens if data returned from each node does not match.
• #16: Cross-datacenter latency vs. local consistency / consistency across datacenters
• #17: Important to understand how the storage engine works, since that directly impacts data size. No reads before a write.Writes go to commit log for durability, and memtablePeriodically memtable data is flushed to disk into a SSTable, or sorted strings table. This will destroy the memtable so that the memory can be reused. Relevant commitlog entries also marked as cleared.
• #18: Important to understand how the storage engine works, since that directly impacts data size.
• #19: Important to understand how the storage engine works, since that directly impacts data size.
• #21: Now that you have a basic understanding of how Cassandra works and the possible benefits to select and use, we can talk about the primary factors for sizing your database.Although not as key, I will also discuss some considerations for the replication factor as well
• #23: If RF = 1, we are not making use of Cassandra’s advantages of being available. One node = single point of failure.If just using Cassandra for durability, may use RF=2 just to ensure we have two copies of your data on separate nodes.Next slide will talk a bit more about RF < 3.
• #25: PerformanceFor high availability use cases, there are clusters configured to use a replication factor as high as 5. Not very common.
• #27: Each Cassandra node has a certain data capacity, and out of that capacity it can only be used for data to a certain limit. These are some of the factors.
• #28: Of course your data set needs to be accounted for. In addition there is overhead for writing the data in Cassandra, as well as certain structures used for read optimizations (Partition index, summary, Bloom filter)
• #29: If using a RF > 1, must account for those additional copies. At RF=3, if your data set is 5TB it means C* will be saving 15TB.
• #30: One consequence of log structured storage is that data that’s no longer needed will exist until a compaction will clean it up. That means additional space remains used until a compaction occurs.
• #31: Free disk space must be reserved for compaction so that data can be merged into a new file. See above.
• #32: Backing up your data is very easy with Cassandra. Since the data files are immutable, a snapshot can be taken which creates a hard link or copy of the relevant SSTables. Hard links in particular are pretty much zero cost, since it takes negligible disk space and time to create the hard link.However just be careful. If you are creating snapshots, or configured Cassandra to automatically create snapshots, that’s also going to eat up your disk space unless user does housekeeping.
• #33: DataStax recommended disk capacity, size your cluster so that your data fits.Why can’t we just add more disks? Limited by performance of each node handling that much data (contention from reads/writes, flushing, compaction, limit on JVM heap memory allocation).
• #35: For cluster sizing, you want to have enough nodes so that read and write performance meet any SLAs, or are otherwise acceptable to users.
• #36: Failure conditions must also be taken into account. If a node fails, the workload from that node must be absorbed by the other nodes in the cluster. When recovering the node, this can result in further impact to performance.Don’t size cluster to fully utilize each node, leave room so that cluster can still perform acceptably during failure.Rule of thumb: Some Cassandra MVPs recommend having no less than 6 nodes in your cluster. With less than 6, if you lose one node, you lose a good chunk of your cluster’s throughput capability (at least 20%).