Hecuba2: Cassandra Operations Made Easy (Radovan Zvoncek, Spotify) | C* Summit 2016
Download as PPTX, PDF3 likes510 views
The document discusses Hecuba2, a tool used by Spotify to manage Apache Cassandra operations, highlighting two significant case studies demonstrating its functionalities. It outlines Spotify's infrastructure, including challenges faced during cluster expansions and node replacements, simplified by Hecuba2's CLI commands. The presentation emphasizes the ease of use and robustness of Hecuba2 over its year of operation, along with its capabilities to manage cluster topologies.
Hecuba2: Cassandra Operations Made Easy (Radovan Zvoncek, Spotify) | C* Summit 2016
- 1. Hecuba2 Cassandra Operations Made Easy Radovan Zvoncek [email protected]
- 2. Agenda 1 Two peace stories 2 Cassandra infrastructure at Spotify 3 What exactly is Hecuba2? 4 Wrap up 2© DataStax, All Rights Reserved.
- 3. About Radovan Likes pancakes © DataStax, All Rights Reserved. 3
- 4. About Radovan Likes pancakes Now knows where to get them © DataStax, All Rights Reserved. 4
- 5. About Radovan Likes pancakes Now knows where to get them Works at Spotify © DataStax, All Rights Reserved. 5
- 6. Spotify Music streaming service ● ~ 100 million active users ● ~ 2 billion playlists © DataStax, All Rights Reserved. 6
- 7. Spotify Music streaming service ● ~ 100 million active users ● ~ 2 billion playlists Happy Apache Cassandra user ● ~ 100 Cassandra clusters ● ~ 1000 nodes altogether © DataStax, All Rights Reserved. 7
- 8. The Playlist Cluster The largest cluster we have ● 2 x 45 nodes ● ~ 1TB of data on each node © DataStax, All Rights Reserved. 8
- 9. The Playlist Cluster The largest cluster we have ● 2 x 45 nodes ● ~ 1TB of data on each node © DataStax, All Rights Reserved. 9
- 10. The Playlist Cluster The largest cluster we have ● 2 x 45 nodes ● ~ 1TB of data on each node Had to expand by 50% © DataStax, All Rights Reserved. 10
- 11. The previous large cluster expansion went so wrong… Peace Story #1: The Playlist Expansion © DataStax, All Rights Reserved. 11
- 12. The previous large cluster expansion went so wrong… How exactly to do the expansion now? Peace Story #1: The Playlist Expansion © DataStax, All Rights Reserved. 12
- 13. The previous large cluster expansion went so wrong… How exactly to do the expansion now? What tokens should the new nodes have? Peace Story #1: The Playlist Expansion © DataStax, All Rights Reserved. 13
- 14. Peace Story #1: The Playlist Expansion The previous large cluster expansion went so wrong… How exactly to do the expansion now? What tokens should the new nodes have? How to bootstrap the nodes? © DataStax, All Rights Reserved. 14
- 15. Peace Story #1: The Playlist Expansion Turns out, we needed just one command: © DataStax, All Rights Reserved. 15 hecuba2-cli expand-cluster dc1-playlistcassandra-a{31..45}.foo.net
- 16. Peace Story #1: The Playlist Expansion Turns out, we needed just one command: And one peer review © DataStax, All Rights Reserved. 16 hecuba2-cli expand-cluster dc1-playlistcassandra-a{31..45}.foo.net
- 17. Peace Story #1: The Playlist Expansion Turns out, we needed just one command: And one peer review And then a week of waiting © DataStax, All Rights Reserved. 17 hecuba2-cli expand-cluster dc1-playlistcassandra-a{31..45}.foo.net
- 18. Peace Story #1: The Playlist Expansion 18
- 19. Peace Story #2: The Slush Incident © DataStax, All Rights Reserved. 19
- 20. Peace Story #2: The Slush Incident © DataStax, All Rights Reserved. 20
- 21. Peace Story #2: The Slush Incident Got paged with 2 out of 3 nodes being down © DataStax, All Rights Reserved. 21 Datacenter: dc1 ========== Address Rack Status Load Owns Token dc1-slush-1.foo.net rac1 Up Normal 797.27 GB 33.33% 0 dc1-slush-2.foo.net rac1 Down Normal 798.58 GB 33.33% 56... dc1-slush-3.foo.net rac1 Down Normal 797.58 GB 33.33% 11...
- 22. Peace Story #2: The Slush Incident Turns out, we needed just two commands © DataStax, All Rights Reserved. 22 hecuba2-cli replace-nodes --old-host dc1-slush-2.foo.net --new-host dc1-slush-4.foo.net hecuba2-cli replace-nodes --old-host dc1-slush-3.foo.net --new-host dc1-slush-5.foo.net
- 23. Peace Story #2: The Slush Incident What gave us two peer reviews, such as: © DataStax, All Rights Reserved. 23
- 24. Peace Story #2: The Slush Incident After a while, we ended up with © DataStax, All Rights Reserved. 24 Datacenter: dc1 ========== Address Rack Status Load Owns Token dc1-slush-1.foo.net rac1 Up Normal 797.27 GB 33.33% 0 dc1-slush-4.foo.net rac1 Up Normal 798.58 GB 33.33% 56... dc1-slush-5.foo.net rac1 Up Normal 797.58 GB 33.33% 11...
- 25. The Peace Stories Very pleasant experience operating Cassandra @ Spotify All because our infrastructure © DataStax, All Rights Reserved. 25
- 26. Agenda 1 Two peace stories 2 Cassandra infrastructure at Spotify 3 What exactly is Hecuba2? 4 Wrap up 26© DataStax, All Rights Reserved.
- 27. Cassandra Infrastructure @ Spotify Let’s just create a Cassandra cluster like a Spotifier © DataStax, All Rights Reserved. 27
- 28. Creating C* Cluster Like A Spotifier Step 1: Get some machines © DataStax, All Rights Reserved. 28
- 29. Creating C* Cluster Like A Spotifier Step 1: Get some machines © DataStax, All Rights Reserved. 29
- 30. Creating C* Cluster Like A Spotifier Step 1: Get some machines More info about System-Z Modelling Microservices at Spotify by Petter Måhlén https://youtu.be/7XDA044tl8k © DataStax, All Rights Reserved. 30
- 31. Creating C* Cluster Like A Spotifier Step 1: Get some machines Step 2: Install the operating system © DataStax, All Rights Reserved. 31
- 32. Creating C* Cluster Like A Spotifier Step 1: Get some machines Step 2: Install the operating system © DataStax, All Rights Reserved. 32
- 33. Creating C* Cluster Like A Spotifier Step 1: Get some machines Step 2: Setup the cluster © DataStax, All Rights Reserved. 33
- 34. Creating C* Cluster Like A Spotifier Step 1: Get some machines Step 2: Setup the cluster © DataStax, All Rights Reserved. 34 hecuba2-cli create-cluster --cluster-name "My new cluster" --owner mySquad dc1-userdatacass- {1..3}.foo.net
- 35. Creating C* Cluster Like A Spotifier © DataStax, All Rights Reserved. Step 1: Get some machines Step 2: Setup the cluster Step 3: Check PR 35
- 36. Creating C* Cluster Like A Spotifier © DataStax, All Rights Reserved. Step 1: Get some machines Step 2: Setup the cluster Step 3: Check PR Step 4: Wait 36
- 37. Creating C* Cluster Like A Spotifier © DataStax, All Rights Reserved. Step 1: Get some machines Step 2: Setup the cluster Step 3: Check PR Step 4: Wait This will get picked up by our conf. management system and ● Install software ● Configure Cassandra 37
- 38. Configuring Cassandra © DataStax, All Rights Reserved. 38
- 39. Configuring Cassandra Is (almost) all about putting things into config files © DataStax, All Rights Reserved. 39
- 40. Configuring Cassandra Is (almost) all about putting things into config files Mostly tokens and seeds ● Tokens are clunky large strings ● V-nodes would help, but... © DataStax, All Rights Reserved. 40
- 41. Configuring Cassandra Is (almost) all about putting things into config files Mostly tokens and seeds ● Tokens are clunky large strings ● V-nodes would help, but... Also node bootstrap ● All nodes joining at once is not desired © DataStax, All Rights Reserved. 41
- 42. Configuring Cassandra Is (almost) all about putting things into config files Mostly tokens and seeds ● Tokens are clunky large strings ● V-nodes would help, but... Also node bootstrap ● All nodes joining at once is not desired Both are handled by Hecuba2 © DataStax, All Rights Reserved. 42
- 43. Agenda 1 Two peace stories 2 Cassandra infrastructure at Spotify 3 What exactly is Hecuba2? 4 Wrap up 43© DataStax, All Rights Reserved.
- 44. What Exactly is Hecuba2 © DataStax, All Rights Reserved. 44 Hecuba YAML File Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 SeedPro vider Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cassandra NodeHecuba2 Client Library C* Process
- 45. Hecuba YAML File Represents the truth about the cluster © DataStax, All Rights Reserved. 45
- 46. Hecuba YAML File dc1-mytestcass-1.foo.net: cluster_name: mytestcass dc: dc1 seed: true token: 0 dc1-mytestcass-2.foo.net: cluster_name: mytestcass dc: dc1 seed: false token: 56713727820156410577229101238628035242 © DataStax, All Rights Reserved. 46
- 47. Hecuba2 Client Library Manipulates the Hecuba YAML file © DataStax, All Rights Reserved. 47
- 48. Hecuba2 Client Library Manipulates the Hecuba YAML file Does this in a smart way © DataStax, All Rights Reserved. 48
- 49. Hecuba2 Client Library Manipulates the Hecuba YAML file Does this in a smart way Doubling the cluster © DataStax, All Rights Reserved. 49 Double the size New node Existing node stayed in place
- 50. Manipulates the Hecuba YAML file Does this in a smart way Doubling the cluster Expand by 50% Hecuba2 Client Library © DataStax, All Rights Reserved. 50 Add 2 nodes New node Existing node stayed in place Existing node moved
- 51. Hecuba2 Server-Side Components © DataStax, All Rights Reserved. 51 Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cassandra Node C* Process
- 52. Hecuba2 Server-Side Components Again, three things: ● hecuba2-agent ○ State machine managing the C* process © DataStax, All Rights Reserved. 52
- 53. hecuba2-agent © DataStax, All Rights Reserved. 53
- 54. hecuba2-agent © DataStax, All Rights Reserved. 54
- 55. Hecuba2 Server-Side Components Again, three things: ● hecuba2-agent ○ State machine managing the C* process ● hecuba2-jmxproxy ○ nodetool with JSON output © DataStax, All Rights Reserved. 55
- 56. Hecuba2 Server-Side Components Again, three things: ● hecuba2-agent ○ State machine managing the C* process ● hecuba2-jmxproxy ○ nodetool with JSON output ● hecuba2-seedprovider ○ Picks seeds from Hecuba YAML © DataStax, All Rights Reserved. 56
- 57. How Does It All Work Together © DataStax, All Rights Reserved. 57
- 58. How Does It All Work Together © DataStax, All Rights Reserved. 58 Hecuba2 Client Library
- 59. How Does It All Work Together © DataStax, All Rights Reserved. 59 Hecuba YAML File Hecuba2 Client Library Creates Hecuba2 Not in scope
- 60. How Does It All Work Together © DataStax, All Rights Reserved. 60 Hecuba YAML File Hecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates
- 61. How Does It All Work Together © DataStax, All Rights Reserved. 61 Hecuba YAML File Hecuba YAML File Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution C* Process Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider
- 62. How Does It All Work Together © DataStax, All Rights Reserved. 62 Hecuba YAML File Hecuba YAML File Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet C* Process Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider
- 63. How Does It All Work Together © DataStax, All Rights Reserved. 63 Hecuba YAML File Hecuba YAML File Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cron executes C* Process
- 64. How Does It All Work Together © DataStax, All Rights Reserved. 64 Hecuba YAML File Hecuba YAML File Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cron executes calls C* Process
- 65. How Does It All Work Together © DataStax, All Rights Reserved. 65 Hecuba YAML File Hecuba YAML File Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cron executes calls manages C* Process
- 66. How Does It All Work Together © DataStax, All Rights Reserved. 66 Hecuba YAML File Hecuba2 Agent manages Cron executes Hecuba2 jmx- proxy Hecuba2 SeedPro vider Hecuba YAML File calls calls Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet C* Process
- 67. How Does It All Work Together © DataStax, All Rights Reserved. 67 Hecuba YAML File Hecuba2 Agent manages Cron executes Hecuba2 jmx- proxy Hecuba2 SeedPro vider Hecuba YAML File calls calls Hecuba2 Agent Hecuba2 jmx- proxy Hecuba2 Seed Provider Cassandra NodeHecuba2 Client Library Text Editor Hecuba2 Not in scope Alternative Creates Distribution Manual Puppet C* Process
- 68. To Recap Hecuba2 manages cluster topologies ● create-cluster ● expand-cluster ● replace-node Many things are out of scope Missing features: ● parallelism @ hecuba2-agent ● cluster shrinking © DataStax, All Rights Reserved. 68
- 69. Agenda 1 Two peace stories 2 Cassandra infrastructure at Spotify 3 What exactly is Hecuba2? 4 Wrap up 69© DataStax, All Rights Reserved.
- 70. Our Experience So Far It’s been in use for a year It hasn’t let us down yet But it has surprised us by being more robust than we thought State machine testable, visualisable, and easily extensible Peer review for changes © DataStax, All Rights Reserved. 70
- 71. FAQ Why is it called Hecuba2? Does it support v-nodes? Does it support Cassandra version X? Can I use it on Y? Is it FOSS? © DataStax, All Rights Reserved. 71
- 72. Actual Q © DataStax, All Rights Reserved. 72
- 73. Thank You! [email protected] [email protected] Eventually https://github.com/spotify/hecuba2 © DataStax, All Rights Reserved. 73