Ibm spectrum scale fundamentals workshop for americas part 2 IBM Spectrum Scale Quorum and I/O Concepts
- 1. Spectrum Scale 4.1 System Administration Quorums, Quorum Loss and recovery © Copyright IBM Corporation 2015
- 2. Unit objectives After completing this unit, you should be able to: • Review Quorums • Review some Quorum best practices • Describe recovery from quorum and cluster configuration node failures © Copyright IBM Corporation 2015
- 3. Spectrum Scale Quorum Nodes Why do we need Quorum nodes? • Quorum Nodes prevent multiple nodes from assuming the role of the file system manager • A majority of the quorum nodes MUST remain active in order for the cluster to sustain normal file system activity. • This prevents the cluster from becoming partitioned off where two different nodes think they are each in control and corrupt the file system © Copyright IBM Corporation 2015
- 4. Best Practices: Quorum • Quorum – set of 3-7 nodes use to determine the state of the cluster • Small subset of nodes designated as quorum nodes • If there are only two nodes tiebreaker disks can be used to determine quorum • Recommendations – Use an odd number of nodes – Choose your most reliable nodes – Choose nodes in separate: cabinets, power systems – Use tiebreaker disks only on small clusters – For small clusters, a simple VM can serve as a quorum node – Quorum nodes should not be placed on extremely busy servers © Copyright IBM Corporation 2015
- 5. RECOVERY SCENARIOS © Copyright IBM Corporation 2015
- 6. How to recover: Node Quorum How to recover from loss of majority for quorum nodes • Scenario: You have 3 quorum nodes and 2 quorum nodes die. – Nodes go into Arbitrating state – Add 2 new quorum nodes – mmchnode -–quorum –N node4,node5 – Now you have 5 quorum nodes and 3 are running – Running nodes return to Active state – Delete missing quorum nodes – mmdelnode –N badnode1,badnode2 – or – mmchnode –-nonquorum –N badnode1,badnode2 – Now you have 3 quorum nodes and 3 are running – Running nodes remain in Active state • You may have to clean up original quorum nodes once they recover. © Copyright IBM Corporation 2015
- 7. Recover from loss of primary configuration server How to recover from loss of a primary configuration server • Scenario: Primary cluster configuration server fails – Secondary steps in – Reassign primary – mmchcluster –p newprimarynode – To sync up the new primary – mmchcluster –p LATEST • Use the same process for failure of secondary cluster configuration server. © Copyright IBM Corporation 2015
- 8. How to recover: Loss of both configuration servers How to recover from permanent loss of the primary configuration and secondary servers • Scenario: Primary and secondary cluster configuration servers fail – Cluster status • Data is online • mmgetstate -a no longer works – Reassign both node configuration servers at the same time – mmchcluster -p nodenewprimary –s nodenewsecondary – Old servers still think they are in charge until – mmchcluster –p Latest © Copyright IBM Corporation 2015
- 9. I/O Concepts: Node config parameters (1 of 2) • Spectrum Scale configuration parameters • File system configuration • Storage configuration • Interconnect tuning • Application design © Copyright IBM Corporation 2015
- 10. I/O Concepts: Node config parameters (2 of 2) • Spectrum Scale configuration parameters • pagepool • maxFilesToCache • maxStatCache • prefetchThreads • socketRcvBufferSize • socketSndBufferSize • seqDiscardThreshhold • Worker1threads • generalWorkerThreads <<< Verify Role • tokenMem <<< Verify Spelling © Copyright IBM Corporation 2015
- 11. Performance tuning: File system parameters • File system configuration • -B Blocksize • -n numnodes • -l LogFileSize • Storage Consideration – RAID level – RAID Strip Size (Segment Size) – Storage Type: Flash, 10k RPM, 7,200 RPM © Copyright IBM Corporation 2015
- 12. Performance tuning: Storage • Storage configuration – IO balance • Even number of LUNS per path • Even number of LUNS per NSD server – RAID level • Right level for IO pattern – Cache configuration • Read ahead not recommended for –j scatter © Copyright IBM Corporation 2015
- 13. Performance tuning: Network configuration • Interconnect selection and tuning – TCP/IP tuning – InfiniBand configuration • Test Network performance with nsdperf – In /usr/lpp/mmfs/samples/net/* – README is documentation – Some tools on Spectrum Scale Wiki © Copyright IBM Corporation 2015
- 14. Performance tuning: Application design • Make sure your applications are cluster aware • Avoid and/or optimize for heavy operations – Multiple nodes write locking a common entire file • Use byte range locking • Separate files – Reduce use of IO heavy operations • fsync() – Some local node optimizations not applicable on cluster • mmap © Copyright IBM Corporation 2015
- 15. Direct I/O • What is it? – Bypass Spectrum Scale cache and transfer data directly from disk into the user space buffer versus placing data in kernel memory. • Why do I want it? – Application with low file system cache hit rates – Application with very large I/O – When the application knows how to optimize IO – Example: Oracle Database environment. • How do I enable it? – mmchattr –D yes <file_name>, or – Specify O_DIRECT file access mode on the open() of the file. © Copyright IBM Corporation 2015
- 16. RECOVERY SCENARIOS © Copyright IBM Corporation 2015
- 17. How to recover: Node Quorum How to recover from loss of majority for quorum nodes • Scenario: You have 3 quorum nodes and 2 quorum nodes die. – Nodes go into Arbitrating state – Add 2 new quorum nodes – mmchnode -–quorum –N node4,node5 – Now you have 5 quorum nodes and 3 are running – Running nodes return to Active state – Delete missing quorum nodes – mmdelnode –N badnode1,badnode2 – or – mmchnode –-nonquorum –N badnode1,badnode2 – Now you have 3 quorum nodes and 3 are running – Running nodes remain in Active state • You may have to clean up original quorum nodes once they recover. © Copyright IBM Corporation 2015
- 18. Recover from loss of primary configuration server How to recover from loss of a primary configuration server • Scenario: Primary cluster configuration server fails – Secondary steps in – Reassign primary – mmchcluster –p newprimarynode – To sync up the new primary – mmchcluster –p LATEST • Use the same process for failure of secondary cluster configuration server. © Copyright IBM Corporation 2015
- 19. How to recovery: Lose both configuration servers How to recover from permanent loss of the primary configuration and secondary servers • Scenario: Primary and secondary cluster configuration servers fail – Cluster status • Data is online • mmgetstate -a no longer works – Reassign both node configuration servers at the same time – mmchcluster -p nodenewprimary –s nodenewsecondary – Old servers still think they are in charge until – mmchcluster –p Latest © Copyright IBM Corporation 2015
- 20. Unit summary Having completed this unit, you should be able to: • Review some Spectrum Scale best practices • Discuss approaches to Spectrum Scale performance tuning • Describe recovery and recovery from quorum and cluster configuration node failures © Copyright IBM Corporation 2015