MongoDB World 2019: Finding the Right MongoDB Atlas Cluster Size: Does This Instance Make My App Look Fast?
0 likes1,129 views
The document outlines a three-step process for right-sizing Atlas clusters to optimize application performance while minimizing costs. It emphasizes the importance of verifying database configurations, evaluating remaining capacity, and forecasting future needs based on changes in workload and data. Key metrics for analysis include CPU utilization, IOPS, RAM, query targeting, and cache activity to ensure optimal resource allocation.
![session.start_transaction()
order = { line_items : [ { item : 5, quantity: 6 }, …
] }
db.orders.insertOne( order, session=session );
for x in order.line_items:
db.inventory.update( { _id : x.item } ,
{ $inc : { number : -1 *
x.qty } },
session=session )
session.commit_transaction()
Code with title](https://image.slidesharecdn.com/gramercyday1200pmsizingatlasclustersv13-190628132815/85/MongoDB-World-2019-Finding-the-Right-MongoDB-Atlas-Cluster-Size-Does-This-Instance-Make-My-App-Look-Fast-103-320.jpg)
![order = { line_items : [ { item : 5,
quantity : 6 },
… ] }
db.orders.insertOne( order );
for x in order.line_items:
db.inventory.update( { _id : x.item },
{ $inc : { number :
-1 * x.qty } })
Code comparison with title
session.start_transaction()
order = { line_items : [ { item : 5,
quantity: 6 }, … ] }
db.orders.insertOne( order, session=session
);
for x in order.line_items:
db.inventory.update( { _id : x.item } ,
{ $inc : { number : -1 * x.qty } },
session=session )
session.commit_transaction()](https://image.slidesharecdn.com/gramercyday1200pmsizingatlasclustersv13-190628132815/85/MongoDB-World-2019-Finding-the-Right-MongoDB-Atlas-Cluster-Size-Does-This-Instance-Make-My-App-Look-Fast-104-320.jpg)
![session.start_transaction()
order = { line_items : [ { item : 5, quantity: 6 }, …
] }
db.orders.insertOne( order, session=session );
for x in order.line_items:
db.inventory.update( { _id : x.item } ,
{ $inc : { number : -1 *
x.qty } },
session=session )
session.commit_transaction()](https://image.slidesharecdn.com/gramercyday1200pmsizingatlasclustersv13-190628132815/85/MongoDB-World-2019-Finding-the-Right-MongoDB-Atlas-Cluster-Size-Does-This-Instance-Make-My-App-Look-Fast-105-320.jpg)
MongoDB World 2019: Finding the Right MongoDB Atlas Cluster Size: Does This Instance Make My App Look Fast?
- 1. Finding the Right Atlas Cluster Size: Does this Instance Make my App Look Fast? Jay Runkel, Master Solutions Architect Dawoud Ibrahim, Solutions Architect jayrunkel
- 2. Master Solutions Architect Senior Solutions Architect Dawoud IbrahimJay Runkel
- 3. Airplane pilot flys you from A to B safely
- 4. He uses a lot of information:
- 5. With Atlas, your must deploy a cluster that: 1. Meets your customers performance expectations 2. Achieves this with minimal cost
- 6. Critical responsibility: cluster right sizing time application workload Cluster capacity Self Managed Deployment
- 7. Critical responsibility: cluster right sizing time application workload Cluster capacity Self Managed Deployment Excess Capacity
- 8. Right sizing with Atlas is easier time application workload Cluster capacity
- 10. They both look confusing
- 11. application workload Cluster capacity How do you do this? With this?
- 12. Earning your Atlas Wings
- 13. Right sizing in 3 steps 1. Verify database is optimally configured 2. Evaluate remaining capacity § RAM § CPU § IOPS 3. Extrapolate to future
- 14. Step 1 – Verifying Database is Optimally Configured Optimal Indexes? • Document Metrics • Query Targeting • Scan and Order • Performance Advisor
- 15. Document Metrics : What to look for? What do they mean? Deleted Average rate per second of documents deleted meets your specified threshold. Inserted Average rate per second of documents inserted meets your specified threshold. Returned Average rate per second of documents returned meets your specified threshold. Updated Average rate per second of documents updated meets your specified threshold.
- 16. Document Metrics : Why Are they important? They are your Throughput Metrics! • Most Crucial and help avoid performance issues • Understand how cluster is being used • optimize performance and • Avoid overloading Database • Prevent resource saturation ü Know when to scale up or out
- 17. Query Targeting What is it used For? Scanned Objects/Returned Objects Number of documents examined to fulfill a query relative to the actual number of returned documents – Default: >= 1000 Scanned/Returned Number of index keys examined to fulfill a query relative to the actual number of returned documents – User Defined ; Not enabled by Default • alerts indicate inefficient queries Ø Ideal ratio should be 1 Ø High ratios indicate lack of indexes (Collection Scan) or partial index usage
- 18. Query Targeting What is it used For? Ø Ideal ratio should be 1 Ø High ratios indicate lack of indexes (Collection Scan) or partial index usage
- 19. Query Targeting When does it happens? • No index available to support queries or • Index partially supports query • How determine which Query: • Real-Time Performance Panel monitors • Cursor.explain() in mongo shell • MongoDB logs • Performance Advisor
- 20. Query Targeting When does it happens? • No index available to support queries or • Index partially supports query • How determine which Query: • MongoDB logs
- 21. Performance Advisor • No index available to support queries or • Index partially supports query • How determine which Query: • Performance Advisor
- 22. Query Targeting Ideal Scenario example Ideal ratio of 1!
- 23. Query Targeting Inefficient Scenarios/examples High ratio of 46.8K & 600K!
- 24. Scan and Order It is an indication that MongoDB did not use an index to sort data; it was sorted in memory Why is it important? Scan and Order The average rate per second over the selected sample period of queries that return sorted results that cannot perform the sort operation using an index.
- 25. Step 2 – How much extra capacity do we have? Start with CPU and IO • CPU Utilization • IO Utilization • IOPS Next Dig Deeper • Cache Activity • Cache Usage
- 26. There are only 4 resources: 1. CPU 2. RAM 3. Disk Space 4. IOPS
- 27. There are only 4 resources: 1. CPU 2. RAM 3. Disk Space 4. IOPS Atlas will automatically scale this one for you
- 28. CPU
- 29. Atlas Provides 3 CPU Metrics 1. System CPU – All processes (may be > 100%, multiple cores) 2. Normalized – All processes, normalized 0 – 100% 3. Process – MongoDB only
- 30. Minimal Additional CPU Capacity M40 • 15 GB RAM • 4 vCPUs
- 31. Extra CPU Capacity M40 • 15 GB RAM • 4 vCPUs
- 32. RAM and IO
- 33. RAM and Disk activity are interrelated • More RAM è less disk activity • Less RAM à more disk activity • Why?
- 34. Data and indexes are cached in RAM File System collections indexes CPU Memory indexes documents {} {} {} {} {} More RAM à more data in cache à less read from disk
- 35. MongoDB allocates ~50% RAM for cache OS MDB ? File System Cache Wired Tiger Cache Uncompressed Documents Indexes Compressed Documents Indexes Disk
- 36. How much RAM is enough? RAM > Working Set Working Set = Indexes + Frequently accessed documents Atlas displays the index size:
- 37. Index size gives us the lower bound for RAM Working Set = Indexes + Frequently accessed documents How big are the frequently accessed documents? There isn’t a an Atlas chart for that:
- 38. Cache Activity can provide some visibility Disk FSCacheWTCache Application
- 39. Working set doesn’t fit in RAM Disk FSCacheWTCache Application High cache and disk activity
- 40. What does this look like in Atlas? M40 • 15 GB RAM • 4 vCPUs Cache Activity Cache Usage
- 41. What does this look like in Atlas? M40 • 15 GB RAM • 4 vCPUs Cache Activity Cache Usage Disk Utilization
- 42. Available runway is short Queues are empty Query execution times aren’t horrible
- 43. How do we add more runway (capacity)? 1. Add RAM § More of working set in RAM § Shorter latency § Less IO 2. Add more IOPS § Read more data from disk per second Which should you choose?
- 44. Add more RAM or IO? Memory is much faster • ~ .1 us = memory access • ~150 us = 4K random read from SSD • Putting more of the working set in RAM will have big impact • Upgrade instance size • Add shard • Adding IOPS will provide a smaller boost
- 45. Example: working set fits in RAM M10 • 1.7 GB RAM • .5 vCPU • Lots of queries • Low cache activity
- 46. Low disk activity à working set fits in RAM • Excellent query performance • Low disk utilization
- 47. What about writes? • Available IOPS must be greater than: • IOPS required for writes • IOPS for reads not in cache • Cache activity will provide visibility into write IOPS as well
- 48. Cache usage with max write throughput M10 • 1.7 GB RAM • .5 vCPU • Write intensive workload
- 49. Cache usage with max write throughput Reads Writes WT Cache Size: • 256 MB • Max Dirty (20%) : 51.2 MB
- 50. In this case, CPU is the limiting factor
- 51. Step 3 – Do we have enough future capacity? At this point: 1. MongoDB is optimally configured 2. We understand our current level of utilization § CPU § IO/IOPS 3. We estimated our working set 4. Validated our working set estimate by looking at cache activity
- 52. Step 3 – Do we have enough future capacity? We need to understand how the workload will change in the future: • How much more data? • How many more queries? • New data types and queries?
- 53. For writes # Writes Today # Write IOPS Today # Future Writes # Future IOPS = # Future IOPS = (# Future Writes * Write IOPS Today) / # Writes Today Assumes application doesn’t change Make sure there is sufficient CPU capacity for additional writes
- 54. For Reads Need to understand how working set will change: 1. Static working set • More queries against same working set 2. Linear growth • More data, but same documents/indexes/queries 3. New working set • Changes to documents, indexes, queries
- 55. Static Working Set, but more queries Disk FSCacheWTCache Application W o r k i n g S e t
- 56. Static Working Set, but more queries 1. Scale reads IOPS proportionally (similar to writes) 2. Verify available CPU capacity § Scale up or out as necessary
- 57. Linear growth, same queries, same documents Disk FSCacheWTCache Application W o r k i n g S e t W o r k i n g S e t
- 58. Linear growth, same data, same documents 1. Scale write IOPS proportionally 2. Scale reads IOPS proportionally 3. Scale RAM proportionally 4. Verify available CPU capacity § Scale up or out as necessary
- 59. Changes to documents, queries, indexes, volume, … Extrapolate if changes are small If not, estimate RAM, Disk, IOPS as in: § Sizing MongoDB Clusters § MongoDB World 2017
- 60. Review 1. Optimally configured § Indexes support queries § Query targeting, Scan and Order, Performance Profiler 2. Evaluate remaining capacity § CPU § IO § Use cache activity to evaluate working set size 3. Extrapolate to future § Extrapolate linearly, if application changes are small
- 62. Updating your existing deck with this template Follow the steps below to update your slide deck to the new template style. Make a copy of this template Click File > Make a copy... Select slides from your current deck and click copy. Select desired slides from left sidebar > Click Cmd+C (Mac) / Ctrl+C (PC) Paste slides into the copy of this template. Click into left sidebar > Click Cmd+V (Mac) / Ctrl+V (PC) Change the theme of your pasted slides to match the new template. Select newly pasted slides in sidebar > Click Theme > Click In this presentation > Select MDBW19 Make sure to change all fonts in the slides to DIN (DINOT). Select each textbox in each slide > Click into font drop-down menu > Select DIN(DINOT) Download DIN(DINOT) here: DIN
- 63. Presenter name or subtitle here – keep it to one line or 57 characters Title of the presentation goes here – keep title to two lines maximum and/or 112 characters with spaces SocialMedia
- 64. Use this title slide layout when there are two speakers Speaker One, Title Speaker Two, Title SocialMedia SocialMedia
- 65. Use this title slide layout when there are two speakers and speaker name and title need to go on two lines Speaker One, Title and/or Company Speaker Two, Title and/or Company SocialMediaSocialMedia
- 66. Title with content slide – Keep the title to two lines maximum or 91 characters with spaces First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. § Bullet one - use Paragraph > Increase List Level to add bullet § Bullet two – click Increase List Level again for 2nd level bullet § Bullet three
- 67. Titles on one line looks so much better First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. § Bullet one - use Paragraph > Increase List Level to add bullet § Bullet two – click Increase List Level again for next level bullet § Bullet three – click Increase List Level again for next level bullet
- 68. Title with content and subtitle Subtitle First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. § Bullet one - use Paragraph > Increase List Level to add bullet § Bullet two – click Increase List Level again for 2nd level bullet § Bullet three
- 69. Title with bar chart 1 4.3 2.5 3.5 4.5 2.4 4.4 1.8 2.8 2 2 3 5 Category 1 Category 2 Category 3 Category 4 Series 1 Series 2 Series 3
- 70. Title with bar chart 2 – use green to highlight MongoDB data 4.3 2.5 3.5 4.5 2.4 4.4 1.8 2.8 2 2 3 5 Category 1 Category 2 Category 3 Category 4 Series 1 Series 2 Series 3
- 71. Title with build animation bar chart 3 150 250 200 100 100 50 200 100 250 200 100 300 CATEGORY 1 CATEGORY 2 CATEGORY 3 CATEGORY 4 Series 1 Series 2 Series 3
- 72. Title with build animation doughnut cart 35% 15%15% 15% 10% 10% 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 5th Qtr 6th Qtr
- 73. Title and table 1 Column 1 Column 2 Column 3 Column 4 Column 5 Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content Content
- 74. Title and table 2 Column 1 Column 2 Column 3 Column 4 Row 1 Content Content Content Content Row 2 Content Content Content Content Row 3 Content Content Content Content Row 4 Content Content Content Content Row 5 Content Content Content Content
- 75. Title two content First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. § Bullet one § Bullet § Bullet First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. § Bullet one § Bullet § Bullet
- 76. Title two content with subheads Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 77. Title with doughnut chart and text 35% 15%15% 15% 10% 10% 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 5th Qtr 6th Qtr First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 78. Title three content First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 79. Title three content with subheads Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 80. Title four content First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 81. Title four content with subheads Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 82. Title with content and big picture First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content.
- 83. Title left content right First line of copy is not bulleted. Use bold or green font treatment to place emphasize on content. Subhead
- 84. Timeline
- 85. Screenshot Slide
- 86. Title with infographic – 1 TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style. TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style. TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style. TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style. TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style. TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. It can be replaced with your own style.
- 87. Title with build animation infographic – 2 TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. 40% TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. 50% TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. 70% TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. 50% TITLE GOES HERE This is a sample text. You simply add your own text and description here. This text is fully editable. 90%
- 88. Statement or divider – Leaf
- 89. Statement or divider – Graphite
- 92. Eliot Horowitz CTO & Co-Founder
- 93. Eliot Horowitz CTO & Co-Founder
- 94. CTO & Co-Founder CTO & Co-Founder Eliot HorowitzEliot Horowitz
- 95. CTO & Co-Founder CTO & Co-Founder CTO & Co-Founder Eliot Horowitz CTO & Co-Founder Eliot Horowitz
- 96. CTO & Co-Founder CTO & Co-Founder CTO & Co-Founder Eliot Horowitz CTO & Co-Founder Eliot Horowitz CTO & Co-Founder Eliot Horowitz
- 97. URL/Hashtag can go here.
- 98. Title – no branding
- 99. This is a quote slide with white background for your presentation. Use green bold treatment if you want to emphasize content. Attributor Name
- 100. This is a quote slide with graphite background for your presentation. Use yellow bold treatment if you want to emphasize content. Attributor Name
- 102. URL/Hashtag can go here.
- 103. session.start_transaction() order = { line_items : [ { item : 5, quantity: 6 }, … ] } db.orders.insertOne( order, session=session ); for x in order.line_items: db.inventory.update( { _id : x.item } , { $inc : { number : -1 * x.qty } }, session=session ) session.commit_transaction() Code with title
- 104. order = { line_items : [ { item : 5, quantity : 6 }, … ] } db.orders.insertOne( order ); for x in order.line_items: db.inventory.update( { _id : x.item }, { $inc : { number : -1 * x.qty } }) Code comparison with title session.start_transaction() order = { line_items : [ { item : 5, quantity: 6 }, … ] } db.orders.insertOne( order, session=session ); for x in order.line_items: db.inventory.update( { _id : x.item } , { $inc : { number : -1 * x.qty } }, session=session ) session.commit_transaction()
- 105. session.start_transaction() order = { line_items : [ { item : 5, quantity: 6 }, … ] } db.orders.insertOne( order, session=session ); for x in order.line_items: db.inventory.update( { _id : x.item } , { $inc : { number : -1 * x.qty } }, session=session ) session.commit_transaction()
- 106. Graphic Assets
- 107. MDBW19 Logos Core Logo – Full Color For use on white or very light backgrounds Core Logo – Full Color For use on white or very light backgrounds
- 108. Icons – MongoDB Charts Analytics Mobile SyncBI 2.5 FunctionsODBC Driver Functions Triggers Database / MDB) QueryAuthorizationServer MobileDocuments / MDB Zoned Sharding Query Anywhere
- 109. Icons – generic 24/7 Support API API Tools Cloud Download Cluster Commercial License Community Data Subset FlexibleFAQEnterprise Features Insight Marketing Performance Presentation Pricing Quick Start Rocket Scale Security Support Type Conversion University Use Cases User Visibility Computer Download Flexible Schema Visualization Webinar Consistency Management Integration Search White Paper Deployment Flexibility
- 110. Partner Logos – from 2018 MDBW
- 111. Statement – Leaf Background
- 112. Statement – Graphite Background