MongoDB Performance Debugging
9 likes3,802 views
The document discusses debugging performance problems in MongoDB. It describes solutions tried such as denormalizing data, adding indexes, sharding, scaling hardware, and tagging shards. While performance improved with these solutions at times, slowness still occasionally occurred. The document advocates using monitoring tools like New Relic and Skylight to further analyze issues and find additional solutions.
![{
"cursor":"BtreeCursor_id_reverse",
"isMultiKey":false,
"n":0,
"nscannedObjects":227756,
"nscanned":227756,
"nscannedObjectsAllPlans":227756,
"nscannedAllPlans":227756,
"scanAndOrder":false,
"indexOnly":false,
"nYields":1779,
"nChunkSkips":0,
"millis":461,
"indexBounds":{
"_id":[
[
{
"$maxElement":1
},
{
"$minElement":1
}
]
]
},
"server":"Jaroussky.local:27017",
"filterSet":false
}](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-20-320.jpg)
![{
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minElement" : 1}]
]
}
}](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-21-320.jpg)
![{
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minElement" : 1}]
]
}
}](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-22-320.jpg)
![{"cursor" : "BtreeCursor
project_id_1__id_1",
"nscanned" : 6,
"indexBounds" : {
"project_id" : [[
ObjectId(x),ObjectId(x)
]],
"_id" : [[
{"$minElement" : 1},
{"$maxElement" : 1}
]]
}}](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-25-320.jpg)
![db.stats()['indexSize']
= 3852071824
= 3932614656
free -b(m3.medium)](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-43-320.jpg)
![db.stats()['indexSize']
= 3852071824
= 15775363072
free -b(r3.large)](https://image.slidesharecdn.com/dyaqo2hxtwcql2xfknoi-signature-1fff2c55bcb3dde9a35194ae2b283f3e1bcf0b42460983f225b77971d17f867d-poli-140707150427-phpapp01/85/MongoDB-Performance-Debugging-45-320.jpg)
MongoDB Performance Debugging
- 3. 20M crashes/day 2TB data / 50GB index 11 nodes / 3* availability zones
- 4. I don't care if MongoDB is slow I care if my app is slow
- 5. How to make app fast agai it's slow bec of the way I'm
- 6. What does slow mean?
- 11. It's slow :(
- 16. It's fast :)
- 17. It's slow :(
- 18. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)
- 19. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()
- 21. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
- 22. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
- 23. Solution 2 Index
- 25. {"cursor" : "BtreeCursor project_id_1__id_1", "nscanned" : 6, "indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]] }}
- 26. It's fast :)
- 27. It's slow :(
- 28. Only 1 query... Indexed properly...
- 29. mongostat
- 30. insert queryupdatedeletegetmorecommandflushesmapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g vsize resfaults lockeddbidxmiss% qr|qw ar|aw 641g 8.65g 7bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3bugsnag:23.4% 0 0|0 0|0 netInnetOut conn setrepl time 47k 73k 145bugsnag1 PRI 06:34:05 63k 99k 146bugsnag1 PRI 06:34:06 98k 124k 146bugsnag1 PRI 06:34:07
- 31. command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07
- 33. Solution 3 Shard
- 36. command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14
- 37. It's fast :)
- 38. It's slow :(
- 39. Only 1 query... Indexed properly... Lock % ok...
- 40. iostat
- 41. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/savgrq-sz xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01 xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23 xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41 avgqu-sz awaitr_awaitw_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40
- 42. rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60
- 43. db.stats()['indexSize'] = 3852071824 = 3932614656 free -b(m3.medium)
- 44. Solution 4 Scale
- 45. db.stats()['indexSize'] = 3852071824 = 15775363072 free -b(r3.large)
- 46. It's fast :)
- 47. It's slow :(
- 48. Main query seems fine... "Quick" queries sometimes slow...
- 49. mongotop
- 50. ns total read write 2 014-06-22T19:11:35 bugsnag.events 35ms 0ms 35ms bugsnag.errors 30ms 6ms 24ms bugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms
- 51. ns tot r w bugsnag.events 35 0 35 bugsnag.errors 30 6 24 bugsnag.projects 12 5 7 bugsnag.users 15 7 8
- 53. It's fast :)
- 55. Solution 2 Index
- 56. Solution 3 Shard
- 57. Solution 4 Scale
- 59. Solution 6 ...
- 61. It will be slow
- 62. You can speed it up