MongoDB Europe 2016 - Debugging MongoDB Performance

Asya Kamsky,
Lead Product Manager
MongoDB
Diagnostics and Debugging 3.4

“
I thought of my old teacher Joe Bell, ... of his eerie
trick of spotting details. If he were a detective he
would surely reduce this ... business to something
nearer an exact science.
—Arthur Conan Doyle

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the ﬁle again. If the red
x still appears, you may have to delete the image and then insert it again.

#MDBW16
Understanding The Patient
Understand the system
Knowledge
Monitor trends over time
Trends
Record all metrics "at rest"
Baseline

Gathering Data
“Data! Data! Data!
I can't make bricks without clay.”
-- Sherlock Holmes,
The Adventure of the Copper Beeches

“There is nothing like ﬁrst-hand evidence.”
-- Sherlock Holmes, A Study in Scarlet

#MDBW16
Available Tools: Say "AH"

#MDBW16
> db.isMaster( )
> rs.conf( )
> rs.status( )
> sh.status( )
> db.version( )
> db.serverCmdLineOpts( )

#MDBW16
> show dbs
> show collections
> db.getCollectionNames().forEach(function(c) {
printjson(db.getCollection(c).getIndexes());
});

#MDBW16
Available Tools: Compass

#MDBW16
Available Tools: OS
% free
% blockdev --report
% dmesg
% ulimit -a
% ifconfig, ip <...>, iptables
% iostat
% netstat
% top; htop
% perf
% iperf3

#MDBW16
Available Tools: OS for MongoDB
% mongostat
% mongotop

2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
mongod logs

2015-05-26T23:33:32.774-0500 I
2015-05-28T12:37:11.440-0500 I
2015-05-28T12:38:35.839-0500 I
D, I, W, E, F
3.0 mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND
2015-05-28T12:37:11.440-0500 I NETWORK
2015-05-28T12:38:35.839-0500 I QUERY
3.0
COMMAND,NETWORK,QUERY,REPL,ACCESS,INDEX,JOURNAL,SHARDING,WRITE,-
mongod logs

2015-05-26T23:33:32.774-0500 I COMMAND [conn45] command admin.$cmd command:
listDatabases { listDatabases: 1.0 } ntoskip:0 keyUpdates:0 writeConflicts:0
numYields:0 reslen:393 locks:{ Global: { acquireCount: { r: 12 } },
Database: { acquireCount: { r: 6 } } } 321ms
2015-05-28T12:37:11.440-0500 I NETWORK [initandlisten] connection accepted
from 127.0.0.1:48625 #183 (21 connections now open)
2015-05-28T12:38:35.839-0500 I QUERY [conn183] getmore
tableau.flights201406 query: { origin_city_market_id: 31703.0 } cursorid:
61957110347 ntoreturn:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0
numYields:176 nreturned:22579 reslen:1332181 locks:{ Global: { acquireCount:
{ r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection:
{ acquireCount: { r: 177 } } } 114ms
3.0 mongod logs

2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c
command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt:
4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }
keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1
keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187
locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r:
2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms
3.2 mongod logs

3.4 mongod logs
2016-06-01T15:30:04.373-0700 I COMMAND [conn99] command socialite.following
command: aggregate { aggregate: "following", pipeline: [ { $match: { _f: "45705" } },
{ $group: { _id: null, followees: { $addToSet: "$_t" } } }, { $lookup: { from: "following",
localField: "followees", foreignField: "_f", as: "fofollowees" } }, { $project: { fofs: { $setUnion:
[ "$followees", "$fofollowees._t" ] } } } ] } planSummary: IXSCAN { _f: 1, _t: 1 } keysExamined:1
docsExamined:0 numYields:11 reslen:316214 locks:{ Global: { acquireCount: { r: 3800 } },
Database: { acquireCount: { r: 1900 } }, Collection: { acquireCount: { r: 1900 } } }
protocol:op_query 218ms

3.4 mongod logs
2016-06-25T23:38:27.346-0500 I WRITE [conn128] update ycsb.usertable query:
{ _id: "user7074965863272626663" } planSummary: IDHACK update: { $set: { field1: BinData(0, 2
} } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: {
acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount:
{ w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 11ms

> db.getLogComponents()
{ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},
"control" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},
"index" : { "verbosity" : -1},
"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},
"replication" : { "verbosity" : -1},
"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,
"journal" : { "verbosity" : -1}},
"write" : { "verbosity" : -1}
}

> db.getLogComponents()
{ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},
"control" : { "verbosity" : -1},
"executor" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},
"index" : { "verbosity" : -1},
"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},
"replication" : { "verbosity" : -1},
"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,
"journal" : { "verbosity" : -1}},
"write" : { "verbosity" : -1},
"ftdc" : { "verbosity" : -1}
}

> db.setLogLevel( logLevel, component )
> db.setLogLevel( 1, "sharding" )
> db.setLogLevel( 2, "query" )

"It is of the highest importance ... to be able to recognize,
out of a number of facts, which are incidental and which vital. "
Sherlock Holmes, The Reigate Puzzle

https://github.com/rueckstiess/mtools
byThomasRückstieß

#MDBW16
mplotqueries
$ mplotqueries --help
usage: mplotqueries [OPTIONS] [logfile [logfile ...]]
A script to plot various information from logfiles. ...

#MDBW16
mplotqueries
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event}
type of plot (default=scatter with --yaxis duration)

#MDBW16
mplotqueries
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event}
type of plot (default=scatter with --yaxis duration)
--group GROUP specify value to group on.
All basic plot types can group on
'namespace', 'operation', 'thread', 'pattern' ...

"... what is out of the common is usually a guide
rather than a hindrance."
— Sherlock Holmes, A Study in Scarlet

mplotqueries
% mplotqueries firstmongo.log --type nscanned/n

2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
mplotqueries

2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 }
planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172
nreturned:56 reslen:18745 13086ms
mplotqueries

"Eliminate all other factors, and the
one which remains must be the truth."

Sherlock Holmes -The Sign of Four

#MDBW16
Available Tools
% mongostat

#MDBW16
mongostat
insert query update delete getmore command % dirty % used flushes vsize res faults qr|qw ar|aw netIn netOut conn ReplSetName role ts

#MDBW16
Available Tools: in the Cloud

% mplotqueries –type connchurn

#MDBW16
db.currentOp()
> db.currentOp({secs_running:{$gt:10}})
{ "desc" : "conn3482",
"threadId" : "140512575297280",
"connectionId" : 3482,
"client_s" : "10.0.149.179:55295",
"active" : true,
"opid" : "SocialiteAtlas-shard-2:55547103",
"secs_running" : 13,
"microsecs_running" : NumberLong(13483812),
"op" : "query",
"ns" : "db1.coll1",
"numYields" : 48815,
"query" : {
"field1" : 5,

#MDBW16
db.currentOp()
> db.currentOp({desc:/^conn/,secs_running:{$gt:0}}).inprog.forEach(function(op) {
print(op.opid, op.op, op.ns, op.secs_running);
});
SocialiteAtlas-shard-0:68479014 query socialite.content 3
>
db.killOp ( opid )

Title Slide Option 1
Click here to add speaker name and title

Title Slide Option 2
Click here to add speaker name
and title

This is divider slide option 2

#MDBW16
Click to add title: keep it to 56 characters w/spaces
This is a typical content slide with full width body.
•  First level bullet list
•  Second level bullet list
•  Third level bullet list

#MDBW16
Two content
Click to add text.
Click to add text.

#MDBW16
Left content
Click to add text.

#MDBW16
Pie Chart
64%
25%
11%
1st Qtr 2nd Qtr 3rd Qtr
1st Quarter
Lorem ipsum dolor sit amet, onsectetur adipiscing
elit. Praesent sodales odio sit amet odio tristique .
2nd Quarter
3rd Quarter

#MDBW16
Bar Graph
0
1
2
3
4
5
6
Category 1 Category 2 Category 3 Category 4
Chart Title
Series 1 Series 2 Series 3

#MDBW16
Column Header 1 Column Header 2 Column Header 3 Column Header 4 Column Header 5
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Table

#MDBW16
Coding Example – Light Background
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});

#MDBW16
Coding Example – Dark Background
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});

#MDBW16
List slide – can also be used for agenda
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
01 Lorem ipsum dolor sit amet,
sit elit.
Lorem Ipsum
03Lorem ipsum dolor sit amet,
sit elit.
Lorem Ipsum
02
sit elit.
Lorem Ipsum
05 Lorem ipsum dolor sit amet,
sit elit.
Lorem Ipsum
06Lorem ipsum dolor sit amet,
sit elit.
Lorem Ipsum
05

#MDBW16
Columns and icons with copy (option 1)
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique.
Linked
amet, onsectetur
adipiscing elit.
Planning
amet, onsectetur
adipiscing elit.
Writing
amet, onsectetur
adipiscing elit.
Research

#MDBW16
Columns and icons with copy (option 2)
amet, onsectetur
adipiscing elit.
sit amet odio tristique .
Linked
amet, onsectetur
adipiscing elit.
Planning
amet, onsectetur
adipiscing elit.
Writing
amet, onsectetur
adipiscing elit.
Research

#MDBW16
Timeline or progress
2013
onsectetur adipiscing elit.
Praesent sodales odio sit
amet odio tristique sit elit.
2014
2015
2016

“
Quote sample. Lorem ipsum dolor sit amet,
onsectetur adipiscing elit amet sodales. Praesent
sodales odio sit amet odio tristique. Lorem ipsum
dolor sit amet, onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique. Lorem ipsum
dolor sit amet, onsectetur adipiscing elit.”

MongoDB Europe 2016 - Debugging MongoDB Performance

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MongoDB Europe 2016 - Debugging MongoDB Performance (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB Europe 2016 - Debugging MongoDB Performance