MongoDB World 2019: Becoming an Ops Manager Backup Superhero!

Joanna Cheng
Becoming an Ops Manager Backup Superhero!
MongoDB joannac-

Joanna Cheng
Team Lead, Technical Services, MongoDB

Who has used Ops Manager before?
• On-prem software – comes with Enterprise Advanced
• Monitoring MongoDB deployments
• Backup
• Management of MongoDB
• Upgrades
• Index builds
• Cluster administration

Who has used Ops Manager backup before?

My backups are stuck
My backups are too slow
I don’t know what’s happening
What’s wrong?

MongoDB World 2019: Becoming an Ops Manager Backup Superhero!

Technical Services can save
the day!

Agenda
• How Ops Manager Backup works
• How to diagnose some common errors
• What’s changed for backup in Ops Manager 4.2

How OM Backup works
1. Initial sync 2. Oplog apply 3. Snapshot

How OM Backup works – initial sync
Source
Backup agent HTTP service
Sync
store
Oplog
store
Backup daemon
HEAD
DB
Sync slices
Oplog slices
Ops Manager
Source
HEAD
DB

How OM Backup works – oplog apply
Source
Backup agent
Oplog
store
Backup daemon
Oplog slices
HTTP service
HEAD
DB

How OM Backup works – snapshot
Oplog
store
Backup daemon
Oplog slices
HEAD DB files
Break up
files into
blocks
MongoDB
Blockstore
Filesystem
store
S3HEAD
DB

How OM Backup works – initial sync
Source
Backup agent HTTP service Backup daemon
HEAD
DB
Sync slices
Oplog slices
Ops Manager
Sync
store
Oplog
store

Insufficient oplog size
- how to diagnose
q Look at metrics for Oplog Window
q Wait for the oplog window to increase naturally
q Increase size of oplog

Initial sync process
Source
Sync slices
Oplog slices
HEAD
DBSync
store
Oplog
store

Starting… (forever)
– backup agent down

– alerts

– backup agent logs

[2019/05/05 21:23:35.245] [agent.error]
[components/agent.go:manageReplicaSets:202] Error starting syncs.
RsId: myReplicaSetError getting a valid session. Err: ip-172-31-9-
252.ap-southeast-2.compute.internal:27020 error dialing: Failed to
make a direct connection. Address: ip-172-31-9-252.ap-southeast-
2.compute.internal:27020 Err: no reachable servers
…
error dialing primary: Error dialing to replica set. RsId:
myReplicaSet Hosts: [ip-172-31-9-252.ap-southeast-
2.compute.internal:27020] Err: no reachable servers
– connectivity issue to source

Sending collection information. Sync ID:
ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") Namespace:
admin.system.version
Sending collection information. Sync ID:
ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") Namespace: foo.bar
Sending numStreamingNamespaces information. Sync ID:
ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") number: 2
Finished namespace collectionInfo.
– large number of collections

– how to diagnose
q Check backup agent is functional
q Connectivity issues to source – check backup agent logs
q Still gathering collection info – do you have a lot of collections?

[2019/05/19 04:13:15.191]
[agent.sync.myReplicaSet.5ce0d7a80bb131032173e503.error]
[components/sync.go:Run:246] Error sending sync slices.
Error syncing a collection. Namespace: `database.collection`
Err: operation was interrupted
Transferring … (forever)
– connectivity issue to source

Initial sync process
Source
Sync
store
Oplog
store
Backup daemon
Sync slices
Oplog slices
HEAD
DB

[2019/05/13 12:18:08.993]
Pushing sync slice. Sync ID:
ObjectIdHex("5cd958960bb131032172269e") Namespace: test.foo Slice
#356
[2019/05/13 12:18:27.993]
Finished pushing sync slice. Sync ID:
ObjectIdHex("5cd958960bb131032172269e") Namespace: test.foo Slice
#356 Request Time 18589ms
- slow transfer to Sync Store

[2019/05/02 12:37:21.357]
[agent.sync.production.5cd4d1bb0bb1317ecc9d0a17.info]
[components/mothership.go:PushSyncSlice:320] Total Slice #1003 -
server syncStore is full. 11th attempt. Will resend this slice
again soon.
– issues writing to Sync Store

Sync Store
Backup agent Backup daemon

On Linux
/opt/mongodb/mms/logs/*
On Windows
C:MMSDataServerLog
– backup daemon logs

2018-02-10T20:22:06.780-0600 [class
com.xgen.svc.brs.svc.AssignmentThread =>
Space used: 2,252,138,352,640 bytes,
Space free: 324,684,738,560 bytes
2018-02-10T20:22:06.782-0600 [class
com.xgen.svc.brs.svc.AssignmentThread =>
backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production] ERROR
backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production
[assignmentFailed:527]
Assignment failed: No Daemon found with suitable conditions.
Filesize: 402.7 GB Oplog Churn: 0.0 GB/hr RequireSSD: false,
rsId=production, groupId=5ccbd7d20bb1317ecc8256ee
– no suitable daemon

Failed to decompress mongodb-linux-x86_64-rhel70-4.0.9.tgz to
/opt/mongodb/mms/mongodb-releases java.io.IOException: No
space left on device
Could not find appropriate mongod in /opt/mongodb/mongodb-
releases/, versions available to MMS: 3.4.3, 3.4.4, 3.4.2,
3.4.9. Expecting version 3.6.12 or greater, module preference:
enterprisePreferred
– missing binaries

– other issues on the daemon
• Missing prerequisites for MongoDB Enterprise
• Problems with the HEAD database
• Corruption
• Many collections (which slows mongod startup)
• Resource contention

– how to diagnose
q Check backup agent logs
q Problems connecting to source?
q Problems sending to sync store?
q Is the job assigned to a daemon?
q Check backup daemon logs
q Is the daemon working on the job?

[2019/05/13 12:20:12.513] [agent.oplog.myReplicaSet.debug]
[components/agent.go:func1:359] Successfully finished pushing oplog
slice. {ts: 1557740046:1} -> {ts: 1557740106:1} Num slices: 1 Num
docs: 5. Request Time 11ms
[2019/05/13 13:12:39.277] [agent.oplog.myReplicaSet.debug]
[components/agent.go:func1:359] Successfully finished pushing oplog
slice. {ts: 1557752912:192} -> {ts: 1557752912:3292} Num slices: 1
Num docs: 3100. Request Time 1004ms
Oplog behind
- Backup agent too slow

[2019/05/02 12:37:21.208] [agent.oplog.production.error]
[components/mothership.go:doChunkedPushRequest:1070] Failed doing a
chunked request.
Oplog behind
- Failing to push slice

Oplog behind
- Failing to push slice
Source
Oplog
store
Backup daemon
Oplog slices
HEAD
DB

Oplog behind
– how to diagnose
q Check agents page
q Check backup agent logs
q Check network to oplog store
q Check health of oplog store

2019-05-18T11:38:29.170-0500 [Daemon #1: class
com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs.
5cd958960bb131032172269e.production
[OplogSliceCompiler.java.work:270] - OplogSlice: Range:
1557217382:2778 -> 1557217385:6790; NumDocs: 22930
2019-05-18T11:38:38.817-0500 [Daemon #1: class
com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs.
[OplogSliceCompiler.java.work:421] - Oplogs to apply: 22930;
Skipped before: 0; Skipped from overlap: 0, Skipped after: 0
Snapshot behind
– daemon can’t keep up

2019-05-18T13:00:54.809-0500 [Daemon #1: class
com.xgen.svc.brs.job.SnapshotJob] INFO backup.jobs.
[SnapshotJob.java.initiateSnapshot:123] - Starting a snapshot job.
2019-05-18T15:01:07.072-0500 [Daemon #1: class
com.xgen.svc.brs.job.SnapshotJob =>
5cd958960bb131032172269e/production ] DEBUG
com.xgen.svc.brs.grid.Daemon [Daemon.java.iterate:138] - Job: class
com.xgen.svc.brs.job.SnapshotJob finished. JobResult: OK.
Snapshot behind
– snapshot is taking too long to complete

Snapshot behind
– snapshot is taking too long to complete
• Change in data
• Many updates - less deduplication (if applicable)
• Many inserts - more data to save
• Network issue
• Storage speed issue

Snapshot behind
– how to diagnose
q Did the daemon start the snapshot?
q Check the logs to see why it’s falling behind
q Is the snapshot taking a long time?
q Storage slowness
q Network slowness

Oplog apply process
Source
Oplog
store
Backup daemon
Oplog slices
HEAD
DB

[2019/05/10 06:39:51.135] [agent.oplog.myReplicaSet.warn]
[components/oplog.go:TailOplog:253] Bad match. Expected: {ts:
1557469781:1 h: -786638763670375692, t: 1} Received: {ts:
1557470350:1 h: -8704482524044120259, t: -1}
…
[2019/05/10 06:40:51.152] [agent.commonPoints.myReplicaSet.warn]
[components/rollback.go:Run:127] Failed to find a common point.
Needs resync
– lost oplog tail

In the daemon logs:
Error applying ops. Requesting resync.
In the HEAD DB logs:
2019-05-21T03:13:53.140+0000 [conn3] replication update of non-mod
failed: { ts: Timestamp 1558162386103|12, h: 361936184013300100, v:
2, op: "u", ns: ”database.collection", o2: {actual update here}
Needs resync
– error in applyOps

In the daemon logs:
2019-05-24T00:01:17.522+0000 [Daemon #3: class
com.xgen.svc.brs.job.ApplyOpsJob =>
5ccbd7d20bb1317ecc8256ee/myReplicaSet] DEBUG
backup.jobs.5ccbd7d20bb1317ecc8256ee.myReplicaSet
[ReplicaSetJob.java.startMongo:127] - MongodManager - Requested
Version: 4.0.9, Matching Version: 4.0.9, Matching Path:
/opt/mongodb/mms/mongodb-releases/mongodb-linux-x86_64-amazon-
4.0.9/bin/mongod, HEAD Path:
/backup/5ccbd7d20bb1317ecc8256ee/myReplicaSet/head/
Needs resync
– HEAD db logs

Needs resync
– how to diagnose
q Resync the backup to get it running again
q Backup agent initiated
q Why did it fail? Slowness? Rollback?
q Backup daemon initiated
q What operation did it fail on?

Investigation – where in the process?
q Where in the process am I stuck?
q Initial sync
q Oplog apply
q Snapshot

Investigation – which component?
q Which components are involved?
q Source replica set
q Backup agent
q Sync store / oplog store
q Backup daemon
q HEAD DB
q Snapshot storage

Investigation – what to look at?
q Backup agent logs
q Backup daemon logs
q HEAD DB logs
q Monitoring metrics
q Storage
q Network
q General utilisation

Investigation – I’m still stuck!
• Open a support ticket and
we’ll help you out!
• Make sure you include the
relevant data
§ Logs
§ Backup daemon
§ Backup agent
§ Screenshots
§ The diagnostic archive

Investigation – the diagnostic archive

What’s new in Ops
Manager 4.2

How OM Backup works – currently
Source
HEAD
DB
Ops Manager
Snapshot
storageSync
store
Oplog
store

How OM Backup works – 4.2 and above
Backup agent HTTP service Snapshot
storage
Source
(v4.2+)

New backup process in OM 4.2
• No more HEAD DBs
• Backup directly from source (via WiredTiger snapshots) to
snapshot storage
• For more information, attend Ben Cefalo’s talk at 1:45pm today

Next Steps
• Put this into practice!
• (don’t break your backups)
• Other talks
• Today, 1:45pm - Modern Data Backup and Recovery from On-
Premises to the Public Cloud, Ben Cefalo, MongoDB
• Tomorrow – attend Builder’s Fest (Atlas, Charts, Stitch,
Games, and more!)
• Come chat to me in the Leaf Lounge
• Reach out to me on LinkedIn: joannac-

Joanna Cheng - Team Lead, Technical Services
Any feedback would be greatly appreciated!
Thank You!

MongoDB World 2019: Becoming an Ops Manager Backup Superhero!

More Related Content

What's hot (20)

Similar to MongoDB World 2019: Becoming an Ops Manager Backup Superhero! (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB World 2019: Becoming an Ops Manager Backup Superhero!