SlideShare a Scribd company logo

Chapman: Building a High-Performance Distributed Task Service with MongoDB

MongoDB, profile picture
MongoDB

The document outlines the process of building a distributed job queue using MongoDB, emphasizing the importance of task management in handling long-running processes and prioritizing tasks. It discusses various queue options, reliability issues with workers, and provides a roadmap with code snippets for implementing a scheduled priority queue while ensuring low latency. Key approaches are also mentioned for managing worker reliability through timeouts and client identity handling.

Technology
1 of 164
Downloaded 22 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
Chapman: Building a Distributed Job Queue in MongoDB Rick Copeland @rick446 @synappio rick@synapp.io
@rick446 @synappio Getting to Know One Another
@rick446 @synappio Getting to Know One Another Rick
@rick446 @synappio Getting to Know One Another Rick
@rick446 @synappio WhatYou’ll Learn
@rick446 @synappio WhatYou’ll Learn How to…
@rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB
@rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB
@rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
@rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
@rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions) Build low-latency reactive systems
@rick446 @synappio Why a Queue?
@rick446 @synappio Why a Queue? • Long-running task (or longer than the web can wait)
@rick446 @synappio Why a Queue? • Long-running task (or longer than the web can wait) • Farm out chunks of work for performance
@rick446 @synappio Things I Worry About
@rick446 @synappio Things I Worry About • Priority
@rick446 @synappio Things I Worry About • Priority • Latency
@rick446 @synappio Things I Worry About • Priority • Latency • Unreliable workers
@rick446 @synappio Queue Options
@rick446 @synappio Queue Options • SQS? No priority
@rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory
@rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility
@rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility • ZeroMQ? Lack of persistence
@rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility • ZeroMQ? Lack of persistence • What about MongoDB?
@rick446 @synappio Chapman Graham Arthur Chapman 8 January 1941 – 4 October 1989
@rick446 @synappio Roadmap
@rick446 @synappio Roadmap • Building a scheduled priority queue
@rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers
@rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers • Shared resources
@rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers • Shared resources • Managing Latency
@rick446 @synappio Building a Scheduled Priority Queue
@rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );!
@rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );! FIFO
@rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );! FIFO Get earliest message for processing
@rick446 @synappio Step 1: Simple Queue
@rick446 @synappio Step 1: Simple Queue Good
@rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO
@rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad
@rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad • No priority (other than FIFO)
@rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad • No priority (other than FIFO) • No handling of worker problems
@rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );
@rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); MinValid Time
@rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); MinValid Time Get earliest message for processing
@rick446 @synappio Step 2: Scheduled Messages
@rick446 @synappio Step 2: Scheduled Messages Good
@rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks
@rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks Bad
@rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks Bad • Be careful with the word “now”
@rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );
@rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); Add Priority
@rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); Add Priority
@rick446 @synappio Step 3: Priority
@rick446 @synappio Step 3: Priority Good
@rick446 @synappio Step 3: Priority Good • Priorities are handled
@rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority
@rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority Bad
@rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority Bad • No handling of worker problems
@rick446 @synappio Handling Unreliable Workers
@rick446 @synappio Approach 1 Timeouts db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z"),! "ts_timeout" : ISODate("2025-01-01T00:00:00.000Z")! }! });! ! db.message.ensureIndex({“s.status": 1, “s.ts_timeout": 1})! !
@rick446 @synappio Approach 1 Timeouts db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z"),! "ts_timeout" : ISODate("2025-01-01T00:00:00.000Z")! }! });! ! db.message.ensureIndex({“s.status": 1, “s.ts_timeout": 1})! ! Far-future placeholder
@rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.ts_timeout': now + processing_time } }! }! );! ! // Timeout message ("unlock")! db.message.update(! {'s.ts_status': 'reserved', 's.ts_timeout': {'$lt': now}},! {'$set': {'s.status': 'ready'}},! {'multi': true}); Approach 1 Timeouts
@rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.ts_timeout': now + processing_time } }! }! );! ! // Timeout message ("unlock")! db.message.update(! {'s.ts_status': 'reserved', 's.ts_timeout': {'$lt': now}},! {'$set': {'s.status': 'ready'}},! {'multi': true}); Client sets timeout Approach 1 Timeouts
@rick446 @synappio Approach 1 Timeouts
@rick446 @synappio Approach 1 Timeouts Good
@rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout
@rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad
@rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad • Requires periodic “unlock” task
@rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad • Requires periodic “unlock” task • Slow (but “live”) workers can cause spurious timeouts
@rick446 @synappio db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "cli": "--------------------------"! "ts_enqueue" : ISODate("2015-03-02T..."),! "ts_timeout" : ISODate("2025-...")! }! }); Approach 2 Worker Identity
@rick446 @synappio db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "cli": "--------------------------"! "ts_enqueue" : ISODate("2015-03-02T..."),! "ts_timeout" : ISODate("2025-...")! }! }); Client / worker placeholder Approach 2 Worker Identity
@rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Approach 2 Worker Identity
@rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Mark the worker who reserved the message Approach 2 Worker Identity
@rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Mark the worker who reserved the message Messages reserved by dead workers are unlocked Approach 2 Worker Identity
@rick446 @synappio Approach 2 Worker Identity
@rick446 @synappio Approach 2 Worker Identity Good
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad • Requires periodic “unlock” task
@rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad • Requires periodic “unlock” task • Unlock updates can be slow
@rick446 @synappio Shared Resources
@rick446 @synappio Complex Tasks Group check_smtp Analyze Results Update Reports Pipeline
@rick446 @synappio Semaphores
@rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp)
@rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp) • Some services just have a hard time with 144 threads hitting them simultaneously
@rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp) • Some services just have a hard time with 144 threads hitting them simultaneously • Need a way to limit our concurrency
@rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, …
@rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays
@rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays • Releasing the semaphore makes queued messages available for dispatch
@rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays • Releasing the semaphore makes queued messages available for dispatch • Use $slice (2.6) to keep arrays the right size
@rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False
@rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False Pessimistic update
@rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False Pessimistic update Compensation
@rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
@rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release
@rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release Awaken queued message(s)
@rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release Awaken queued message(s) Some magic (covered later)
@rick446 @synappio Message States ready acquirequeued busy
@rick446 @synappio Message States ready acquirequeued busy • Reserve the message
@rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources
@rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources • Process the message
@rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources • Process the message • Release resources
@rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
@rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
@rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores # semaphores acquired message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
@rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores # semaphores acquired message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...} Prefer partially-acquired messages
@rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True
@rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress
@rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress Failure to acquire (already queued)
@rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress Failure to acquire (already queued) Resources acquired, message ready to be processed
@rick446 @synappio Acquire Resources def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 1 (race condition)'''! if reserve(sem_id, msg_id, sem_size):! return True! else:! db.message.update(! {'_id': msg_id},! {'$set': {'s.status': 'queued'}})! return False
@rick446 @synappio Acquire Resources def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 1 (race condition)'''! if reserve(sem_id, msg_id, sem_size):! return True! else:! db.message.update(! {'_id': msg_id},! {'$set': {'s.status': 'queued'}})! return False Here be dragons!
@rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}})
@rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’?
@rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’?
@rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’? That’s the dragon.
@rick446 @synappio Release Resources (v2) def make_dispatchable(msg_id):! res = db.message.update(! {'_id': msg_id, 's.status': 'acquire'},! {'$set': {'s.event': True}})! if not res['updatedExisting']:! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}})
@rick446 @synappio Release Resources (v2) def make_dispatchable(msg_id):! res = db.message.update(! {'_id': msg_id, 's.status': 'acquire'},! {'$set': {'s.event': True}})! if not res['updatedExisting']:! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) Hey, something happened!
@rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False
@rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False Nothing’s happened yet!
@rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False Nothing’s happened yet! Check if something happened
@rick446 @synappio One More Race…. def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
@rick446 @synappio One More Race…. def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
@rick446 @synappio Compensate! def fixup_queued_messages():! for msg in db.message.find({'s.status': 'queued'}):! sem_id = msg['semaphores'][msg['s']['sub_status']]! sem = db.semaphore.find_one(! {'_id': sem_id, 'queued': msg['_id']})! if sem is None:! db.message.m.update(! {'_id': msg['_id'], ! 's.status': 'queued', ! 's.sub_status': msg['sub_status']},! {'$set': {'s.status': 'ready'}})
@rick446 @synappio Managing Latency
@rick446 @synappio Managing Latency • Reserving messages is expensive • Use Pub/Sub system instead • Publish to the channel whenever a message is ready to be handled • Each worker subscribes to the channel • Workers only ‘poll’ when they have a chance of getting work
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Make cursor tailable
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Holds open cursor for a while Make cursor tailable
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Holds open cursor for a while Make cursor tailable Don’t use indexes
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur import re, time! while True:! cur = get_cursor(! db.capped_collection, ! re.compile('^foo'), ! await_data=True)! for msg in cur:! do_something(msg)! time.sleep(0.1) Holds open cursor for a while Make cursor tailable Don’t use indexes
@rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur import re, time! while True:! cur = get_cursor(! db.capped_collection, ! re.compile('^foo'), ! await_data=True)! for msg in cur:! do_something(msg)! time.sleep(0.1) Holds open cursor for a while Make cursor tailable Don’t use indexes Still some polling when no producer, so don’t spin too fast
@rick446 @synappio Building in retry... def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'id': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur
@rick446 @synappio Building in retry... def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'id': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Integer autoincrement “id”
@rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur
@rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur id ==> ts
@rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur id ==> ts Co-opt the oplog_replay option
@rick446 @synappio The Oplog • Capped collection that records all operations for replication • Includes a ‘ts’ field suitable for oplog_replay • Does not require a separate publish operation (all changes are automatically “published”)
@rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None
@rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry
@rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries
@rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries skip most recent
@rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries skip most recent return on anything new
@rick446 @synappio What We’ve Learned
@rick446 @synappio What We’ve Learned How to…
@rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB
@rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB
@rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
@rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
@rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions) Build low-latency reactive systems
@rick446 @synappio Tips
@rick446 @synappio Tips • findAndModify is ideal for queues
@rick446 @synappio Tips • findAndModify is ideal for queues
@rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system
@rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system
@rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system • Use the oplog to build reactive, low-latency systems
Questions? Rick Copeland rick@synapp.io @rick446

More Related Content

PDF
Building a High-Performance Distributed Task Queue on MongoDB
MongoDB
 
KEY
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Mike Friedman
 
PDF
Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann
Danny Abukalam
 
PDF
Raymond Kuiper - Working the API like a Unix Pro
Zabbix
 
PDF
Aaron Mildenstein - Using Logstash with Zabbix
Zabbix
 
PPTX
Automating Zabbix with Puppet (Werner Dijkerman / 26-11-2015)
Nederlandstalige Zabbix Gebruikersgroep
 
PDF
Python in the database
pybcn
 
PPT
{{more}} Kibana4
琛琳 饶
 
Building a High-Performance Distributed Task Queue on MongoDB
MongoDB
 
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Mike Friedman
 
Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann
Danny Abukalam
 
Raymond Kuiper - Working the API like a Unix Pro
Zabbix
 
Aaron Mildenstein - Using Logstash with Zabbix
Zabbix
 
Automating Zabbix with Puppet (Werner Dijkerman / 26-11-2015)
Nederlandstalige Zabbix Gebruikersgroep
 
Python in the database
pybcn
 
{{more}} Kibana4
琛琳 饶
 

What's hot (20)

PDF
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
PPTX
Cassandra Java APIs Old and New – A Comparison
shsedghi
 
PDF
Storing 16 Bytes at Scale
Fabian Reinartz
 
PDF
Altitude NY 2018: Programming the edge workshop
Fastly
 
PPTX
Back to Basics Spanish 4 Introduction to sharding
MongoDB
 
PDF
Async and Non-blocking IO w/ JRuby
Joe Kutner
 
PDF
Altitude NY 2018: Leveraging Log Streaming to Build the Best Dashboards, Ever
Fastly
 
PPTX
Monitoring Docker with ELK
Daniel Berman
 
PDF
Query planner
Miguel Angel Nieto
 
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
PDF
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Zabbix
 
PDF
Back to Basics 2017: Mí primera aplicación MongoDB
MongoDB
 
PPTX
ELK Stack
Phuc Nguyen
 
PDF
Altitude SF 2017: Advanced VCL: Shielding and Clustering
Fastly
 
PDF
«Scrapy internals» Александр Сибиряков, Scrapinghub
it-people
 
KEY
What is the ServiceStack?
Demis Bellot
 
PPTX
Dapper performance
Suresh Loganatha
 
PPTX
What you need to know for postgresql operation
Anton Bushmelev
 
PPTX
ELK Ruminating on Logs (Zendcon 2016)
Mathew Beane
 
PDF
MongoDB World 2016: From the Polls to the Trolls: Seeing What the World Think...
MongoDB
 
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
Cassandra Java APIs Old and New – A Comparison
shsedghi
 
Storing 16 Bytes at Scale
Fabian Reinartz
 
Altitude NY 2018: Programming the edge workshop
Fastly
 
Back to Basics Spanish 4 Introduction to sharding
MongoDB
 
Async and Non-blocking IO w/ JRuby
Joe Kutner
 
Altitude NY 2018: Leveraging Log Streaming to Build the Best Dashboards, Ever
Fastly
 
Monitoring Docker with ELK
Daniel Berman
 
Query planner
Miguel Angel Nieto
 
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Zabbix
 
Back to Basics 2017: Mí primera aplicación MongoDB
MongoDB
 
ELK Stack
Phuc Nguyen
 
Altitude SF 2017: Advanced VCL: Shielding and Clustering
Fastly
 
«Scrapy internals» Александр Сибиряков, Scrapinghub
it-people
 
What is the ServiceStack?
Demis Bellot
 
Dapper performance
Suresh Loganatha
 
What you need to know for postgresql operation
Anton Bushmelev
 
ELK Ruminating on Logs (Zendcon 2016)
Mathew Beane
 
MongoDB World 2016: From the Polls to the Trolls: Seeing What the World Think...
MongoDB
 
Ad

Similar to Chapman: Building a High-Performance Distributed Task Service with MongoDB (20)

PDF
Shaddy Zeineddine: Queuing w/ MongoDB & BreakMedia's API
Caroline_Rose
 
PPTX
MongoDB as a Cloud Queue
MongoDB
 
PPTX
Queue in the cloud with mongo db
Nuri Halperin
 
PDF
Silicon Valley Code Camp 2016 - MongoDB in production
Daniel Coupal
 
PDF
MongoDB as Message Queue
MongoDB
 
PDF
cbq - Jobs and Tasks in the Background by Ortus
Ortus Solutions, Corp
 
PDF
MongoDB Tokyo - Monitoring and Queueing
Boxed Ice
 
PPTX
Job Queues Overview
joeyrobert
 
PDF
Production MongoDB in the Cloud
bridgetkromhout
 
PPTX
MongoDB Server Provisioning - From 2 Months to 2 Minutes
MongoDB
 
PDF
3450 - Writing and optimising applications for performance in a hybrid messag...
Timothy McCormick
 
PDF
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
PDF
MongoDB Solution for Internet of Things and Big Data
Stefano Dindo
 
PDF
Silicon Valley Code Camp 2014 - Advanced MongoDB
Daniel Coupal
 
ODP
Drupal 7 Queues
Philip Norton
 
PPTX
MongoDB World 2018: MongoDB for High Volume Time Series Data Streams
MongoDB
 
PDF
MongoDB Indexing Constraints and Creative Schemas
MongoDB
 
PDF
ITB 2023 cbq - Jobs And Tasks In the Background - Eric Peterson.pdf
Ortus Solutions, Corp
 
DOCX
White paper for High Performance Messaging App Dev with Oracle AQ
Jeff Jacobs
 
PPTX
Performance Tuning on the Fly at CMP.LY
MongoDB
 
Shaddy Zeineddine: Queuing w/ MongoDB & BreakMedia's API
Caroline_Rose
 
MongoDB as a Cloud Queue
MongoDB
 
Queue in the cloud with mongo db
Nuri Halperin
 
Silicon Valley Code Camp 2016 - MongoDB in production
Daniel Coupal
 
MongoDB as Message Queue
MongoDB
 
cbq - Jobs and Tasks in the Background by Ortus
Ortus Solutions, Corp
 
MongoDB Tokyo - Monitoring and Queueing
Boxed Ice
 
Job Queues Overview
joeyrobert
 
Production MongoDB in the Cloud
bridgetkromhout
 
MongoDB Server Provisioning - From 2 Months to 2 Minutes
MongoDB
 
3450 - Writing and optimising applications for performance in a hybrid messag...
Timothy McCormick
 
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
MongoDB Solution for Internet of Things and Big Data
Stefano Dindo
 
Silicon Valley Code Camp 2014 - Advanced MongoDB
Daniel Coupal
 
Drupal 7 Queues
Philip Norton
 
MongoDB World 2018: MongoDB for High Volume Time Series Data Streams
MongoDB
 
MongoDB Indexing Constraints and Creative Schemas
MongoDB
 
ITB 2023 cbq - Jobs And Tasks In the Background - Eric Peterson.pdf
Ortus Solutions, Corp
 
White paper for High Performance Messaging App Dev with Oracle AQ
Jeff Jacobs
 
Performance Tuning on the Fly at CMP.LY
MongoDB
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 

Recently uploaded (20)

PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
KodekX | Application Modernization Development
KodekX
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
KodekX | Application Modernization Development
KodekX
 

Chapman: Building a High-Performance Distributed Task Service with MongoDB

  • 1. Chapman: Building a Distributed Job Queue in MongoDB Rick Copeland @rick446 @synappio [email protected]
  • 2. @rick446 @synappio Getting to Know One Another
  • 3. @rick446 @synappio Getting to Know One Another Rick
  • 4. @rick446 @synappio Getting to Know One Another Rick
  • 7. @rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB
  • 8. @rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB
  • 9. @rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
  • 10. @rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
  • 11. @rick446 @synappio WhatYou’ll Learn How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions) Build low-latency reactive systems
  • 13. @rick446 @synappio Why a Queue? • Long-running task (or longer than the web can wait)
  • 14. @rick446 @synappio Why a Queue? • Long-running task (or longer than the web can wait) • Farm out chunks of work for performance
  • 16. @rick446 @synappio Things I Worry About • Priority
  • 17. @rick446 @synappio Things I Worry About • Priority • Latency
  • 18. @rick446 @synappio Things I Worry About • Priority • Latency • Unreliable workers
  • 21. @rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory
  • 22. @rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility
  • 23. @rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility • ZeroMQ? Lack of persistence
  • 24. @rick446 @synappio Queue Options • SQS? No priority • Redis? Can’t overflow memory • Rabbit-MQ? Lack of visibility • ZeroMQ? Lack of persistence • What about MongoDB?
  • 25. @rick446 @synappio Chapman Graham Arthur Chapman 8 January 1941 – 4 October 1989
  • 27. @rick446 @synappio Roadmap • Building a scheduled priority queue
  • 28. @rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers
  • 29. @rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers • Shared resources
  • 30. @rick446 @synappio Roadmap • Building a scheduled priority queue • Handling unreliable workers • Shared resources • Managing Latency
  • 31. @rick446 @synappio Building a Scheduled Priority Queue
  • 32. @rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );!
  • 33. @rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );! FIFO
  • 34. @rick446 @synappio Step 1: Simple Queue db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );! FIFO Get earliest message for processing
  • 36. @rick446 @synappio Step 1: Simple Queue Good
  • 37. @rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO
  • 38. @rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad
  • 39. @rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad • No priority (other than FIFO)
  • 40. @rick446 @synappio Step 1: Simple Queue Good • Guaranteed FIFO Bad • No priority (other than FIFO) • No handling of worker problems
  • 41. @rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );
  • 42. @rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); MinValid Time
  • 43. @rick446 @synappio Step 2: Scheduled Messages db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! “ts_after" : ISODate(…),! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex(! {'s.status': 1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready', ’s.ts_after': {$lt: now }},! sort: {'s.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); MinValid Time Get earliest message for processing
  • 44. @rick446 @synappio Step 2: Scheduled Messages
  • 45. @rick446 @synappio Step 2: Scheduled Messages Good
  • 46. @rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks
  • 47. @rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks Bad
  • 48. @rick446 @synappio Step 2: Scheduled Messages Good • Easy to build periodic tasks Bad • Be careful with the word “now”
  • 49. @rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! );
  • 50. @rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); Add Priority
  • 51. @rick446 @synappio Step 3: Priority db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z")! }! });! ! db.message.ensureIndex({'s.status': 1, 's.pri': -1, 's.ts_enqueue': 1});! ! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {'s.status': 'reserved'} },! }! ); Add Priority
  • 53. @rick446 @synappio Step 3: Priority Good
  • 54. @rick446 @synappio Step 3: Priority Good • Priorities are handled
  • 55. @rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority
  • 56. @rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority Bad
  • 57. @rick446 @synappio Step 3: Priority Good • Priorities are handled • Guaranteed FIFO within a priority Bad • No handling of worker problems
  • 59. @rick446 @synappio Approach 1 Timeouts db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z"),! "ts_timeout" : ISODate("2025-01-01T00:00:00.000Z")! }! });! ! db.message.ensureIndex({“s.status": 1, “s.ts_timeout": 1})! !
  • 60. @rick446 @synappio Approach 1 Timeouts db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "ts_enqueue" : ISODate("2015-03-02T15:27:29.228Z"),! "ts_timeout" : ISODate("2025-01-01T00:00:00.000Z")! }! });! ! db.message.ensureIndex({“s.status": 1, “s.ts_timeout": 1})! ! Far-future placeholder
  • 61. @rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.ts_timeout': now + processing_time } }! }! );! ! // Timeout message ("unlock")! db.message.update(! {'s.ts_status': 'reserved', 's.ts_timeout': {'$lt': now}},! {'$set': {'s.status': 'ready'}},! {'multi': true}); Approach 1 Timeouts
  • 62. @rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.ts_timeout': now + processing_time } }! }! );! ! // Timeout message ("unlock")! db.message.update(! {'s.ts_status': 'reserved', 's.ts_timeout': {'$lt': now}},! {'$set': {'s.status': 'ready'}},! {'multi': true}); Client sets timeout Approach 1 Timeouts
  • 65. @rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout
  • 66. @rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad
  • 67. @rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad • Requires periodic “unlock” task
  • 68. @rick446 @synappio Approach 1 Timeouts Good • Worker failure handled via timeout Bad • Requires periodic “unlock” task • Slow (but “live”) workers can cause spurious timeouts
  • 69. @rick446 @synappio db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "cli": "--------------------------"! "ts_enqueue" : ISODate("2015-03-02T..."),! "ts_timeout" : ISODate("2025-...")! }! }); Approach 2 Worker Identity
  • 70. @rick446 @synappio db.message.insert({! "_id" : NumberLong("3784707300388732067"),! "data" : BinData(...),! "s" : {! "status" : "ready",! "pri": 30128,! "cli": "--------------------------"! "ts_enqueue" : ISODate("2015-03-02T..."),! "ts_timeout" : ISODate("2025-...")! }! }); Client / worker placeholder Approach 2 Worker Identity
  • 71. @rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Approach 2 Worker Identity
  • 72. @rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Mark the worker who reserved the message Approach 2 Worker Identity
  • 73. @rick446 @synappio // Reserve message! db.runCommand(! {! findAndModify: "message",! query: { 's.status': 'ready' },! sort: {'s.pri': -1, 's.ts_enqueue': 1},! update: { '$set': {! 's.status': 'reserved',! 's.cli': ‘client_name:pid',! 's.ts_timeout': now + processing_time } }! }! );! ! // Unlock “dead” client messages! db.message.update(! {'s.status': 'reserved', ! 's.cli': {'$nin': active_clients} },! {'$set': {'s.status': 'ready'}},! {'multi': true});! Mark the worker who reserved the message Messages reserved by dead workers are unlocked Approach 2 Worker Identity
  • 76. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers
  • 77. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers
  • 78. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers
  • 79. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad
  • 80. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad • Requires periodic “unlock” task
  • 81. @rick446 @synappio Approach 2 Worker Identity Good • Worker failure handled via out- of-band detection of live workers • Handles slow workers Bad • Requires periodic “unlock” task • Unlock updates can be slow
  • 85. @rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp)
  • 86. @rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp) • Some services just have a hard time with 144 threads hitting them simultaneously
  • 87. @rick446 @synappio Semaphores • Some services perform connection- throttling (e.g. Mailchimp) • Some services just have a hard time with 144 threads hitting them simultaneously • Need a way to limit our concurrency
  • 88. @rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, …
  • 89. @rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays
  • 90. @rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays • Releasing the semaphore makes queued messages available for dispatch
  • 91. @rick446 @synappio Semaphores Semaphore Active: msg1, msg2, msg3, … Capacity: 16 Queued: msg17, msg18, msg19, … • Keep active and queued messages in arrays • Releasing the semaphore makes queued messages available for dispatch • Use $slice (2.6) to keep arrays the right size
  • 92. @rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False
  • 93. @rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False Pessimistic update
  • 94. @rick446 @synappio Semaphores:Acquire db.semaphore.insert({! '_id': 'semaphore-name',! 'value': 16,! 'active': [],! 'queued': []});! ! def acquire(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$push': {! 'active': {! '$each': [msg_id], ! '$slice': sem_size},! 'queued': msg_id}},! new=True)! if msg_id in sem['active']:! db.semaphore.update(! {'_id': 'semaphore-name'},! {'$pull': {'queued': msg_id}})! return True! return False Pessimistic update Compensation
  • 95. @rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
  • 96. @rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release
  • 97. @rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release Awaken queued message(s)
  • 98. @rick446 @synappio Semaphores: Release def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated Actually release Awaken queued message(s) Some magic (covered later)
  • 101. @rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources
  • 102. @rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources • Process the message
  • 103. @rick446 @synappio Message States ready acquirequeued busy • Reserve the message • Acquire resources • Process the message • Release resources
  • 104. @rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
  • 105. @rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
  • 106. @rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores # semaphores acquired message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...}
  • 107. @rick446 @synappio Reserve a Message msg = db.message.find_and_modify(! {'s.status': 'ready'},! sort=[('s.sub_status', -1), ('s.pri', -1), ('s.ts', 1)],! update={'$set': {'s.w': worker, 's.status': 'acquire'}},! new=True) message.s == {! pri: 10,! semaphores: ['foo'],! status: 'ready',! sub_status: 0,! w: '----------',! ...} Required semaphores # semaphores acquired message.s == {! pri: 10,! semaphores: ['foo'],! status: 'acquire! sub_status: 0,! w: worker,! ...} Prefer partially-acquired messages
  • 108. @rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True
  • 109. @rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress
  • 110. @rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress Failure to acquire (already queued)
  • 111. @rick446 @synappio Acquire Resources def acquire_resources(msg):! for i, sem_id in enumerate(msg['s']['semaphores']):! if i < msg['sub_status']: # already acquired! continue! sem = db.semaphore.find_one({'_id': 'sem_id'})! if try_acquire_resource(sem_id, msg['_id'], sem['value']):! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.sub_status': i}})! else:! return False! db.message.update(! {'_id': msg['_id']}, {'$set': {'s.status': 'busy'}})! return True Save forward progress Failure to acquire (already queued) Resources acquired, message ready to be processed
  • 112. @rick446 @synappio Acquire Resources def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 1 (race condition)'''! if reserve(sem_id, msg_id, sem_size):! return True! else:! db.message.update(! {'_id': msg_id},! {'$set': {'s.status': 'queued'}})! return False
  • 113. @rick446 @synappio Acquire Resources def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 1 (race condition)'''! if reserve(sem_id, msg_id, sem_size):! return True! else:! db.message.update(! {'_id': msg_id},! {'$set': {'s.status': 'queued'}})! return False Here be dragons!
  • 114. @rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}})
  • 115. @rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’?
  • 116. @rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’?
  • 117. @rick446 @synappio Release Resources (v1) “magic” def make_dispatchable(msg_id):! '''Version 1 (race condition)'''! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) But what if s.status == ‘acquire’? That’s the dragon.
  • 118. @rick446 @synappio Release Resources (v2) def make_dispatchable(msg_id):! res = db.message.update(! {'_id': msg_id, 's.status': 'acquire'},! {'$set': {'s.event': True}})! if not res['updatedExisting']:! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}})
  • 119. @rick446 @synappio Release Resources (v2) def make_dispatchable(msg_id):! res = db.message.update(! {'_id': msg_id, 's.status': 'acquire'},! {'$set': {'s.event': True}})! if not res['updatedExisting']:! db.message.update(! {'_id': msg_id, 's.status': 'queued'},! {'$set': {'s.status': 'ready'}}) Hey, something happened!
  • 120. @rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False
  • 121. @rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False Nothing’s happened yet!
  • 122. @rick446 @synappio Acquire Resources (v2) def try_acquire_resource(sem_id, msg_id, sem_size):! '''Version 2'''! while True:! db.message.update(! {'_id': msg_id}, {'$set': {'event': False}})! if reserve(sem_id, msg_id, sem_size):! return True! else:! res = db.message.update(! {'_id': msg_id, 's.event': False},! {'$set': {'s.status': 'queued'}})! if not res['updatedExisting']:! # Someone released this message; try again! continue! return False Nothing’s happened yet! Check if something happened
  • 123. @rick446 @synappio One More Race…. def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
  • 124. @rick446 @synappio One More Race…. def release(sem_id, msg_id, sem_size):! sem = db.semaphore.find_and_modify(! {'_id': sem_id},! update={'$pull': {! 'active': msg_id, ! 'queued': msg_id}},! new=True)! ! while len(sem['active']) < sem_size and sem['queued']:! wake_msg_ids = sem['queued'][:sem_size]! updated = self.cls.m.find_and_modify(! {'_id': sem_id},! update={'$pullAll': {'queued': wake_msg_ids}},! new=True)! for msgid in wake_msg_ids:! make_dispatchable(msgid)! sem = updated
  • 125. @rick446 @synappio Compensate! def fixup_queued_messages():! for msg in db.message.find({'s.status': 'queued'}):! sem_id = msg['semaphores'][msg['s']['sub_status']]! sem = db.semaphore.find_one(! {'_id': sem_id, 'queued': msg['_id']})! if sem is None:! db.message.m.update(! {'_id': msg['_id'], ! 's.status': 'queued', ! 's.sub_status': msg['sub_status']},! {'$set': {'s.status': 'ready'}})
  • 127. @rick446 @synappio Managing Latency • Reserving messages is expensive • Use Pub/Sub system instead • Publish to the channel whenever a message is ready to be handled • Each worker subscribes to the channel • Workers only ‘poll’ when they have a chance of getting work
  • 128. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 129. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 130. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 131. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 132. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 133. Capped Collections Capped Collection • Fixed size • Fast inserts • “Tailable” cursors Tailable Cursor
  • 134. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur
  • 135. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Make cursor tailable
  • 136. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Holds open cursor for a while Make cursor tailable
  • 137. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Holds open cursor for a while Make cursor tailable Don’t use indexes
  • 138. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur import re, time! while True:! cur = get_cursor(! db.capped_collection, ! re.compile('^foo'), ! await_data=True)! for msg in cur:! do_something(msg)! time.sleep(0.1) Holds open cursor for a while Make cursor tailable Don’t use indexes
  • 139. @rick446 @synappio Getting a Tailable Cursor def get_cursor(collection, topic_re, await_data=True):! options = { 'tailable': True }! if await_data:! options['await_data'] = True! cur = collection.find(! { 'k': topic_re },! **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur import re, time! while True:! cur = get_cursor(! db.capped_collection, ! re.compile('^foo'), ! await_data=True)! for msg in cur:! do_something(msg)! time.sleep(0.1) Holds open cursor for a while Make cursor tailable Don’t use indexes Still some polling when no producer, so don’t spin too fast
  • 140. @rick446 @synappio Building in retry... def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'id': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur
  • 141. @rick446 @synappio Building in retry... def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'id': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! return cur Integer autoincrement “id”
  • 142. @rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur
  • 143. @rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur id ==> ts
  • 144. @rick446 @synappio Ludicrous Speed from pymongo.cursor import _QUERY_OPTIONS! ! def get_cursor(collection, topic_re, last_id=-1, await_data=True):! options = { 'tailable': True }! spec = { ! 'ts': { '$gt': last_id }, # only new messages! 'k': topic_re }! if await_data:! options['await_data'] = True! cur = collection.find(spec, **options)! cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes! if await:! cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])! return cur id ==> ts Co-opt the oplog_replay option
  • 145. @rick446 @synappio The Oplog • Capped collection that records all operations for replication • Includes a ‘ts’ field suitable for oplog_replay • Does not require a separate publish operation (all changes are automatically “published”)
  • 146. @rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None
  • 147. @rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry
  • 148. @rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries
  • 149. @rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries skip most recent
  • 150. @rick446 @synappio Using the Oplog def oplog_await(oplog, spec):! '''Await the very next message on the oplog satisfying the spec'''! last = oplog.find_one(spec, sort=[('$natural', -1)])! if last is None:! return # Can't await unless there is an existing message satisfying spec! await_spec = dict(spec)! last_ts = last['ts']! await_spec['ts'] = {'$gt': bson.Timestamp(last_ts.time, last_ts.inc - 1)}! curs = oplog.find(await_spec, tailable=True, await_data=True)! curs = curs.hint([('$natural', 1)])! curs = curs.add_option(_QUERY_OPTIONS['oplog_replay'])! curs.next() # should always find 1 element! try:! return curs.next()! except StopIteration:! return None most recent oplog entry finds most recent plus following entries skip most recent return on anything new
  • 152. @rick446 @synappio What We’ve Learned How to…
  • 153. @rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB
  • 154. @rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB
  • 155. @rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
  • 156. @rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions)
  • 157. @rick446 @synappio What We’ve Learned How to… Build a task queue in MongoDB Bring consistency to distributed systems (without transactions) Build low-latency reactive systems
  • 161. @rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system
  • 162. @rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system
  • 163. @rick446 @synappio Tips • findAndModify is ideal for queues • Atomic update + compensation brings consistency to your distributed system • Use the oplog to build reactive, low-latency systems