Back to Basics German 3: Einführung in Replica Sets
Download as PPTX, PDF0 likes592 views
The document provides an overview of MongoDB's replica sets, explaining their importance for data durability and high availability. It covers the lifecycle of replica sets, client-side interactions, and error handling protocols in development. Additionally, it includes practical examples and references to further reading on server discovery and monitoring in MongoDB.
![23
Client Side View
Secondary
host2
Secondary
host3
Primary
host1
Mongo
Client
Monitor
Thread 1
Monitor
Thread 2
{ ismaster : False,
secondary: True,
hosts : [ host1, host2, host3 ] }](https://image.slidesharecdn.com/b2b-webinar-3-replica-sets-1-170518103133/85/Back-to-Basics-German-3-Einfuhrung-in-Replica-Sets-23-320.jpg)
![24
What Does ismaster show?
>>> pprint.pprint( db.command( "ismaster" ))
{u'hosts': [u'JD10Gen-old.local:27017',
u'JD10Gen-old.local:27018',
u'JD10Gen-old.local:27019'],
u'ismaster' : False,
u'secondary': True,
u'setName' : u'replset',
…}
>>>](https://image.slidesharecdn.com/b2b-webinar-3-replica-sets-1-170518103133/85/Back-to-Basics-German-3-Einfuhrung-in-Replica-Sets-24-320.jpg)
![43
What Does This Mean? - Inserts
def insert_with_recovery( collection, doc ) :
doc[ "_id" ] = ObjectId()
try:
collection.insert_one( doc )
except pymongo.errors.ConnectionFailure, e:
logging.info( "Connection error: %s" % e )
try:
collection.insert_one( doc )
except DuplicateKeyError:
pass](https://image.slidesharecdn.com/b2b-webinar-3-replica-sets-1-170518103133/85/Back-to-Basics-German-3-Einfuhrung-in-Replica-Sets-43-320.jpg)
Back to Basics German 3: Einführung in Replica Sets
- 2. Back to Basics 2017 : Webinar 3 Einführung in Replica Sets Benjamin Lorenz Senior Solutions Architect MongoDB Frankfurt
- 3. 3 Zusammenfassung von Teil 1 und Teil 2 • Warum NoSQL • Unterschiedliche Typen von NoSQL-Datenbanken • MongoDB: Detaillierte Übersicht • Datenbankkonzepte • Installieren von MongoDB • Erstellen einer einfachen Blogging-Anwendung • Hinzufügen eines Index • Abfrageoptimierung mit explain()
- 4. 4 Ablauf heute • Dauerhaftigkeit (durability) von Daten • Der MongoDB-Ansatz: Das Replica Set • Replica Set-Lebenszyklus • Programmierung mit einem Replica Set
- 5. 5 HighAvailability and Data Durability – Replica Sets SecondarySecondary Primary
- 7. 7 Replica Set Node Failure SecondarySecondary Primary No Heartbeat
- 9. 9 New Replica Set – 2 Nodes SecondaryPrimary Heartbeat And New Primary
- 10. 10 Replica Set Repair SecondaryPrimary Secondary Rejoin and resync
- 12. Developing with Replica Sets
- 13. 13 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 14. 14 Strong Consistency SecondarySecondary Primary Client Application Client Driver Write Read
- 16. 16 Write Concerns • Network acknowledgement • Wait for error • Wait for journal sync • Wait for replication
- 17. 17 Write Concern : 0 (Unacknowledged)
- 18. 18 Write Concern : 1 (Acknowledged)
- 19. 19 Write Concern : majority
- 20. 20 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 21. 21 Start MongoClient c = MongoClient( "host1, host2", replicaSet="replset" )
- 22. 22 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client MongoClient( "host1, host2", replicaSet="replset" )
- 23. 23 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 { ismaster : False, secondary: True, hosts : [ host1, host2, host3 ] }
- 24. 24 What Does ismaster show? >>> pprint.pprint( db.command( "ismaster" )) {u'hosts': [u'JD10Gen-old.local:27017', u'JD10Gen-old.local:27018', u'JD10Gen-old.local:27019'], u'ismaster' : False, u'secondary': True, u'setName' : u'replset', …} >>>
- 26. 26 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2
- 27. 27 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3
- 28. 28 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 29. 29 Next Is Insert c = MongoClient( "host1, host2", replicaSet="replset" ) client.db.col.insert_one( { "a" : "b" } )
- 30. 30 Insert Will Block Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert
- 31. 31 ismaster response from Host 1 Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert ismaster
- 32. 32 Now Write Can Proceed Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert Insert
- 33. 33 Later Host 3 Responds Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 34. 34 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 35. 35 Life Intervenes Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖
- 36. 36 Monitor may not detect Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert ConnectionFailure
- 37. 37 So Retry Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 38. 38 Check for Primary Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 39. 39 Host 2 Is Primary Primary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 40. 40 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 41. 41 What Does This Mean? - Connect import pymongo client = pymongo.MongoClient() try: client.admin.command( "ismaster" ) except pymongo.errors.ConnectionFailure, e : print( "Cannot connect: %s" % e )
- 42. 42 What Does This Mean? - Queries import pymongo def find_with_recovery( collection, query ) : try: return collection.find_one( query ) except pymongo.errors.ConnectionFailure, e : logging.info( "Connection failure : %s" e ) return collection.find_one( query )
- 43. 43 What Does This Mean? - Inserts def insert_with_recovery( collection, doc ) : doc[ "_id" ] = ObjectId() try: collection.insert_one( doc ) except pymongo.errors.ConnectionFailure, e: logging.info( "Connection error: %s" % e ) try: collection.insert_one( doc ) except DuplicateKeyError: pass
- 44. 44 What Does This Mean? - Updates collection.update( { "_id" : 1 }, { "$inc" : { "counter" : 1 }})
- 45. 45 More Reading • The spec author Jess Jiryu Davis has a collection of links and his better version of this talk https://emptysqua.re/blog/server-discovery-and-monitoring-in-mongodb- drivers/ • The full server discovery and monitoring spec is on GitHub https://github.com/mongodb/specifications/blob/master/source/server- discovery-and-monitoring/server-discovery-and-monitoring.rst
- 46. Q&A
Editor's Notes
- #3: Who I am, how long have I been at MongoDB.
- #14: Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- #21: Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- #23: Calls i
- #24: Calls i
- #26: State machine, full set of states defined in spec.
- #27: Calls i
- #28: Calls i
- #29: Calls i
- #31: Needs a primary to complete a write.
- #32: Needs a primary to complete a write.
- #33: Needs a primary to complete a write.
- #34: Needs a primary to complete a write.
- #35: Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- #36: Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- #37: Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- #38: Primary is marked as unknown Wakes up all monitor threads to check for a primary.
- #39: Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- #40: Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- #41: Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- #43: Try once. This will accomdate elections. Other errore should be propagated.
- #44: Try once. This will accomdate elections. Other errore should be propagated.
- #45: Can you afford to over or under count. Operations need to be idempotent. Turn an update into a write of a document, cf EventSourcing. Then aggregate on the server.