Seamless database migration case study - from Firebase real-time database to PostgreSQL
Download as pptx, pdf0 likes92 views
This document describes a case study of seamlessly migrating a database from Firebase Realtime Database to PostgreSQL. It discusses: 1) The services provided by inline and reasons for initial choice of Firebase 2) Issues that arose with Firebase as the company grew that required a new database solution 3) A 7-phase process for migrating to PostgreSQL while maintaining service uptime and real-time capabilities 4) Lessons learned around planning, testing, and choosing cloud services to minimize vendor lock-in.
![/ 23
Decouple Firebase SDK, finally
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
+request(in path) : InvokableRestRequest
RestClientImpl
+request(in path) : InvokableRestRequest
<<interface>>
RestClient
+query(in name , in value)
+where(in name, in value)
+get(in handler : ResponseHandler)
+post(in handler : ResponseHandler)
+put(in handler : ResponseHandler)
+delete(in handler : ResponseHandler)
+getPlainText(in handler : ResponseHandler)
+invoke(in method : string, in body : object, in handler : ResponseHandler)
InvokableRestRequestImpl
InvokableRestRequest
+create(in reservation)
MutableReservationFirebaseWebService
<<interface>>
MutableReservationWebService
+create(in reservation)
MutableReservationRestfulWebService
+get() : MutableReservationWebServiceProvider
MutableReservationWebServiceProvider
+add(in request : SerializableRequest)
-persist(in request : SerializableRequest)
-remove(in request : SerializableRequest)
+runNext()
SerializableRestRequestQueue
+invoke()
+init()
-id : string
-timestamp : long
-completedURL : string
-httpHeaders : [string: string]
-body : object
-handler : ResponseHandler
SerializableRequest
Codeable
Only serialize the write operations, i.e., POST, PUT, and
DELETE
The requests must be
executed in order
Move the original Firebase
implementation to here
9](https://image.slidesharecdn.com/seamlessdatabasemigrationcasestudy-fromfirebasereal-timedatabasetopostgresql-201203152729/85/Seamless-database-migration-case-study-from-Firebase-real-time-database-to-PostgreSQL-10-320.jpg)
Seamless database migration case study - from Firebase real-time database to PostgreSQL
- 1. Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 8th Oct, 2020 Software Development Manager Pin-Ying Tu [email protected]
- 2. / 23 About • High-scale software engineer • US$1M+ spent on cloud • Hiring Manager :-) • Ph.D. NTUT • 8 years software engineering experience • inline Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 1 github.com/dbi1463
- 3. / 23 inline services 1/2 • Start from automated queueing – Realtime status update • Booking management – Multiple sources – Full capcaity management Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 2
- 4. / 23 inline services 2/2 • CRM – Actionable insights – Campaign SMS – Survey and coupon • Takeout and delivery and more – Menu management – Integration with multiple delivery partners Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 3
- 5. / 23 We are growing, quickly Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 4
- 6. / 23 Why did we choose Firebase real-time database? • Less backend work • Real-time data synchronization is a very important feature Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 5
- 7. / 23 Sharding for growth Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL main Shard 1 Shard 2 Cust 1 Cust 2 Cust 3 Cust 4 Without our own API server, clients need to be aware of which shard to connect Moving data around shards takes time and money Shards got slow again when data growth Script to move data from master to the new shard and deploy functions to each shard 6
- 8. / 23 New synchronization mechanism 1/2 • Minimize memory usage in client-side endpoints – Firebase iOS SDK consume huge memory if off-line mode is enabled – Firebase iOS SDK read/write performance become very bad when the network environment is not good • We want a faster and smaller memory footprint sync method – Data synchronization between multiple devices • 95% reservations should be synchronized in 5 seconds – Support off-line first • Still provide good user experience (do not block user operations) • Can not lose any reservation even the network environment is not good –Seamless • No service down time • The restaurants are not aware of the changes Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 7
- 9. / 23 New synchronization mechanism 2/2 Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL Persistent, ordered, and parallel requests Three long polling APIs for better UX experience Trigger cloud functions Publish a sync event to PubSub Background jobs receive sync events Background jobs handle the out of order issue and write data to Elasticsearch Background jobs touch Redis to notify hooked long polling requests Monitor the pending writes requests Monitor the cost distribution Unsync reporter 8
- 10. / 23 Decouple Firebase SDK, finally Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL +request(in path) : InvokableRestRequest RestClientImpl +request(in path) : InvokableRestRequest <<interface>> RestClient +query(in name , in value) +where(in name, in value) +get(in handler : ResponseHandler) +post(in handler : ResponseHandler) +put(in handler : ResponseHandler) +delete(in handler : ResponseHandler) +getPlainText(in handler : ResponseHandler) +invoke(in method : string, in body : object, in handler : ResponseHandler) InvokableRestRequestImpl InvokableRestRequest +create(in reservation) MutableReservationFirebaseWebService <<interface>> MutableReservationWebService +create(in reservation) MutableReservationRestfulWebService +get() : MutableReservationWebServiceProvider MutableReservationWebServiceProvider +add(in request : SerializableRequest) -persist(in request : SerializableRequest) -remove(in request : SerializableRequest) +runNext() SerializableRestRequestQueue +invoke() +init() -id : string -timestamp : long -completedURL : string -httpHeaders : [string: string] -body : object -handler : ResponseHandler SerializableRequest Codeable Only serialize the write operations, i.e., POST, PUT, and DELETE The requests must be executed in order Move the original Firebase implementation to here 9
- 11. / 23 Performance improvement Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL Off-line to online Bad network Create and edit n reservations and wait all reservations were synchronized to all devices X: time Y: pending writes per second Blue: Firebase SDK Red: Firebase REST API Normal network 10
- 12. / 23 The device sync dashboard Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL Device-to-device cost by location Upload latency by location Device-to-device cost by day 11
- 13. / 23 So far so good, but we want better • Scaling goals we want to achieve – 99.99% availability of customer-facing services, with: – Fast and flexible queries • Like other database, without index the query is very slow • Firebase real-time database query only supports one index • Need to build the indices programmatically by ourselves – Fully managed DB shard and persistence migrations • Firebase team migrate the instances without any notification • We can not schedule the migration • During the time to rebuild in-memory indices after migration, all accesses to Firebase database are queued – Able to scale workers horizontally in one AZ and across Azs • Firebase cloud functions have 1,000 deployment quota Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL As a result, we decide to migrate our main database. 12
- 14. / 23 Architecture Evolution • Current architecture • New architecture Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL Persistent, ordered, and parallel requests inline APIs write to PostgreSQL and touch Redis to notify the subscribed WebSocket handlers Devices receive events through WebSocket with some APIs to improve UX experience Background jobs to perform the functionalities of the original cloud functions 13
- 15. / 23 Phase 0 – Performance evaluation • Make sure PostgreSQL can meet our performance goals – 50 M data set – Maximum query time • Current 36ms avg Goal: 200 rows in 30 ms – Queries per second • Current: 117/sec sustained Goal (5x): 585/sec sustained – Inserts/Updates per second • Current: 40/sec sustained Goal (5x): 200/sec sustained Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 14 Connection pool
- 16. / 23 Phase 1 – Sync new reservations to PostgreSQL • Not breaking any existing system – Still a few restaurants use the version before long polling – Not every restaurant upgrade app frequently • Use feature toggle to control the switch between two methods Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 15
- 17. / 23 Phase 2 – Make sure WebSocket works • Use feature toggle (remote control) to switch between WebSocket and long polling Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 16
- 18. / 23 Phase 3 – Copy old reservations to PostgreSQL • Controllable copy scopes – By brand or by shard Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 17
- 19. / 23 Phase 4 – Move cloud functions • Totally 31 cloud functions moved – Use feature toggle to enable/disable the cloud functions Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 18
- 20. / 23 Phase 5 – inline APIs • Implement the new inline APIs and clients adopt APIs – Also write back to Firebase • Fully tested with all combination on beta – Firebase REST API – inline API – WebSocket – Long polling Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 19
- 21. / 23 Phase 6 – controlled roll out • Planned roll out – Start to migrate a company that is willing to try new method – Start to migrate shard by shard until all shards migrated Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 20
- 22. / 23 Phase 7 - Remove Firebase dependency • After migrate all shards and companies, trigger the force upgrade notification • Remove Firebase dependency from inline APIs Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 21
- 23. / 23 Lession learned 1/2 • Abstraction layer – Try to minimize vendor lock-in for fundamentals of your system – Even if it costs more in time during the first implementation • Cost estimation – Don't forget to turn off the load generator – Set up a spending rate monitor • Planning is important – Help design thinking • Testing is important – Unit testing, end-to-end testing, and load testing all help us not breaking anything important to our customers depend on Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 22
- 24. / 23 Lession learned 2/2 • Always contact the cloud service support team – They have more than the public information and wouldn’t let you know if you did not ask • Choose cloud service platform carefully – Do some evaluation for your needs before choosing a cloud service platform • Data synchronization is extreme difficult – Do not build your own synchronization unless the current solution can not meet your needs Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL Any question? 23
Editor's Notes
- #25: AND unless you have plenty of time, and can deal effectively with complex GCP/AWS configurations