Falcon Storage Engine Designed For Speed Presentation
1 like733 views
This document discusses the Falcon database system and its approach to achieving high performance. It provides an overview of Falcon's architecture including its use of record caching, multi-version concurrency control, and separate storage of indexes and data. It then covers Falcon's history and goals, techniques for handling different workload characteristics, challenges in competing with InnoDB on benchmarks, approaches to multi-threading, and its use of cycle locking to enable concurrent operations while avoiding locking conflicts.
Falcon Storage Engine Designed For Speed Presentation
- 1. Falcon - built for speed Ann Harrison Kevin Lewis MySQL Users' Conference April 2009
- 2. If it's so fast, why isn't it done yet?
- 3. Talk overview Falcon at a glance Project history Multi-threading for the database developer Cycle locking
- 4. Falcon at a glance – read first record MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 5. Falcon at a glance – read complete MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 6. Falcon at a glance – read again MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 7. Falcon at a glance – write new record MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 8. Falcon at a glance – commit MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 9. Falcon at a glance – write complete MySQL Record Cache Server Serial Page Cache Log Windows Serial Log Database Files Tablespaces
- 10. Falcon history Origin Transactional SQL Engine for Web App Environment Bought by MySQL in 2006 MVCC Consistent Read Verisons control write access Memory only – no steal Indexes and data separate Data encoded on disk and in memory Fine grained multi-threading
- 11. Falcon Goals circa 2006 Exploit large memory for more than just a bigger cache Use threads and processors for data migration Eliminate tradeoffs, minimize tuning Scale gracefully to very heavy loads Support web applications
- 12. Web application characteristics Large archive of data Smaller active set High read:write ratio Uneven, bursty activity
- 13. What we did instead Enforce limit on record cache size Respond to simple atypical loads Autocommit single record access Repeat “insert ... select” Single pass read of large data set Challenge InnoDB on DBT2 Large working set Continuous heavy load Hired the world's most vicious test designer
- 14. Record Cache Record Cache contains: Committed records with no versions
- 15. Record Cache Record Cache contains: Committed records with no versions New, uncommitted records
- 16. Record Cache Record Cache contains: Committed records with no versions New, uncommitted records Records with multiple versions
- 17. Record Cache cleanup – step 1 Cleanup old committed single version records Scavenger Runs on schedule or demand Removes oldest mature records Settable limits – start and stop
- 18. Record Cache Cleanup – step 2 Clean out record versions too old to be useful Prune Remove old, unneeded versions
- 19. Record Cache Cleanup – step 3 Clean up a cache full of new records Chill Copy new record data to log Done by transaction thread Settable start size
- 20. Record Cache Cleanup – step 4 Clean up multiple versions of a single record created by a single transaction Remove intermediate versions Created by a single transaction Rolled back to save point Repeated updates
- 21. Record Cache Cleanup – step 5 Clean up records with multiple versions, still potentially visible Backlog Copy entire record tree to disk Expensive Not yet working
- 22. Simple, atypical loads Challenge: Autocommit single record access Record cache is useless Record encoding is useless Transaction creation / destruction is too expensive Response: Reuse read only transactions Result: Multi-threaded bookkeeping nightmare
- 23. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records
- 24. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records First solution Scavenge old records Chill new record data
- 25. Simple, atypical loads Challenge: Repeat “insert ... select...” Fill cache with old and new records First solution Scavenge old records Chill new records Second solution Move the records headers out Also helps index creation
- 26. Simple, atypical loads Single pass read of large data set Read more records than Read them over and over Caches are useless Encoding is overhead Response: Make encoding optional?
- 27. Challenge InnoDB on DBT2 Initial results were not encouraging (2007) 30000 25000 20000 Transactions Falcon2007 15000 InnoDB2007 10000 5000 0 10 20 50 100 150 200 Connections
- 28. Challenge InnoDB on DBT2 But Falcon has improved a lot since April 2007 30000 25000 20000 Transactions Falcon2007 15000 InnoDB2007 Falcon2009 10000 5000 0 10 20 50 100 150 200 Connections
- 29. Challenge InnoDB on DBT2 So did InnoDB 30000 25000 20000 Transactions Falcon2007 InnoDB2007 15000 Falcon2009 InnoDB2009 10000 5000 0 10 20 50 100 150 200 Connections
- 30. Bug trends
- 31. Multi-threading Databases are a natural fit for multi-threading Connections Gophers Scavenger Disk reader/writer Except for shared structures Locking blocks parallel operations Challenge – sharing without locking
- 32. Multi-threading Non-locking operation Purge old record versions
- 33. Multi-threading Non-locking operation Purge old record versions
- 34. Multi-threading Locking operation Remove intermediate versions
- 35. Multi-threading Locking operation Remove intermediate versions What granularity of lock?
- 36. Multi-threading – Lock granularity One per record: Too many interlocked instructions One per record group: Thread reading one record prevents scavenge of another No answer is right – more options?
- 37. Cycle locking – read record chain Before starting to read a record chain, get a shared lock on a “cycle” Cycle 1 = 3 Cycle 2 shared inactive Transaction A Transaction B Transaction C
- 38. Cycle locking – clean a record chain Before starting to read a record chain, get a shared lock on a “cycle” Cycle 1 = 4 Cycle 2 shared inactive Transaction A active in Cycle 1 Transaction B active in Cycle 1 Transaction C active in Cycle 1 Scavenger unlinks versions from record chain and links them to a “to be deleted” list.
- 39. Cycle locking – records relinked Cycle 1 = 1 Cycle 2 shared inactive Transaction A releases lock Transaction B releases lock Transaction C still active Scavenger releases lock
- 40. Cycle locking – swap cycles New access locks cycle 2 Cycle 1 = 1 Cycle 2 = 1 shared shared Transaction C holds Cycle 1 lock Cycle Manager requests exclusive on Cycle 1 (pumps cycle) Transaction A acquires Cycle 2 lock
- 41. Cycle locking – cleanup phase Cycle 1 = 0 Cycle 2 = 2 shared shared exclusive Transaction C releases lock Transaction B acquires Cycle 2 lock Cycle manager exclusive Cycle 1
- 42. Cycle locking – cleanup complete Cycle 1 Cycle 2 = 2 exclusive shared Transaction C acquires Cycle 2 lock Cycle manager exclusive Cycle 1 Remove unlinked, unloved, old versions When cleanup is done, Cycle manager releases cycle 1
- 43. Questions