Yet Another Replication Tool: RubyRep
0 likes2,283 views
Denish Patel, a database architect at OmniTi, gave a presentation on the RubyRep database replication tool. RubyRep is an asynchronous master-master and master-slave replication tool that is easy to install, platform independent, and supports large objects and tables without primary keys. It was developed as an alternative to Slony because Slony has limitations such as requiring tables to have primary keys and not supporting large objects or master-master replication. The presentation covered how to install, configure, scan, sync, and replicate databases using RubyRep.
![Scan
• To scan all tables starting with ‘c’:
• rubyrep scan -c omniti.conf /^c/
• Scan with default options will produce output like this:
• clients 100% ......................... 0
• clients_details 100% ......................... 5
• Options
• -s, --summary[=detailed]
• -d, --detailed[=mode]
• full , keys, diff
• -b, --progress-bar[=length]
• -c, --config=CONFIG_FILE
• --help
Friday, March 26, 2010 11](https://image.slidesharecdn.com/rubyrepv2-100326211120-phpapp01/85/Yet-Another-Replication-Tool-RubyRep-11-320.jpg)
![Replicate
RR::Initializer::run do |config|config.left = {
:adapter => 'postgresql', # or 'mysql'
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:schema_search_path => 'omniti‘ }
config.right = {
:adapter => 'postgresql',
:database => 'omniti',
:username => 'dba',
:password => 'securepasswd',
:host => '127.0.0.1',
:port => '5484' }
config.options[:sync_conflict_handling] = :left_wins
config.options[:replication_conflict_handling] = :left_wins
config.add_table_options 'admin',
:sync_conflict_handling => :right_wins,
:replication_conflict_handling => :right_wins
config.include_tables /./
end
Friday, March 26, 2010 16](https://image.slidesharecdn.com/rubyrepv2-100326211120-phpapp01/85/Yet-Another-Replication-Tool-RubyRep-16-320.jpg)
![Master-Slave Config
config.include_tables /./ # regexp matching all tables in the database
config.options[:auto_key_limit] = 40
config.options[:adjust_sequences] = false
config.options[:sequence_increment] = 1
config.options[:replication_conflict_handling] = :left_wins
config.options[:logged_replication_events] = [
:ignored_changes,
:ignored_conflicts
]
end
Friday, March 26, 2010 17](https://image.slidesharecdn.com/rubyrepv2-100326211120-phpapp01/85/Yet-Another-Replication-Tool-RubyRep-17-320.jpg)
![Customize as needed ...
• Tables without PK
• config.options[:auto_key_limit] = 2
• config.add_table_options /_/, :auto_key_limit => 2
• config.include_tables 'articles_tags', :key => ['article_id', 'tag_id']
• Checks before and after
• :before_table_sync => "SET foreign_key_checks = 0"
• :after_table_sync => "SET foreign_key_checks = 1"
• Low bandwidth with RubyRep Proxy
• Benefits only Scan and Compare ; only difference is transferred
• :proxy_host => '172.16.1.5', , :proxy_port => '9876'
• Cascading replication
• :rep_prefix to use separate triggers and tables
Friday, March 26, 2010 18](https://image.slidesharecdn.com/rubyrepv2-100326211120-phpapp01/85/Yet-Another-Replication-Tool-RubyRep-18-320.jpg)
Yet Another Replication Tool: RubyRep
- 1. Yet Another Replication Tool RubyRep / Denish Patel Database Architect Friday, March 26, 2010 1
- 2. Who am I? • With OmniTi for more than 3 years • Manage high traffic database systems • Replication database system deployments • Not a core hacker of RubyRep • “Oh, We are hiring!!” • Contact : [email protected] Friday, March 26, 2010 2
- 3. Next 30 minutes .. • Replication • Various Tools • Slony? Why another tool ? • RubyRep • Install • Features • Examples • Tweaking replication policies Friday, March 26, 2010 3
- 4. Replication • Types • Synchronous • Semi-synchronous (O My!) • Asynchronous • Methods • Master - Slave • Master - Master • Needs • Horizontal Scaling • Standby Database • Backup Database Friday, March 26, 2010 4
- 5. Tools Program Type Method Based on PgCluster -II Synchronous M-M Shared Disk Slony-I Asynchronous M-S Trigger Bucardo Asynchronous M-M, M-S Trigger Londiste Asynchronous M-S Trigger Mammoth Asynchronous M-S Log RubyRep Asynchronous M-M, M-S Trigger Friday, March 26, 2010 5
- 6. Why not Slony? • Replicated tables MUST need PK or UK • Doesn’t support large objects • Doesn’t support synchronizing tables outside of replication • Limitations of version compatibility • Difficult to setup • Doesn’t support Master - Master • Difficult to monitor and manage Friday, March 26, 2010 6
- 7. RubyRep Toolset • Scan • Sync • Replicate Friday, March 26, 2010 7
- 8. RubyRep Key Features • Easy to install, setup, configure and monitor • Platform independent • Table design independent • Simple PK • Combined PK • No PK • Supports large objects i.e bytea, tsvector • Database Version Independent • Database independent • PostgreSQL • MySQL Friday, March 26, 2010 8
- 9. Install • Standard Ruby Version • Install Ruby Environment , Install DB drivers , Install RubyRep • Difficult to install • Slower performance • JRuby Version • Install JVM, Download and Extract gzip file • Easy to install • Faster performance Friday, March 26, 2010 9
- 10. Scan • Features • Compare “Left” and “Right” Databases/Tables • Customize batch size • Load on client server • Steps: • Generate config file • ./rubyrep generate omniti.conf • Configure • Run the Scan • ./rubyrep scan -c omniti.conf • Production Results • Users table with 60M rows and 45 columns • 3 hours • Running from monitor server Friday, March 26, 2010 10
- 11. Scan • To scan all tables starting with ‘c’: • rubyrep scan -c omniti.conf /^c/ • Scan with default options will produce output like this: • clients 100% ......................... 0 • clients_details 100% ......................... 5 • Options • -s, --summary[=detailed] • -d, --detailed[=mode] • full , keys, diff • -b, --progress-bar[=length] • -c, --config=CONFIG_FILE • --help Friday, March 26, 2010 11
- 12. Config RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.include_tables 'users,users_backup' config.include_tables /^o/ # regexp matches all tables starting with o end Friday, March 26, 2010 12
- 13. Sync • Compare and Sync • :sync_conflict_handling - :ignore (default), :left_wins, :right_wins • :sync_record_handling • :left_record_handling , :right_record_handling • :ignore, :delete, :insert(default) • Logging • :ignored_changes , :all_changes , • :ignored_conflicts , :all_conflicts • Others • :row_buffer_size , :commit_frequency • http://www.rubyrep.org/configuration.html Friday, March 26, 2010 13
- 14. Sync RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti’ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } # To match all tables except those ending with ‘backup’: config.include_tables /./ config.exclude_tables /backup$/ #exclude tables ending with backup end Friday, March 26, 2010 14
- 15. Replicate • Replicate Database/Tables • Global vs Table specific settings • rubyrep replicate -c omniti.conf • Setup necessary infrastructure tables • Setup necessary triggers for the replicated tables • Initial scan is executed and synched • Tweak sequences • Left -Odd , Right -Even for M-M replication • Default M-M replication setting • :replicate_record _handling = :replicate • Replication Conflict handling • :later_wins, :earlier_wins • Automatically Sync newly created table(Restart required) • Uninstall • rubyrep uninstall -c omniti.conf Friday, March 26, 2010 15
- 16. Replicate RR::Initializer::run do |config|config.left = { :adapter => 'postgresql', # or 'mysql' :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :schema_search_path => 'omniti‘ } config.right = { :adapter => 'postgresql', :database => 'omniti', :username => 'dba', :password => 'securepasswd', :host => '127.0.0.1', :port => '5484' } config.options[:sync_conflict_handling] = :left_wins config.options[:replication_conflict_handling] = :left_wins config.add_table_options 'admin', :sync_conflict_handling => :right_wins, :replication_conflict_handling => :right_wins config.include_tables /./ end Friday, March 26, 2010 16
- 17. Master-Slave Config config.include_tables /./ # regexp matching all tables in the database config.options[:auto_key_limit] = 40 config.options[:adjust_sequences] = false config.options[:sequence_increment] = 1 config.options[:replication_conflict_handling] = :left_wins config.options[:logged_replication_events] = [ :ignored_changes, :ignored_conflicts ] end Friday, March 26, 2010 17
- 18. Customize as needed ... • Tables without PK • config.options[:auto_key_limit] = 2 • config.add_table_options /_/, :auto_key_limit => 2 • config.include_tables 'articles_tags', :key => ['article_id', 'tag_id'] • Checks before and after • :before_table_sync => "SET foreign_key_checks = 0" • :after_table_sync => "SET foreign_key_checks = 1" • Low bandwidth with RubyRep Proxy • Benefits only Scan and Compare ; only difference is transferred • :proxy_host => '172.16.1.5', , :proxy_port => '9876' • Cascading replication • :rep_prefix to use separate triggers and tables Friday, March 26, 2010 18
- 19. References • Installation Notes • http://rubyforge.org/frs/?group_id=7932 • User Guide • http://www.rubyrep.org/documentation.html • API Documentation • http://rubyrep.rubyforge.org/ • Tutorials • http://www.rubyrep.org/tutorial.html • Blogs • http://denishjpatel.blogspot.com/2009/08/yet-another- postgresql-replication-tool.html Friday, March 26, 2010 19
- 20. Thanks • Arndt Lehmann • Thank you!! Friday, March 26, 2010 20
- 21. Questions ? Friday, March 26, 2010 21