SlideShare a Scribd company logo
Database Scalability
                   {Patterns}

                          / Robert Treat



Thursday, April 8, 2010
robert treat
                                 omniti
                               postgres
                            oracle - mysql
                          mssql - sqlite -nosql




Thursday, April 8, 2010
What are “Database Scalability Patterns?”




Thursday, April 8, 2010
Part Design Patterns



                                  Part Application Life-Cycle




Thursday, April 8, 2010
MyFirstDatabase




Thursday, April 8, 2010
Vertical Partitioning




Thursday, April 8, 2010
Vertical Scaling




Thursday, April 8, 2010
Vertical Scaling




Thursday, April 8, 2010
Federated Data Storage


               “sharding”                Horizontal Partitioning

                          Read Slaves
                                                Multi-Master
                   Horizontal Scaling




Thursday, April 8, 2010
Read Slaves / Master - Slave
                               Scale Read Load




Thursday, April 8, 2010
Read Slaves / Master - Slave
                               Scale Read Load

        • Typically
          • Full Copy of Data
             On Each Node
             • Asynchronous




Thursday, April 8, 2010
Read Slaves / Master - Slave
                               Scale Read Load

        • Typically                       • Consider
          • Full Copy of Data               • Partial Copy
             On Each Node                   • Synchronous
             • Asynchronous                 • Don’t use a RDBMS?




Thursday, April 8, 2010
Read Slaves / Master - Slave
                                Scale Read Load

        • Typically                        • Consider
          • Full Copy of Data                • Partial Copy
             On Each Node                    • Synchronous
             • Asynchronous                  • Don’t use a RDBMS?
                          Requires Application Changes



Thursday, April 8, 2010
Read Slaves / Master - Slave
                                Scale Read Load

        • Typically                        • Consider
          • Full Copy of Data                • Partial Copy
             On Each Node                    • Synchronous
             • Asynchronous                  • Don’t use a RDBMS?
                          Requires Application Changes
                                     “easy”

Thursday, April 8, 2010
Scaling Writes




                           “not easy”




Thursday, April 8, 2010
Multi-Master
                          many different ways to implement this,
                          few that actually work in production




Thursday, April 8, 2010
Multi-Master
                          many different ways to implement this,
                          few that actually work in production


                          write to any node, database syncs data




Thursday, April 8, 2010
Multi-Master
                          many different ways to implement this,
                          few that actually work in production


                          write to any node, database syncs data
                            can reduce cpu, doesn’t reduce i/o




Thursday, April 8, 2010
Multi-Master
                          many different ways to implement this,
                          few that actually work in production


                          write to any node, database syncs data
                            can reduce cpu, doesn’t reduce i/o


                                     failover solution
                                 not a scalability solution

Thursday, April 8, 2010
Horizontal Partitioning

    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed




Thursday, April 8, 2010
Horizontal Partitioning

    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed

    • You must separate dependencies
    in the app code first!




Thursday, April 8, 2010
Horizontal Partitioning

    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed

    • You must separate dependencies
    in the app code first!


                    Each node is a new instance of vertical scaling

Thursday, April 8, 2010
Horizontal Scaling

               • data split across servers based on algorithm
               • data dropped into buckets (multiple?)
               • someone must keep track of data, and provide
               lookup services




Thursday, April 8, 2010
Universal Truths of Scaling Databases

                     Vertical Scalability is Helpful for Every Pattern


                     Even in a horizontally scaled, fully distributed
                     database, the number of nodes needed is
                     affected by vertical scalability




Thursday, April 8, 2010
Universal Truths of Scaling Databases

                              New Nodes Are Never Free


                          • Add points of failure
                          • Add management costs
                          • Add complexity to architecture
                          • Add complexity to your app code


Thursday, April 8, 2010
MyFirstDB
                                     V
                           Vertical Partitioning
                                     V
                             Vertical Scaling
                                     V
                               Read Slaves
                                     V
                          Horizontal Partitioning



Thursday, April 8, 2010
plan for layered data sources
                          read / write connections in code
                          use schemas to separate services




Thursday, April 8, 2010
plan for layered data sources
                          read / write connections in code
                          use schemas to separate services

                                  use postgres :-)




Thursday, April 8, 2010
THANKS!

                            more:
                           xzilla.net
                          @robtreat2
                          omniti.com




Thursday, April 8, 2010

More Related Content

PDF
A Guide To PostgreSQL 9.0
PDF
What Ops Can Learn From Design
PDF
Intro to pl/PHP Oscon2007
PPT
Scaling With Postgres
PDF
Less Alarming Alerts - SRECon 2016
PDF
Managing Databases In A DevOps Environment 2016
PDF
Advanced WAL File Management With OmniPITR
PDF
Think_your_Postgres_backups_and_recovery_are_safe_lets_talk.pptx
A Guide To PostgreSQL 9.0
What Ops Can Learn From Design
Intro to pl/PHP Oscon2007
Scaling With Postgres
Less Alarming Alerts - SRECon 2016
Managing Databases In A DevOps Environment 2016
Advanced WAL File Management With OmniPITR
Think_your_Postgres_backups_and_recovery_are_safe_lets_talk.pptx

Similar to Database Scalability Patterns (20)

PDF
PPTX
NOSQL DATABASES UNIT-3 FOR ENGINEERING STUDENTS
PPTX
Handling Data in Mega Scale Systems
PDF
What every developer should know about database scalability, PyCon 2010
PPTX
Distribution Models.pptxgdfgdfgdfgfdgdfg
PPTX
UNIT II (1).pptx
PDF
PostgreSQL Scaling And Failover
PDF
Scaling Databases On The Cloud
PDF
Scaing databases on the cloud
PDF
Datastores
PDF
Persistence Smoothie: Blending SQL and NoSQL (RubyNation Edition)
PPTX
Tech Talk Series, Part 2: Why is sharding not smart to do in MySQL?
PDF
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
PPTX
CodeFutures - Scaling Your Database in the Cloud
PDF
Building Scale Free Applications with Hadoop and Cascading
PDF
John D. Rowell - Scaling heterogeneous systems on the cloud
PDF
Scaling Small App
PDF
Scaling with Postgres (Robert Treat)
PPT
Building A Scalable Architecture
PDF
20141206 4 q14_dataconference_i_am_your_db
NOSQL DATABASES UNIT-3 FOR ENGINEERING STUDENTS
Handling Data in Mega Scale Systems
What every developer should know about database scalability, PyCon 2010
Distribution Models.pptxgdfgdfgdfgfdgdfg
UNIT II (1).pptx
PostgreSQL Scaling And Failover
Scaling Databases On The Cloud
Scaing databases on the cloud
Datastores
Persistence Smoothie: Blending SQL and NoSQL (RubyNation Edition)
Tech Talk Series, Part 2: Why is sharding not smart to do in MySQL?
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
CodeFutures - Scaling Your Database in the Cloud
Building Scale Free Applications with Hadoop and Cascading
John D. Rowell - Scaling heterogeneous systems on the cloud
Scaling Small App
Scaling with Postgres (Robert Treat)
Building A Scalable Architecture
20141206 4 q14_dataconference_i_am_your_db
Ad

More from Robert Treat (20)

PDF
Advanced Int->Bigint Conversions
PDF
Explaining Explain
PDF
the-lost-art-of-plpgsql
PDF
Managing Chaos In Production: Testing vs Monitoring
PDF
Postgres 9.4 First Look
PDF
Less Alarming Alerts!
PDF
Past, Present, and Pachyderm - All Things Open - 2013
PDF
Big Bad "Upgraded" Postgres
PDF
Managing Databases In A DevOps Environment
PDF
The Essential PostgreSQL.conf
PDF
Pro Postgres 9
PDF
Scaling with Postgres (Highload++ 2010)
PDF
Intro to Postgres 9 Tutorial
PDF
Check Please!
PDF
Intro to Postgres 8.4 Tutorial
PDF
The Essential postgresql.conf
PDF
PostgreSQL Partitioning, PGCon 2007
ODP
Pro PostgreSQL, OSCon 2008
PDF
Database Anti Patterns
ODP
Pro PostgreSQL
Advanced Int->Bigint Conversions
Explaining Explain
the-lost-art-of-plpgsql
Managing Chaos In Production: Testing vs Monitoring
Postgres 9.4 First Look
Less Alarming Alerts!
Past, Present, and Pachyderm - All Things Open - 2013
Big Bad "Upgraded" Postgres
Managing Databases In A DevOps Environment
The Essential PostgreSQL.conf
Pro Postgres 9
Scaling with Postgres (Highload++ 2010)
Intro to Postgres 9 Tutorial
Check Please!
Intro to Postgres 8.4 Tutorial
The Essential postgresql.conf
PostgreSQL Partitioning, PGCon 2007
Pro PostgreSQL, OSCon 2008
Database Anti Patterns
Pro PostgreSQL
Ad

Recently uploaded (20)

PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PPTX
Comunidade Salesforce São Paulo - Desmistificando o Omnistudio (Vlocity)
PDF
Transforming Manufacturing operations through Intelligent Integrations
PDF
How Onsite IT Support Drives Business Efficiency, Security, and Growth.pdf
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Modernizing your data center with Dell and AMD
PDF
CIFDAQ's Market Wrap: Ethereum Leads, Bitcoin Lags, Institutions Shift
PDF
cuic standard and advanced reporting.pdf
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PPTX
Telecom Fraud Prevention Guide | Hyperlink InfoSystem
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
PDF
Empathic Computing: Creating Shared Understanding
PDF
HCSP-Presales-Campus Network Planning and Design V1.0 Training Material-Witho...
PDF
AI And Its Effect On The Evolving IT Sector In Australia - Elevate
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Advanced Soft Computing BINUS July 2025.pdf
PPTX
Cloud computing and distributed systems.
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Reach Out and Touch Someone: Haptics and Empathic Computing
Comunidade Salesforce São Paulo - Desmistificando o Omnistudio (Vlocity)
Transforming Manufacturing operations through Intelligent Integrations
How Onsite IT Support Drives Business Efficiency, Security, and Growth.pdf
Diabetes mellitus diagnosis method based random forest with bat algorithm
Modernizing your data center with Dell and AMD
CIFDAQ's Market Wrap: Ethereum Leads, Bitcoin Lags, Institutions Shift
cuic standard and advanced reporting.pdf
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Telecom Fraud Prevention Guide | Hyperlink InfoSystem
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
Empathic Computing: Creating Shared Understanding
HCSP-Presales-Campus Network Planning and Design V1.0 Training Material-Witho...
AI And Its Effect On The Evolving IT Sector In Australia - Elevate
Dropbox Q2 2025 Financial Results & Investor Presentation
MYSQL Presentation for SQL database connectivity
Advanced Soft Computing BINUS July 2025.pdf
Cloud computing and distributed systems.
GamePlan Trading System Review: Professional Trader's Honest Take

Database Scalability Patterns

  • 1. Database Scalability {Patterns} / Robert Treat Thursday, April 8, 2010
  • 2. robert treat omniti postgres oracle - mysql mssql - sqlite -nosql Thursday, April 8, 2010
  • 3. What are “Database Scalability Patterns?” Thursday, April 8, 2010
  • 4. Part Design Patterns Part Application Life-Cycle Thursday, April 8, 2010
  • 9. Federated Data Storage “sharding” Horizontal Partitioning Read Slaves Multi-Master Horizontal Scaling Thursday, April 8, 2010
  • 10. Read Slaves / Master - Slave Scale Read Load Thursday, April 8, 2010
  • 11. Read Slaves / Master - Slave Scale Read Load • Typically • Full Copy of Data On Each Node • Asynchronous Thursday, April 8, 2010
  • 12. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Thursday, April 8, 2010
  • 13. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes Thursday, April 8, 2010
  • 14. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes “easy” Thursday, April 8, 2010
  • 15. Scaling Writes “not easy” Thursday, April 8, 2010
  • 16. Multi-Master many different ways to implement this, few that actually work in production Thursday, April 8, 2010
  • 17. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data Thursday, April 8, 2010
  • 18. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o Thursday, April 8, 2010
  • 19. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o failover solution not a scalability solution Thursday, April 8, 2010
  • 20. Horizontal Partitioning • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed Thursday, April 8, 2010
  • 21. Horizontal Partitioning • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed • You must separate dependencies in the app code first! Thursday, April 8, 2010
  • 22. Horizontal Partitioning • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed • You must separate dependencies in the app code first! Each node is a new instance of vertical scaling Thursday, April 8, 2010
  • 23. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) • someone must keep track of data, and provide lookup services Thursday, April 8, 2010
  • 24. Universal Truths of Scaling Databases Vertical Scalability is Helpful for Every Pattern Even in a horizontally scaled, fully distributed database, the number of nodes needed is affected by vertical scalability Thursday, April 8, 2010
  • 25. Universal Truths of Scaling Databases New Nodes Are Never Free • Add points of failure • Add management costs • Add complexity to architecture • Add complexity to your app code Thursday, April 8, 2010
  • 26. MyFirstDB V Vertical Partitioning V Vertical Scaling V Read Slaves V Horizontal Partitioning Thursday, April 8, 2010
  • 27. plan for layered data sources read / write connections in code use schemas to separate services Thursday, April 8, 2010
  • 28. plan for layered data sources read / write connections in code use schemas to separate services use postgres :-) Thursday, April 8, 2010
  • 29. THANKS! more: xzilla.net @robtreat2 omniti.com Thursday, April 8, 2010