SlideShare a Scribd company logo

Fusion-io and MySQL at Craigslist

Download as PPTX, PDF
J
Jeremy Zawodny

This document discusses Craigslist's migration from older MySQL database servers to new servers equipped with Fusion-io SSDs. It describes Craigslist's high database load of over 100 million postings and 1 billion daily page views. The migration involved replacing 14 older, less performant servers with just 3 new servers using Fusion-io SSDs. This reduced total power usage from 4,500 watts to 570 watts while greatly increasing I/O performance and reducing query response times.

Technology
1 of 14
Downloaded 157 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Fusion-io and MySQL 5.5at CraigslistJeremy ZawodnyJeremy@Zawodny.comjzawodn@craigslist.org
Story Time(but first, some architectureand numbers)
Some Numbers~100,000,000 postings in live databaseOver 1,000,000,000 page views dailyHigh churn rate (avg lifetime ~14 days)~350-500GB on diskMySQL 5.5.x and InnoDB CompressionUsed to be ~100-150GB larger (or more!)All records touched multiple times98% of queries are OLTP
The Posting CacheWeb Server Tier(apache/mod_perl)Posting Cache Tier(memcached + perl)Database Tier(MySQL)
The ProblemAdding more memcached nodesLots of cache misses initiallyMySQL boxes take a big query load(time passes)MySQL boxes pegged many hours later(time passes)Next day: WTF?!
Fire!Web Server Tier(apache/mod_perl)We were sending nearly 30% of requests all the way back to the DB tier instead of the normal 2-5%Posting Cache Tier(memcached + perl)Database Tier(MySQL)
SolutionLet’s put the New Hardware in the poolAdd 4 machinesAnd it still sucked…The 4 were fast but only took ~20% of the hitsRemove all the Old HardwareRemove 14 of 18 machinesSounds totally sane, right?
Old Hardware3 years old3U, Dual AMD 2218 HE32GB RAM16 15k RPM SAS disksRAID-10~2,000 iops/sec~325 watts
New HardwareHP DL-380G6Intel Xeon X5570Dual Proc, Quad Core, HT16 “cores”Dual Fusion-io 640GB SLCSoftware RAID-0~80,000 iops/sec (conservative)~200 watts
Before and AfterLoad AverageI/O Capacity of Data DisksNight and Day!Old boxes return to “steady state”
This Chart Should be GreenAverage power for Fusion-io equipped server: ~200 watts.It was closer to 160 when replicating but not serving traffic.
Fusion-io FTWCan you tell when this machine started getting live traffic?OLTP means disk matters WAY more than CPUInto the fire!
The NumbersOld: 2,000iops / 325W = 6.15 iops/wattNew: 40,000iops / 200W = 200 iops/wattConservatively assumes a lot of degradation33-66x performance/wattBut let’s just call it 50x
EpilogueA week later, we re-purposed 1 Fusion-io boxThe cache eventually did fillPoor slab size configuration had been causing early expiration of cached objects14 “old” servers: 4,500 watts28,000 iops/sec capacity3 “new” servers: 570 watts240,000+ iops/sec capacityWhat to do with 10+ spare “db class” boxes?

More Related Content

ODP
MySQL And Search At Craigslist
Jeremy Zawodny
 
KEY
Sphinx at Craigslist in 2012
Jeremy Zawodny
 
PPTX
Lessons Learned Migrating 2+ Billion Documents at Craigslist
Jeremy Zawodny
 
PDF
Realtime Search Infrastructure at Craigslist (OpenWest 2014)
Jeremy Zawodny
 
KEY
Living with SQL and NoSQL at craigslist, a Pragmatic Approach
Jeremy Zawodny
 
PDF
Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger
MongoDB
 
PPTX
Understanding and tuning WiredTiger, the new high performance database engine...
Ontico
 
PDF
Real time fulltext search with sphinx
Adrian Nuta
 
MySQL And Search At Craigslist
Jeremy Zawodny
 
Sphinx at Craigslist in 2012
Jeremy Zawodny
 
Lessons Learned Migrating 2+ Billion Documents at Craigslist
Jeremy Zawodny
 
Realtime Search Infrastructure at Craigslist (OpenWest 2014)
Jeremy Zawodny
 
Living with SQL and NoSQL at craigslist, a Pragmatic Approach
Jeremy Zawodny
 
Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger
MongoDB
 
Understanding and tuning WiredTiger, the new high performance database engine...
Ontico
 
Real time fulltext search with sphinx
Adrian Nuta
 

What's hot (20)

PPTX
Managing 50K+ Redis Databases Over 4 Public Clouds ... with a Tiny Devops Team
Redis Labs
 
PDF
Optimizing MongoDB: Lessons Learned at Localytics
andrew311
 
PDF
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Ontico
 
PDF
«Scrapy internals» Александр Сибиряков, Scrapinghub
it-people
 
PPTX
Introduction to Redis
Arnab Mitra
 
PDF
MongoUK - Approaching 1 billion documents with MongoDB1 Billion Documents
Boxed Ice
 
PDF
Webinar - Approaching 1 billion documents with MongoDB
Boxed Ice
 
PDF
Using Sphinx for Search in PHP
Mike Lively
 
PDF
Boosting Machine Learning with Redis Modules and Spark
Dvir Volk
 
KEY
MongoFr : MongoDB as a log Collector
Pierre Baillet
 
PDF
企業・業界情報プラットフォームSPEEDAにおけるElasticsearchの活用
Akira Kitauchi
 
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
PPTX
A simple introduction to redis
Zhichao Liang
 
PDF
MyRocks Deep Dive
Yoshinori Matsunobu
 
PPTX
Ops Jumpstart: Admin 101
MongoDB
 
PDF
Fluentd - Flexible, Stable, Scalable
Shu Ting Tseng
 
PPTX
Webinar Back to Basics 3 - Introduzione ai Replica Set
MongoDB
 
PDF
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
Waldemar Neto
 
PDF
PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
Ontico
 
PDF
Mongodb
Scott Motte
 
Managing 50K+ Redis Databases Over 4 Public Clouds ... with a Tiny Devops Team
Redis Labs
 
Optimizing MongoDB: Lessons Learned at Localytics
andrew311
 
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Ontico
 
«Scrapy internals» Александр Сибиряков, Scrapinghub
it-people
 
Introduction to Redis
Arnab Mitra
 
MongoUK - Approaching 1 billion documents with MongoDB1 Billion Documents
Boxed Ice
 
Webinar - Approaching 1 billion documents with MongoDB
Boxed Ice
 
Using Sphinx for Search in PHP
Mike Lively
 
Boosting Machine Learning with Redis Modules and Spark
Dvir Volk
 
MongoFr : MongoDB as a log Collector
Pierre Baillet
 
企業・業界情報プラットフォームSPEEDAにおけるElasticsearchの活用
Akira Kitauchi
 
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
A simple introduction to redis
Zhichao Liang
 
MyRocks Deep Dive
Yoshinori Matsunobu
 
Ops Jumpstart: Admin 101
MongoDB
 
Fluentd - Flexible, Stable, Scalable
Shu Ting Tseng
 
Webinar Back to Basics 3 - Introduzione ai Replica Set
MongoDB
 
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
Waldemar Neto
 
PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
Ontico
 
Mongodb
Scott Motte
 
Ad

Viewers also liked (10)

PPT
SphinxSearch
Przemyslaw Wroblewski
 
PDF
Managing Big Data with MySQL
mwasaha mwagambo
 
PPTX
Sphinx - High performance full-text search for MySQL
Nguyen Van Vuong
 
PPTX
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
PPTX
MySQL Performance Tips & Best Practices
Isaac Mosquera
 
PPT
Fast querying indexing for performance (4)
MongoDB
 
PDF
MySQL Performance Tuning: Top 10 Tips
OSSCube
 
PDF
How to Design Indexes, Really
Karwin Software Solutions LLC
 
PDF
10 SQL Tricks that You Didn't Think Were Possible
Lukas Eder
 
PPTX
Indexing with MongoDB
MongoDB
 
SphinxSearch
Przemyslaw Wroblewski
 
Managing Big Data with MySQL
mwasaha mwagambo
 
Sphinx - High performance full-text search for MySQL
Nguyen Van Vuong
 
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
MySQL Performance Tips & Best Practices
Isaac Mosquera
 
Fast querying indexing for performance (4)
MongoDB
 
MySQL Performance Tuning: Top 10 Tips
OSSCube
 
How to Design Indexes, Really
Karwin Software Solutions LLC
 
10 SQL Tricks that You Didn't Think Were Possible
Lukas Eder
 
Indexing with MongoDB
MongoDB
 
Ad

Similar to Fusion-io and MySQL at Craigslist (20)

PPT
Mysql talk
LogicMonitor
 
PPT
Scaling With Postgres
Robert Treat
 
PDF
Answering the Database Scale Out Problem with PCI SSDs
answers
 
PDF
Fusion-IO - Building a High Performance and Reliable VSAN Environment
VMUG IT
 
PDF
Pluk2013 bodybuilding ratheesh
Ratheesh Kaniyala
 
PPTX
What every data programmer needs to know about disks
iammutex
 
PDF
MySQL Performance Tuning London Meetup June 2017
Ivan Zoratti
 
PDF
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
Dave Stokes
 
PPTX
Nexcess Magento Imagine 2014 Performance Breakout
Nexcess.net LLC
 
PDF
The Smug Mug Tale
MySQLConference
 
PDF
My Sql And Search At Craigslist
MySQLConference
 
PDF
Evolution of DBA in the Cloud Era
Mydbops
 
ODP
The care and feeding of a MySQL database
Dave Stokes
 
PDF
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Severalnines
 
PDF
Database performance tuning for SSD based storage
Angelo Rajadurai
 
PDF
OSDC 2016 - Tuning Linux for your Database by Colin Charles
NETWAYS
 
PDF
Perconalive feb-2011-share
mdcallag
 
PDF
2008 MySQL Conference Recap
Chris Barber
 
PDF
Performance Whack A Mole
oscon2007
 
PDF
Presentation database on flash
xKinAnx
 
Mysql talk
LogicMonitor
 
Scaling With Postgres
Robert Treat
 
Answering the Database Scale Out Problem with PCI SSDs
answers
 
Fusion-IO - Building a High Performance and Reliable VSAN Environment
VMUG IT
 
Pluk2013 bodybuilding ratheesh
Ratheesh Kaniyala
 
What every data programmer needs to know about disks
iammutex
 
MySQL Performance Tuning London Meetup June 2017
Ivan Zoratti
 
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
Dave Stokes
 
Nexcess Magento Imagine 2014 Performance Breakout
Nexcess.net LLC
 
The Smug Mug Tale
MySQLConference
 
My Sql And Search At Craigslist
MySQLConference
 
Evolution of DBA in the Cloud Era
Mydbops
 
The care and feeding of a MySQL database
Dave Stokes
 
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Severalnines
 
Database performance tuning for SSD based storage
Angelo Rajadurai
 
OSDC 2016 - Tuning Linux for your Database by Colin Charles
NETWAYS
 
Perconalive feb-2011-share
mdcallag
 
2008 MySQL Conference Recap
Chris Barber
 
Performance Whack A Mole
oscon2007
 
Presentation database on flash
xKinAnx
 

Recently uploaded (20)

PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
A Presentation on Artificial Intelligence
memembruh336
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Teaching material agriculture food technology
LiaRayya
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 

Fusion-io and MySQL at Craigslist

  • 1. Fusion-io and MySQL 5.5at CraigslistJeremy [email protected]@craigslist.org
  • 2. Story Time(but first, some architectureand numbers)
  • 3. Some Numbers~100,000,000 postings in live databaseOver 1,000,000,000 page views dailyHigh churn rate (avg lifetime ~14 days)~350-500GB on diskMySQL 5.5.x and InnoDB CompressionUsed to be ~100-150GB larger (or more!)All records touched multiple times98% of queries are OLTP
  • 4. The Posting CacheWeb Server Tier(apache/mod_perl)Posting Cache Tier(memcached + perl)Database Tier(MySQL)
  • 5. The ProblemAdding more memcached nodesLots of cache misses initiallyMySQL boxes take a big query load(time passes)MySQL boxes pegged many hours later(time passes)Next day: WTF?!
  • 6. Fire!Web Server Tier(apache/mod_perl)We were sending nearly 30% of requests all the way back to the DB tier instead of the normal 2-5%Posting Cache Tier(memcached + perl)Database Tier(MySQL)
  • 7. SolutionLet’s put the New Hardware in the poolAdd 4 machinesAnd it still sucked…The 4 were fast but only took ~20% of the hitsRemove all the Old HardwareRemove 14 of 18 machinesSounds totally sane, right?
  • 8. Old Hardware3 years old3U, Dual AMD 2218 HE32GB RAM16 15k RPM SAS disksRAID-10~2,000 iops/sec~325 watts
  • 9. New HardwareHP DL-380G6Intel Xeon X5570Dual Proc, Quad Core, HT16 “cores”Dual Fusion-io 640GB SLCSoftware RAID-0~80,000 iops/sec (conservative)~200 watts
  • 10. Before and AfterLoad AverageI/O Capacity of Data DisksNight and Day!Old boxes return to “steady state”
  • 11. This Chart Should be GreenAverage power for Fusion-io equipped server: ~200 watts.It was closer to 160 when replicating but not serving traffic.
  • 12. Fusion-io FTWCan you tell when this machine started getting live traffic?OLTP means disk matters WAY more than CPUInto the fire!
  • 13. The NumbersOld: 2,000iops / 325W = 6.15 iops/wattNew: 40,000iops / 200W = 200 iops/wattConservatively assumes a lot of degradation33-66x performance/wattBut let’s just call it 50x
  • 14. EpilogueA week later, we re-purposed 1 Fusion-io boxThe cache eventually did fillPoor slab size configuration had been causing early expiration of cached objects14 “old” servers: 4,500 watts28,000 iops/sec capacity3 “new” servers: 570 watts240,000+ iops/sec capacityWhat to do with 10+ spare “db class” boxes?