MariaDB Performance Tuning and Optimization

MariaDB
Performance Tuning
and Optimization
Jonathan Day, MariaDB Solution Architect
Shree Nair, Monyog Product Manager

Agenda
• MariaDB Performance Tuning - Jon Day
– Common Principles and Best Practices
– Server Hardware and OS
– MariaDB Configuration Settings
– Database Design
– Monitoring and Query Tuning
• Monyog Presentation - Shree Nair
• Q&A

Why Tune?
● Efficient Use of Hardware Resources
● Best Performance for Users
● Avoid Outages Due to Server Slowness
● Extra Capacity
○ Be Prepared for Application Development Requirements
○ Allow for Unexpected Traffic Spikes or Other Changes in
Demand

Performance
Common Principles and Best Practices

Tuning Routine - When to Tune
• Tune from Start of the Application Lifecycle
– Start Early to Ensure Schema is Well Constructed
– Test Queries on Real Data — Watch for Bottlenecks
– Over Tuning without Production Data or Traffic is Counter Productive
• Conduct Periodic Reviews of Production Systems
– Watch for Schema, Query and Significant Changes
– Check Carefully New Application Features
– Monitor System Resources — Disk, Memory, Network, CPU
– Be Proactive in Solving Problems

Tuning Routine - When Not to Tune
• Identify Objectives in Advance
– Adhere to Objectives
• Decide on Best Use of Resources
– Tune Appropriately
– Don't Excessively Tune
• Be Aware of Data Integrity
– Where is Speed Most Important?
– Where is Integrity Most Important?
– Adhere to these Boundaries

The my.cnf configuration file
• Change one setting at a time...
– This is the only way to determine if a change is beneficial.
• Most settings can be changed at runtime with SET GLOBAL.
– It is very handy and it allows you to quickly revert the change if needed.
– To make the change permanent, you need to update the configuration file.
• If a change in the configuration is not visible even after a MariaDB restart…
– Did you use the correct configuration file?
– Did you put the setting in the right section?
•Normally the [mysqld] section for these settings

The my.cnf configuration file
• The server refuses to start after a change:
– Did you use the correct units?
•For instance, innodb_buffer_pool_size should be set in bytes
while max_connection is dimensionless.
• Do not allow duplicate settings in the configuration file.
– If you want to keep track of the changes, use version control.
• Don’t do naive math, like “my new server has 2x RAM, I’ll just make all the values
2x the previous ones”.

Performance
Server Hardware and OS Tuning

Server Hardware
• One Service per Server is Ideal to Prevent Contention
– Have the database server be only a database server etc.
• More CPU Cores is generally Good
• More Disk is usually Better
– Large Datasets, Fast Disks are Ideal
• More RAM is usually Best — Traffic Dependent
– More of Dataset in Memory, Fewer Slow Disk Operations

OS Settings
Linux Settings
•Swappiness
○ Value for propensity of the OS to swap
to disk
○ Defaults are usually 60
○ Commonly set low to 10 or so (not 0)
•Noatime
○ Mount disks with this option
○ Turns off writing of access time to disk
with every file access
○ Without this option every read becomes
an additional write

Which Filesystem to use?
• The short answer is the best filesystem for a MariaDB server is ext4,
XFS, or Btrfs.
All are solid enterprise journaling filesystems that scale well from small
to very large files and very large storage volumes.
Filesystems XFS Ext4 Btrfs
Maximum filesystem size 8EB 1EB 16EB
Maximum file size 8EB 16TB 16EB

Performance
MariaDB Configuration Settings

Configuration Settings
innodb_buffer_pool_size
•The first setting to update
•The buffer pool is where data and indexes
are cached
• Utilize memory for read operations rather
than disk
•80% RAM rule of thumb
•Typical values are
✓ 5-6GB (8GB RAM)
✓ 20-25GB (32GB RAM)
✓ 100-120GB (128GB RAM)

query_cache_size
● Query cache is a well known bottleneck
● Consider setting query_cache_size = 0
● Use other ways to speed up read
queries:
○ Good indexing
○ Adding replicas to spread the read
load

innodb_log_file_size
● Size of the redo logs - 25 to 50% of
innodb_buffer_pool usually
recommended
● Redo logs are used to make sure writes
are fast and durable and also during
crash recovery
● Larger log files can lead to slower
recovery in the event of a server crash
● But! Larger log files also reduce the
number of checkpoints needed and
reduce disk I/O

innodb_file_per_table
● Each .ibd file represents a tablespace of its
own.
● Database operations such as “TRUNCATE”
can be completed faster and you may also
reclaim unused space when dropping or
truncating a database table.
● Allows some of the database tables to be
kept in separate storage device. This can
greatly improve the I/O load on your disks.

Disable MySQL Reverse
DNS Lookups
● MariaDB performs a DNS lookup of the
user’s IP address and Hostname with
connection
● The IP address is checked by resolving it to a
host name. The hostname is then resolved to
an IP to verify
● This allows DNS issues to cause delays
● You can disable and use IP addresses only
○ skip-name-resolve under [mysqld] in
my.cnf

max_connections •‘Too many connections’ error?
•Using a connection pool at the application
level or a thread pool at the MariaDB level
can help

Check for MySQL idle
Connections
● Idle connections consume resources and
should be interrupted or refreshed when
possible.
● Idle connections are in “sleep” state and
usually stay that way for long period of time.
● To look for idled connections:
● # mysqladmin processlist -u root -p | grep
“Sleep”
● You can check the code for the cause if many
idled
● You can also change the wait_timeout value

thread_cache_size
● The thread_cache_size directive sets the amount of
threads that your server should cache.
● To find the thread cache hit rate, you can use the
following technique:
○ show status like 'Threads_created';
○ show status like 'Connections';
● calculate the thread cache hit rate percentage:
○ 100 - ((Threads_created / Connections) * 100)
● Dynamically set to a new value:
○ set global thread_cache_size = 16;

memory parameters
● MariaDB uses temporary tables when
processing complex queries involving joins
and sorting
● The default size of a temporary table is very
small
○ The size is configured in your my.cnf:
tmp-table-size = 1G
max-heap-table-size = 1G
● Both should have the same size and will help
prevent disk writes
● A rule of thumb is giving 64Mb for every GB
of RAM on the server

Buffer Sizes
● join buffer size
○ used to process joins – but only full
joins on which no keys are possible
● sort buffer size
○ Sort buffer size is used to sort data.
○ The system status variable
sort_merge_passes will indicates need
to increase
○ This variable should be as low as
possible.
● These buffers are allocated per connection
and play a significant role in the
performance of the system.

max_allowed_packet
● MariaDB splits data into packets. Usually a
single packet is considered a row that is sent
to a client.
● The max_allowed_packet directive defines
the maximum size of packet that can be sent.
● Setting this value too low can cause a query
to stall and you will receive an error in your
error log.
● It is recommended to set the value to the
size of your largest packet.
○ Some suggest 11 times the largest BLOB

Database Design
Choosing Data Types
● Use Appropriate Data Type (INT for
Numbers, VARCHAR)
● Use Smallest Useful Type
● Variable Length Fields are often Padded
● Use NOT NULL, where Practical
○ A NULL field uses slightly More
Disk and Memory (Depends on
Storage Engine)
● Use PROCEDURE ANALYSE( )

Database Design
Keys and Indexes
● Make PRIMARY KEY as Small as Practical
○ InnoDB Stores Copy of PRIMARY KEY
with each Index Entry
● Secondary Indexes Grow with PRIMARY
KEY
● Use Surrogate PRIMARY KEY , Not Natural
Ones
● Consider Partial (Prefix) Index for String
Indexes

Advantages
• Included in Backup and Replication
• Guaranteed to be Consistent with Other
Data
Disadvantages
• Database gets Too Large
• Sometimes Best to Store Files in the File
System and the Filename and Path in Table
Database Design: File Storage in BLOB

Database Design
Tuning Tables Overall
● Minimize Table Size on Disk and in
Memory
● Archive Table Data if Possible and
Appropriate
● Remove Duplicate or Unused Indexes
● Use Appropriate Data Types — Smaller
is Better
● Consider Sharding Large Tables across
Multiple Servers

Normalizing
• Makes a Data Set as Smaller by
Eliminating Redundant Data
• Ensures Data Set Consistency and
Integrity
• Often Improves Concurrency by Reducing
Locking
• Increases Number of Tables and
Associated Maintenance
Denormalizing
• Complete Normalization can Slow Queries
• More Complex JOIN Queries Harder to
Maintain
• Denormalization Adds Redundant Data to
Schema
• Write Queries become More Complex or
Numerous as Multiple Locations must be
Maintained
• It's Easier to Normalize First, Then
Denormalize when Appropriate
Database Design: Normalization of Tables

Storage Engines
● XtraDB is the best choice in the majority of cases. It is a performance-enhanced fork of InnoDB and is the
MariaDB default engine until MariaDB 10.1.
● InnoDB is a good general transactional storage engine. It is the default MySQL storage engine, and default
MariaDB 10.2 storage engine, but in earlier releases XtraDB is a performance enhanced fork of InnoDB, and is
usually preferred.
● Aria, MariaDB's more modern improvement on MyISAM, has a small footprint and allows for easy copying
between systems.
● MyISAM has a small footprint and allows for easy copying between systems. MyISAM is MySQL's oldest storage
engine. There is usually little reason to use it except for legacy purposes. Aria is MariaDB's more modern
improvement.
● Spider uses partitioning to provide data sharding through multiple servers.
● ColumnStore utilizes a massively parallel distributed data architecture and is designed for big data scaling to
process petabytes of
● MyRocks enables greater compression than InnoDB, as well as less write amplification giving better endurance of
flash storage and improving overall throughput. (Currently Alpha in MariaDB 10.2)

Performance
Monitoring and Query Tuning

Enable Slow query Logs
Logging slow queries can help you determine issues with your database and help you debug them. This can be easily enabled by
adding the following values in your my.cnf configuration file:
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1
The first directive enables the logging of slow queries, while the second one tells MariaDB where to store the actual log file. Use
long_query_time to define the amount of time that is considered long for a MariaDB query to be completed.

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
Query Analysis ● Use the Slow Log to find Problem Queries
● Use mysqldumpslow Utility for Manageable
Reports
● Use EXPLAIN to see how MariaDB Executes a
Troublesome Query and if Indexes are Used
● Use EXPLAIN EXTENDED and SHOW
WARNINGS to see how MariaDB Rearranges a
Query before Execution
EXPLAIN SELECT * FROM employees
WHERE MONTH(birth_date) = 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
Query Tuning Overview
● Try Not to Query Tune on Production Server
● Use Test Server with same Hardware and OS
● Use Copy of Production Data Set
● Query Frequency is as Important as Query
Speed
○ Moderately Slow Queries are often a Bigger
Problem than a Rarely Run Very Slow Query

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
Indexing
● Indexes Improve Read Performance
● Without Index, MariaDB Must Read Every Row —
Full Table Scan
● With Index, MariaDB can jump to Requested
Rows
● Reduced I/O and Improving Performance
● Index Increase cost of Writes
● Find Balance
● Index for Speed, but Avoid Indexing Excessively
or Arbitrarily
● Remove Dead or Redundant Indexes

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
Index Size
● Keep Indexes as Small as Practical
○ Faster since More Likely to Fit in Memory
○ Rebuilds Faster after Writes
○ PRIMARY KEY should be Minimum Useful
Size
● Use Partial Prefix Indexes for String Columns
○ May Slow Searches Slightly, but Reduce
Index Size
● Use Index Cardinality (Uniqueness Measure)
Only If Necessary — Re-evaluate as Data Grows
○ Low Cardinality Indicates many Duplicates
○ High Cardinality is More Useful

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
Tools & Statistics
● Identify Accurately and Carefully Trouble Spots
○ Guessing is Rarely Useful
● Gather Performance Stats with MariaDB and OS
Tools
○ SHOW Statements
○ PERFORMANCE_SCHEMA
○ CPU, Disk, Network, Memory, & Swap Stats
● Retain Snapshots of Multiple Stats
○ Data from a Single Point Shows very Little
● Automate the Collection of Stats into Logs
○ Can be Useful for Emergency Tuning

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
SHOW PROCESSLIST
● Snapshot of mysqld Activity
● mysqld is Multi-Threaded, One Thread per
○ Client Connection (i.e., query, transaction) —
a "process" is a "thread"
● Accumulate SHOW PROCESSLIST Snapshots to
build History of Thread Activities

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
SHOW STATUS
Global or Session
● Returns List of Internal Counters
● GLOBAL for System-Wide Status — Since
Start-Up
● SESSION for Local to Client Connection
● FLUSH STATUS Resets Local Counters
● Monitor Changes to Counters to Identify Hot
Spots
● Collect Periodically Status Snapshots to Profile
Traffic

EXPLAIN SELECT *
FROM employees
WHERE
MONTH(birth_date)
= 8 G
id: 1
select_type: SIMPLE
table: employees
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 299587
Extra: Using where
PERFORMANCE_SCHEMA
● Similar to INFORMATION_SCHEMA , but
Performance Tuning
● Monitors MariaDB Server Events
● Function Calls, Operating System Waits, Internal
Mutexes, I/O Calls
● Detailed Query Execution Stages (Parsing,
Statistics, Sorting)
● Some Features Storage Engine Specific
● Monitoring Lightweight and Requires No
Dedicated Thread
● Designed to be Used Iteratively with Successive
Refinement

Monitoring Tools
Monyog - Agentless and Cost-effective MySQL monitoring tool
Box Anemometer - a MariaDB Slow Query Monitor. This tool is used to analyze slow query logs
collected from MariaDB instances to identify problematic queries
pt-query-digest - Analyzes MariaDB queries from logs, processlist, and tcpdump

Agentless and Cost-effective
MariaDB monitoring tool.

Monitoring databases should be easy
Avoid Visibility Gap Across Servers
Get a bird’s-eye view of your data tier, keep up with
the production demands and avoid the visibility gap.
Achieve Faster Issue Resolution Time
Uncover problematic queries and performance
problems in complex data layer.
Industry-Leading Replication Overview
View the replication hierarchy of servers along with
the details of each replicated server.

SQLITE
REPOSITORY
MYSQL DATA
& OS METRICS
COLLECTOR
DATA AGGREGATOR
MONYOG
WEB
CLIENT
TIMELY ALERTS VIA
MAIL (SMTP) & TRAPS (SNMP)
MONYOG
MYSQL/
MARIADB
CLOUDBASED
MYSQL/
MARIADB
DATA, LOGS
OS METRICS
SFTP / MARIADB C-CONNECTOR MARIADB C-CONNECTOR / REST API / CLOUDWATCH API
DATA, LOGS
OS METRICS
Monyog Architecture
MYSQL DATA
& OS METRICS
COLLECTOR

Benefits of Monyog
•On-premise solution. Secure. Scalable.
•600+ monitors and advisors.
•Minuscule Performance Overhead.
•Saves tons of time on a daily basis.
•Avoids critical visibility gap across MySQL servers.
•Faster Issue Resolution Time.

Why Monyog is a need-to-have?
•Proactively prevents problems before customers notice.
•Ensures great performance of mission-critical applications.
•No deep database expertise needed to use Monyog.
•Makes MariaDB performance tuning easy.

Hardware requirement
• Small & medium size deployments
• Double core CPU with 4GB of memory
• 5GB of storage space per server registered with Monyog
• Large size deployments (>500 servers)
• 16 core CPU with 32GB of memory
• 1.5 TB of storage

How can you try out Monyog?
•Free trial available on www.monyog.com
•Monyog documentation - http://monyogkb.webyog.com/
•Included in a MariaDB Enterprise Subscription.
•Contact us at sales@webyog.com

Thank you
Jonathan.Day@mariadb.com.

MariaDB Performance Tuning and Optimization

More Related Content

What's hot (20)

Similar to MariaDB Performance Tuning and Optimization (20)

More from MariaDB plc (20)

Recently uploaded (20)

MariaDB Performance Tuning and Optimization