The Magic of Tuning in PostgreSQL
Download as PPTX, PDF10 likes8,463 views
The document discusses the importance of tuning and connection management in PostgreSQL for effective scaling, highlighting benchmark tests that measure performance under various configurations. Key learnings indicate that proper tuning can significantly improve performance, but scaling still degrades under high concurrency. Additionally, it emphasizes that a well-designed application with an optimized database can outperform a poorly-tuned database on a higher-end server.
![Low Configuration PostgreSQL server scaling with
Connection Pooling
Database Parameters:
• No of Processors: 1
• Memory: 3GB,
• shared_buffer: 32MB,
• effective_cache_size: 128MB
• work_mem=1MB
Key Learning:
Test 1
- The performance degrades once the load increases beyond a point [sounds familiar?]
Test 2
- Session Level Connection pooling with max 50 connections in Pool
- Connections aquired as needed so not much difference than first test
Test 3
- Transaction Level Connection Pooling with 25 connections
- Does well as concurrency increases [scaling with growth!!!]
Inconsistent/low and
highs can be attributed
to resources contention
during autovacuum](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-6-320.jpg)
![PostgreSQL Performs better after Tuning
Database Parameters:
• No of Processors: 1
• Memory: 3GB,
• shared_buffer: 32MB/256MB,
• effective_cache_size: 128MB/750MB
• work_mem=1MB
Key Learning:
Test 1: Basic Setup
- The performance degrades once the load increases beyond a point [sounds familiar?]
Test 2: Tuned postgresql.conf for shared_buffer and effective_cache_size
- Up to 19% gain over basic configuration
Test 3: Tuned other parameters in postgresql.conf
- Up to 21% gain
- The occasional performance troughs because of vacuum operation is
not there anymore
Still the performance is
not scalable in any of the
3 cases and degrades
with concurrency [25-
30% loss]](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-8-320.jpg)
![Combining the Effect of Connection Pooling and
Tuning
Database Parameters:
• No of Processors: 1
• Memory: 3GB,
• shared_buffer: 256MB,
• effective_cache_size: 750MB
• work_mem=1MB
Key Learning:
Test 1: Basic Setup with tuned shared_buffer and effective_cache_size
- The performance degrades once the load increases beyond a point [sounds familiar?]
Test 2: Tuned other parameters in postgresql.conf
- Performance gain of up to 14%
- The performance continues to degrade [by a margin of 26%] at higher concurrency
Test 3: Transaction Level Connection Pooling
- Performance is more consistent
- Degradation/loss in performance at peak concurrency is quite less [14% only]
Finally we get
a consistently
scaling
Database
Setup](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-9-320.jpg)
![Tuning and proper Transaction Management
enhances the benefit of CPU addition
Database Parameters:
• No of Processors: 2
• Memory: 3GB,
• shared_buffer: 256MB,
• effective_cache_size: 750MB
• work_mem=1MB
Key Learning:
Test 1: Basic Setup with tuned shared_buffer and effective_cache_size
- The performance degrades once the load increases beyond a point
Test 2: Tuned other parameters in postgresql.conf
- Performance gain of 36% at Peak load
- The performance continues to degrade [by a margin of 26%] at higher concurrency
Test 3: Transaction Level Connection Pooling
- Performance is more consistent
- Improvement caused by tuning is more prominent with boosted computing
[1CPU- 46% | 2CPU- 80%]
Even after boosting
computing power,
to get best of
resources, tuning
and connection
pooling plays a
major role](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-10-320.jpg)
![Tuning and proper Transaction Management
enhances the benefit of Memory Upgrade
Database Parameters:
• No of Processors: 2
• Memory: 4.5GB,
• shared_buffer: 750MB,
• effective_cache_size: 2GB
• work_mem=1MB
Key Learning:
Test 1: Basic Setup with tuned shared_buffer[750MB] and effective_cache_size[2GB]
- The performance degrades once the load increases beyond a point
Test 2: Tuned other parameters in postgresql.conf
- Performance gain of 55% at Peak load
- The performance continues to degrade [by a margin of 15%] at higher concurrency
Test 3: Transaction Level Connection Pooling
- Performance is more consistent
- Gain with additional memory is more prominent after tuning and connection pooling
To make best use
of added memory
tuning and
connection
pooling are
important](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-11-320.jpg)
![Low Memory Tuned Postgres Vs Un-Tuned
Postgres with Higher Memory
Database Parameters:
• No of Processors: 2
• Memory: 3GB/4.5GB,
• shared_buffer: 256MB/750MB,
• effective_cache_size: 750MB/2GB
• work_mem=1MB
Key Learning:
Test 1: Un-tuned PostgreSQL [only shared_buffer and effective_cache_size is tuned]
with 4.5 GB RAM
- The performance continues to degrade [by a margin of 38%] at higher concurrency
Test 3: Tune PostgreSQL with Transaction Level Connection Pooling
- Performance is more consistent
- Despite low memory, performance is comparable at low concurrency
- A Tuned database with proper transaction management does better as load increases
Most of your
performance
issues can be
resolved with
proper application
and database
tuning](https://image.slidesharecdn.com/magicoftuning-140603023458-phpapp01/85/The-Magic-of-Tuning-in-PostgreSQL-12-320.jpg)
The Magic of Tuning in PostgreSQL
- 1. Scaling with PostgreSQL Magic of Tuning and Connection Pooling
- 2. Pitfalls where IT Mangers get Trapped • Failure to have a correct plan for scaling with business • Falling prey to proprietary database vendors and their costly hardware • Failing to identify the correct Core-Memory Ratio • Adding Resources will not help all the time
- 3. Benchmark Studies: Objective • See how PostgreSQL scales with hardware • See how PostgreSQL scales with tuning • See the effect of proper Connection and Transaction Management on PostgreSQL performance • Highlight the importance of Tuning despite increasing system hardware
- 4. Benchmark Studies: Test bed Setup We have used HammerDB for this benchmark • An open source benchmarking tool which provides test cases for TPC-C and TPC-H benchmarks • We used TPC-C for our tests All the tests were run on a Virtual Machine (using VM player) on a Laptop • 64-bit Intel 8 core CPU • 16 GB of physical memory • Host OS- Windows 7 64 bit Testing Environment • Postgres Plus Advanced Server v9.2 • Running on Red Hat Enterprise Linux v6.1 (with kernel version 2.6)
- 5. Disclaimer • Performance without tuning and with tuning is shared here to make a point • The tests were done on a virtual environment running on a low end host machine • Scalability or raw performance numbers presented here are not true indicators of capabilities of PostgreSQL/Postgres Plus Advanced Server • With the real server class machines these numbers would be much higher than on the test machine
- 6. Low Configuration PostgreSQL server scaling with Connection Pooling Database Parameters: • No of Processors: 1 • Memory: 3GB, • shared_buffer: 32MB, • effective_cache_size: 128MB • work_mem=1MB Key Learning: Test 1 - The performance degrades once the load increases beyond a point [sounds familiar?] Test 2 - Session Level Connection pooling with max 50 connections in Pool - Connections aquired as needed so not much difference than first test Test 3 - Transaction Level Connection Pooling with 25 connections - Does well as concurrency increases [scaling with growth!!!] Inconsistent/low and highs can be attributed to resources contention during autovacuum
- 7. Some Important Parameters for Tuning in PostgreSQL • shared_buffer • effective_cache_size • work_mem • wal_buffer • bgwriter_delay • effective_io_concurrency • checkpoint_segment • checkpoint_timeout • random_page_cost • autovacuum_naptime • autovacuum_vacuum_threshold • autovacuum_analyze_threshold • autovacuum_vacuum_scale_factor • autovacuum_analyze_scale_factor • autovacuum_vacuum_cost_delay
- 8. PostgreSQL Performs better after Tuning Database Parameters: • No of Processors: 1 • Memory: 3GB, • shared_buffer: 32MB/256MB, • effective_cache_size: 128MB/750MB • work_mem=1MB Key Learning: Test 1: Basic Setup - The performance degrades once the load increases beyond a point [sounds familiar?] Test 2: Tuned postgresql.conf for shared_buffer and effective_cache_size - Up to 19% gain over basic configuration Test 3: Tuned other parameters in postgresql.conf - Up to 21% gain - The occasional performance troughs because of vacuum operation is not there anymore Still the performance is not scalable in any of the 3 cases and degrades with concurrency [25- 30% loss]
- 9. Combining the Effect of Connection Pooling and Tuning Database Parameters: • No of Processors: 1 • Memory: 3GB, • shared_buffer: 256MB, • effective_cache_size: 750MB • work_mem=1MB Key Learning: Test 1: Basic Setup with tuned shared_buffer and effective_cache_size - The performance degrades once the load increases beyond a point [sounds familiar?] Test 2: Tuned other parameters in postgresql.conf - Performance gain of up to 14% - The performance continues to degrade [by a margin of 26%] at higher concurrency Test 3: Transaction Level Connection Pooling - Performance is more consistent - Degradation/loss in performance at peak concurrency is quite less [14% only] Finally we get a consistently scaling Database Setup
- 10. Tuning and proper Transaction Management enhances the benefit of CPU addition Database Parameters: • No of Processors: 2 • Memory: 3GB, • shared_buffer: 256MB, • effective_cache_size: 750MB • work_mem=1MB Key Learning: Test 1: Basic Setup with tuned shared_buffer and effective_cache_size - The performance degrades once the load increases beyond a point Test 2: Tuned other parameters in postgresql.conf - Performance gain of 36% at Peak load - The performance continues to degrade [by a margin of 26%] at higher concurrency Test 3: Transaction Level Connection Pooling - Performance is more consistent - Improvement caused by tuning is more prominent with boosted computing [1CPU- 46% | 2CPU- 80%] Even after boosting computing power, to get best of resources, tuning and connection pooling plays a major role
- 11. Tuning and proper Transaction Management enhances the benefit of Memory Upgrade Database Parameters: • No of Processors: 2 • Memory: 4.5GB, • shared_buffer: 750MB, • effective_cache_size: 2GB • work_mem=1MB Key Learning: Test 1: Basic Setup with tuned shared_buffer[750MB] and effective_cache_size[2GB] - The performance degrades once the load increases beyond a point Test 2: Tuned other parameters in postgresql.conf - Performance gain of 55% at Peak load - The performance continues to degrade [by a margin of 15%] at higher concurrency Test 3: Transaction Level Connection Pooling - Performance is more consistent - Gain with additional memory is more prominent after tuning and connection pooling To make best use of added memory tuning and connection pooling are important
- 12. Low Memory Tuned Postgres Vs Un-Tuned Postgres with Higher Memory Database Parameters: • No of Processors: 2 • Memory: 3GB/4.5GB, • shared_buffer: 256MB/750MB, • effective_cache_size: 750MB/2GB • work_mem=1MB Key Learning: Test 1: Un-tuned PostgreSQL [only shared_buffer and effective_cache_size is tuned] with 4.5 GB RAM - The performance continues to degrade [by a margin of 38%] at higher concurrency Test 3: Tune PostgreSQL with Transaction Level Connection Pooling - Performance is more consistent - Despite low memory, performance is comparable at low concurrency - A Tuned database with proper transaction management does better as load increases Most of your performance issues can be resolved with proper application and database tuning
- 13. Ashnik Message • You don’t always need a high end system in the beginning to scale in future • There are commercially supported Open Source Solutions available which scale as good as their proprietary counterparts • There is no generic core-memory formula- Our consultants help customers identify what is best for them • A well designed application with a well tuned database can work better than a un-optimized database on a high end server
- 14. How Ashnik Helps? Server sizing Health Check contracts on half yearly and annual terms On-demand Health Check and Tuning services Provide in migration services- $$ Saving by migrating from costly proprietary database solutions