Lightening Talk - PostgreSQL Worst Practices

Mar 18, 20171 like350 views

PGConf APAC

Illay talked in 5 minutes about what are some of the worst practices he has come across while working on PostgreSQL

PostgreSQL
worst practices
Ilya Kosmodemiansky
ik@postgresql-consulting.com

Best practices are just boring
• Never follow them, try worst practices
• Only those practices can really help you to screw the things up
most eﬀectively
• PostgreSQL consultants are nice people, so try to make them
happy

1. Use as many count(*) as you can
• Figure 301083021830123921 is very informative for the end
user
• If it changes in a second to 30108302894839434020, it is still
informative
• select count(*) from sometable is a quite light query
• Tuple estimation from pg_catalog can never be precise
enough for you

2. Try to create as many indexes as you can
• Indexes consume no disk space
• Indexes consume no shared_bufers
• There is no overhead on DML if one and every column in a
table covered with bunch of indexes
• Optimizer will deﬁnitely choose your index once you created it
• Keep calm and create more indexes

3. Turn autovacuum oﬀ
• It is quite auxiliary process, you can easily stop it
• There is no problem at all to have 100Gb data in a database
which is 1Tb in size
• 2-3Tb RAM servers are cheap, IO is a fastest thing in modern
computing
• Besides of that, everyone likes BigData

4. Reinvent Slony
• If you need some data replication to another database, try to
implement it from scratch
• That allows you to run into all problems, PostgreSQL have
had since introducing Slony

5. Move joins to your application
• Just select * a couple of tables into the application written in
your favorite programming language
• Than join them at the application level
• Now you only need to implement nested loop join, hash join
and merge join as well as query optimizer and page cache

6. Never use graphical monitoring
• You do not need graphs
• Because it is an easy task to guess what was happened
yesterday at 2 a.m. using command line and grep only

7. Never use Foreign Keys
• Consistency control at application level always works as
expected
• You will never get data inconsistency without constraints

8. Always use text type for all columns
• It is always fun to reimplement date or ip validation in your
code
• You will never mistaken converting ”12-31-2015 03:01AM” to
”15:01 12 of undef 2015” using textﬁelds

Pgpool is middleware that works between PostgreSQL clients and servers to provide connection pooling, replication, and load balancing. The presenter's company deployed pgpool in various architectures including master-slave replication and load balancing configurations. They experienced some issues with pgpool like connection errors when using application pooling, lack of guaranteed connection reuse, and bugs. Tips are provided like ensuring synchronized server times and restricting health check users. Pgpool may not be best when automatic node rejoining is needed or during network instability.

Postgrtesql as a NoSQL Document Store - The JSON/JSONB data typeJumping Bean

Oracle to Postgres Migration - part 2PgTraining

Migration From Oracle to PostgreSQLPGConf APAC

[EPPG] Oracle to PostgreSQL, Challenges to OpportunityEqunix Business Solutions

Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi

This document summarizes a presentation about Presto, an open source distributed SQL query engine. It discusses Presto's distributed and plug-in architecture, query planning process, and cluster configuration options. For architecture, it explains that Presto uses coordinators, workers, and connectors to distribute queries across data sources. For query planning, it shows how SQL queries are converted into logical and physical query plans with stages, tasks, and splits. For configuration, it reviews single-server, multi-worker, and multi-coordinator cluster topologies. It also provides an overview of Presto's recent updates.

Presto in my_use_casewyukawa

The document summarizes the speaker's use of Presto for log analysis. Key points include: - Presto was selected due to familiarity from others and ease of use compared to other options. - Presto is used for batch queries with Hive and interactive queries. Results are accessed through Cognos using Prestogres. - Managing Presto involves deployment with Ansible, configuration tuning, and monitoring with tools like GrowthForecast and jstat2gf. - While Presto has been stable overall, the speaker notes some version upgrade issues but sees leverage from its frequent updates.

Compression talkIlya Ganelin

In the engineering world, we don’t always have the luxury of owning our data pipelines end to end. If only we could influence those outside components… Well, we tried, and this our story - replete with failure, discovery, and the serenity of enlightenment. Join us on our journey as we learned more than we ever wanted to know about compression in different Apache projects, deployed our own ingestion pipeline in Apache Flume, and ultimately unified these in a robust framework built on Apache Apex handling 1 TB of data per day. We end with some reflections on the joys and tribulations of the open source realm and some key lessons for other large applications atop multiple Apache solutions.

Big Data and PostgreSQLPGConf APAC

Your Guide to Streaming - The Engineer's PerspectiveIlya Ganelin

It feels like every week there's a new open-source streaming platform out there. Yet, if you only look at the descriptions, performance metrics, or even the architecture, they all start to look exactly the same! In short, nothing really differentiates itself - whether it be Storm, Flink, Apex, GearPumk, Samza, KafkaStreams, AkkaStreams, or any of the other myriad technologies. So if they all look the same, how do you really pick a streaming platform to solve the problem that YOU have? This talk is about how to really compare these platforms, and it turns out that they do have their key differences, they're just not the ones you usually think about. The way that you need to compare these systems if you're building something to last, a well-engineered system, is to look at how they handle durability, availability, how easy they are to install and use, and how they deal with failures.

Presto Meetup 2016 Small StartHiroshi Toyama

1. The presenter discusses their use of Presto for analytics at their company, including joining data across different data sources and using window functions on MySQL data. 2. They explain how they integrate Presto with other tools like re:dash for visualization and Embulk for ETL workflows. 3. While Presto solves many of their problems, they still require some ETL and have encountered issues like large repository sizes and coordinator bottlenecks.

Realtime classroom analytics powered by apache druid Karthik Deivasigamani

At Noon – The Social Learning Platform, on a daily basis we process close to 100M audio, sketch samples from more than 80K students to help measure the voice & sketch quality of our online classrooms. This talk explores the need for real time analytics in EdTech, how we built a real time analytics platform on Apache Druid & Apache Flink to provide realtime feedback on classroom quality & engagement metrics. We will also share some of the lessons we learnt along the way.

TPC-H in MongoDBAung Thu Rha Hein

This document discusses benchmarking TPC-H queries in MongoDB compared to MySQL. It introduces MongoDB and describes setting up the TPC-H data by embedding all tables into a single MongoDB collection. Six sample queries are presented and run using Map-Reduce and the Aggregation Framework. Benchmark results show MongoDB performing worse than MySQL on all queries due to data conversion difficulties and MongoDB's immature Aggregation Framework. The document concludes that while MongoDB is suitable for some applications, it is not well-suited to complex queries like those in TPC-H due to its lack of standard query language and server-side processing abilities.

Query Parallelism in PostgreSQL: What's coming next?PGConf APAC

Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)Matt Fuller

Tempto is a product test framework that allows developers to write and execute tests for SQL databases running on Hadoop. Individual test requirements such as data generation, HDFS file copy/storage of generated data and schema creation are expressed declaratively and are automatically fulfilled by the framework. Developers can write tests using Java (using a TestNG like paradigm and AssertJ style assertion) or by providing query files with expected results. We will show how we use it for presto product tests. Benchto is a benchmark framework that provides an easy and manageable way to define, run and analyze macro benchmarks in clustered environment. Understanding behavior of distributed systems is hard and requires good visibility intostate of the cluster and internals of tested system. This project was developed for repeatable benchmarking ofHadoop SQL engines, most importantly Presto.

Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Martin Traverso

This document summarizes Presto, an analytics engine used at Facebook. It provides ad-hoc querying for data warehouses and batch processing. It is used for analytics across Facebook's data warehouses and specialized data stores. The document outlines Presto's architecture, deployment, usage statistics, features, and enhancements made for specific Facebook use cases including user-facing products, large datasets, and reliable data loading.

Ambry : Linkedin's Scalable Geo-Distributed Object StoreSivabalan Narayanan

Architecture for building scalable and highly available Postgres ClusterAshnikbiz

As PostgreSQL has made way into business critical applications, many customers who are using Oracle RAC for high availability and load balancing have asked for similar functionality for using PostgreSQL. In this Hangout session we would discuss architecture and alternatives, based on real life experience, for achieving high availability and load balancing functionality when you deploy PostgreSQL. We will also present some of the key tools and how to deploy them for effectiveness of this architecture.

Oracle 12c Parallel Execution New FeaturesRandolf Geist

This document discusses new parallel execution features introduced in Oracle 12c. It begins with an introduction to key aspects of parallel execution, including the producer-consumer model and data distribution skew. The document then covers major new 12c features such as hybrid hash distribution, concurrent UNION ALL, and the 1 slave distribution method. It concludes with a question and answer section.

Internals of Presto ServiceTreasure Data, Inc.

Presto is a distributed SQL query engine that Treasure Data provides as a service. Taro Saito discussed the internals of the Presto service at Treasure Data, including how the TD Presto connector optimizes scan performance from storage systems and how the service manages multi-tenancy and resource allocation for customers. Key challenges in providing a database as a service were also covered, such as balancing cost and performance.

Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting

Even an experienced PostgreSQL DBA can not always say that upgrading between major versions of Postgres is an easy task, especially if there are some special requirements, such as downtime limitations or if something goes wrong. For less experienced DBAs anything more complex than dump/restore can be frustrating. In this talk I will describe why we need a special procedure to upgrade between major versions, how that can be achieved and what sort of problems can occur. I will review all possible ways to upgrade your cluster from classical pg_upgrade to old-school slony or modern methods like logical replication. For all approaches, I will give a brief explanation how it works (limited by the scope of this talk of course), examples how to perform upgrade and some advice on potentially problematic steps. Besides I will touch upon such topics as integration of upgrade tools and procedures with other software — connection brokers, operating system package managers, automation tools, etc. This talk would not be complete if I do not cover cases when something goes wrong and how to deal with such cases.

Presto in my_use_case2wyukawa

Migrating Oracle database to PostgreSQLUmair Mansoob

This document discusses migrating an Oracle database to PostgreSQL. It covers initial discovery of the Oracle database features and data types used. A migration assessment would analyze data type mapping, additional PostgreSQL features, and testing requirements. Challenges include porting PL/SQL code, minimizing downtime during migration, and comprehensive testing of applications on the new PostgreSQL platform. Migrating large data sets and ensuring performance for critical applications are also challenges.

Go faster with_native_compilation Part-2Rajeev Rastogi (KRR)

Presto changesN Masahiro

This document summarizes recent updates to Presto, including new data types, connectors, syntax, features, functions, and configuration options. Some key additions are support for DECIMAL, VARCHAR, and new data types; connectors for Redis, MongoDB, and other data sources; transaction support; and a variety of new SQL functions for strings, dates, aggregation, and more. Upcoming work includes prepared statements, a new optimizer, and other performance and usability improvements.

Presto - Analytical Database. Overview and use cases.Wojciech Biela

PostgreSQL WAL for DBAs PGConf APAC

Devrim Gunduz gives a presentation on Write-Ahead Logging (WAL) in PostgreSQL. WAL logs all transactions to files called write-ahead logs (WAL files) before changes are written to data files. This allows for crash recovery by replaying WAL files. WAL files are used for replication, backup, and point-in-time recovery (PITR) by replaying WAL files to restore the database to a previous state. Checkpoints write all dirty shared buffers to disk and update the pg_control file with the checkpoint location.

PostgreSQL on Amazon RDSPGConf APAC

More Related Content

What's hot (20)

Presto in my_use_casewyukawa

Compression talkIlya Ganelin

Big Data and PostgreSQLPGConf APAC

Your Guide to Streaming - The Engineer's PerspectiveIlya Ganelin

Presto Meetup 2016 Small StartHiroshi Toyama

Realtime classroom analytics powered by apache druid Karthik Deivasigamani

TPC-H in MongoDBAung Thu Rha Hein

Query Parallelism in PostgreSQL: What's coming next?PGConf APAC

Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)Matt Fuller

Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Martin Traverso

Ambry : Linkedin's Scalable Geo-Distributed Object StoreSivabalan Narayanan

Architecture for building scalable and highly available Postgres ClusterAshnikbiz

Oracle 12c Parallel Execution New FeaturesRandolf Geist

Internals of Presto ServiceTreasure Data, Inc.

Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting

Presto in my_use_case2wyukawa

Migrating Oracle database to PostgreSQLUmair Mansoob

Go faster with_native_compilation Part-2Rajeev Rastogi (KRR)

Presto changesN Masahiro

Presto - Analytical Database. Overview and use cases.Wojciech Biela

Presto in my_use_casewyukawa

Compression talkIlya Ganelin

Big Data and PostgreSQLPGConf APAC

Your Guide to Streaming - The Engineer's PerspectiveIlya Ganelin

Presto Meetup 2016 Small StartHiroshi Toyama

Realtime classroom analytics powered by apache druid Karthik Deivasigamani

TPC-H in MongoDBAung Thu Rha Hein

Query Parallelism in PostgreSQL: What's coming next?PGConf APAC

Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)Matt Fuller

Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Martin Traverso

Ambry : Linkedin's Scalable Geo-Distributed Object StoreSivabalan Narayanan

Architecture for building scalable and highly available Postgres ClusterAshnikbiz

Oracle 12c Parallel Execution New FeaturesRandolf Geist

Internals of Presto ServiceTreasure Data, Inc.

Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting

Presto in my_use_case2wyukawa

Migrating Oracle database to PostgreSQLUmair Mansoob

Go faster with_native_compilation Part-2Rajeev Rastogi (KRR)

Presto changesN Masahiro

Presto - Analytical Database. Overview and use cases.Wojciech Biela

Viewers also liked (20)

PostgreSQL WAL for DBAs PGConf APAC

PostgreSQL on Amazon RDSPGConf APAC

PostgreSQL: Past present FuturePGConf APAC

PostgreSQL has evolved from its origins in academic research projects in the 1970s-1980s to a widely used open source database today. It has a large and active user community supporting deployments across industries and organization sizes. The future of PostgreSQL remains bright, as it continues to add new features and performance improvements while maintaining its low cost, flexibility, and reliability advantages over closed source databases. Major areas of focus for ongoing PostgreSQL development include application-specific data types, advanced indexing techniques, and improved single and multi-node scalability.

Security Best Practices for your Postgres DeploymentPGConf APAC

How to teach an elephant to rock'n'rollPGConf APAC

The document discusses techniques for optimizing PostgreSQL queries, including: 1. Using index only scans to efficiently skip large offsets in queries instead of scanning all rows. 2. Pulling the LIMIT clause under joins and aggregates to avoid processing unnecessary rows. 3. Employing indexes creatively to perform DISTINCT operations by scanning the index instead of the entire table. 4. Optimizing DISTINCT ON queries by looping through authors and returning the latest row for each instead of a full sort.

Introduction to Vacuum Freezing and XIDPGConf APAC

Use Case: PostGIS and AgriboticsPGConf APAC

This document discusses using drones and PostgreSQL/PostGIS for agricultural applications. It describes how drones can capture imaging data for tasks like measuring crop health through NDVI analysis. PostgreSQL is useful for organizing the large amounts of drone data, like flight plans, sensor readings, and imagery. The document provides an example of importing this data into PostgreSQL and using PostGIS functions to process imagery, extract waypoints of problem areas, and more.

Swapping Pacemaker Corosync with repmgrPGConf APAC

Secure PostgreSQL deploymentCommand Prompt., Inc

Magnus Hagander PostgreSQL supports several options for securing communications when deployed outside the typical webserver/database combination. This talk will go into some details about the features that make this possible, with some extra focus on the changes in 8.4. The main areas discussed are: * Securing the channel between client and server using SSL, including an overview of the threats and how to secure against them * Securing the login process, using LDAP, Kerberos or SSL certificates, including the use of smartcards to log into the database The talk will not focus on security and access control inside the database once the user is connected and authenticated.

Past, Present, and Future Analysis of the Architectural & Engineering Design ...Lisa Dehner

Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel UrmaJAXLondon_Conference

This document summarizes the past, present, and future of generics in Java and other languages. In the past, generics were added to Java to provide compile-time type safety. Presently, Java generics are commonly used with collections but wildcards are used less. Future areas of exploration include intersection types, declaration-site variance, value types, and unbounded wildcards. Generics usage continues to increase in complexity as new language features are added.

pg_hba.conf 이야기PgDay.Seoul

24/7 Monitoring and Alerting of PostgreSQLInMobi Technology

The digital universe is huge and is growing at a stellar rate and along with it grows the data generated every second. By 2020, there will be nearly as many digital bits as there are stars in this universe. That effectively means infinite as per the reports published by IDC in 2014. InMobi has grown leaps and bounds globally in past few years and that has only caused the data here to grow exponentially. There are thousands of advertisers and publishers on InMobi network, handling the OLTP ( 200-300 GB ) and OLAP ( 14TB ) demands high availability and the best performance. To ensure the smoothness and 24/7 availability of our production database servers, we are using a lot of open source technologies to keep an eye on all the Postgresql servers running across different data centres. We have one of the biggest Postgresql Master-Slave Streaming Replication production setup and it is very important for us to monitor the database performance, production traffic and some analytics on top of each and every database server @InMobi.

PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...Mark Wong

Fifth presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation goes through results of different hardware RAID configurations to show why it is important to test your own hardware: it might be performing in way you don't expect.

Achieving Pci CompliaceDenish Patel

The document discusses achieving PCI compliance when using PostgreSQL for databases. It provides an overview of PCI requirements, how they apply to databases, and how PostgreSQL features like encryption, access control, and logging can help fulfill the requirements. Specific examples are given for how to implement encryption of cardholder data, restrict access according to the principle of least privilege, and maintain regularly updated software in PostgreSQL.

PgDay Asia 2016 - Security Best Practices for your Postgres DeploymentAshnikbiz

Ashnik Database Solution Architect, Sameer Kumar, an Open Source database evangelist talked about the "Security Best Practices for your Postgres Deployment" at the recent pgDAy Asia event held in Singapore in March 2016. Key areas he presented were: - Security Model - Security Features in Postgres - Securing the access - Avoiding common attacks - Access Control and Securing data - Logging and Auditing - Patching – OS and PostgreSQL

Microservices Past, Present, FutureDavid Dawson

This document provides an overview of microservices from past to present to future. It discusses how microservices evolved from earlier concepts like SOA and how new technologies like containers and platforms helped popularize microservices. The key aspects of microservices architecture are defined as isolation and flexibility. Current trends include the rise of platforms like Kubernetes and serverless computing. Issues around data management, communication styles, and industry adoption are also covered at a high level.

Postgres in Production - Best Practices 2014EDB

This presentation explores a broad cross-section of enterprise Postgres deployments to identify key usage patterns and reveals important aspects of performance, scalability, and availability including: * Challenges organizations encounter most frequently during the stages of database development, deployment and maintenance * Tuning parameters used most frequently to improve performance of production databases * Frequently problematic database maintenance processes and configuration parameters * Most commonly-used database back-up and recovery strategies

(Ab)using 4d IndexingPGConf APAC

5 Steps to PostgreSQL PerformanceCommand Prompt., Inc

This document provides an overview of five steps to improve PostgreSQL performance: 1) hardware optimization, 2) operating system and filesystem tuning, 3) configuration of postgresql.conf parameters, 4) application design considerations, and 5) query tuning. The document discusses various techniques for each step such as selecting appropriate hardware components, spreading database files across multiple disks or arrays, adjusting memory and disk configuration parameters, designing schemas and queries efficiently, and leveraging caching strategies.

PostgreSQL WAL for DBAs PGConf APAC

PostgreSQL on Amazon RDSPGConf APAC

PostgreSQL: Past present FuturePGConf APAC

Security Best Practices for your Postgres DeploymentPGConf APAC

How to teach an elephant to rock'n'rollPGConf APAC

Introduction to Vacuum Freezing and XIDPGConf APAC

Use Case: PostGIS and AgriboticsPGConf APAC

Swapping Pacemaker Corosync with repmgrPGConf APAC

Secure PostgreSQL deploymentCommand Prompt., Inc

Past, Present, and Future Analysis of the Architectural & Engineering Design ...Lisa Dehner

Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel UrmaJAXLondon_Conference

pg_hba.conf 이야기PgDay.Seoul

24/7 Monitoring and Alerting of PostgreSQLInMobi Technology

PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...Mark Wong

Achieving Pci CompliaceDenish Patel

PgDay Asia 2016 - Security Best Practices for your Postgres DeploymentAshnikbiz

Microservices Past, Present, FutureDavid Dawson

Postgres in Production - Best Practices 2014EDB

(Ab)using 4d IndexingPGConf APAC

5 Steps to PostgreSQL PerformanceCommand Prompt., Inc

More from PGConf APAC (18)

PGConf APAC 2018: Sponsored Talk by Fujitsu - The growing mandatory requireme...PGConf APAC

Speaker: Rajni Baliyan As the volume of data of a personal nature and commodification of information collected and analysed increases; so is the focus on privacy and data security. Many countries are examining international and domestic laws in order to protect consumers and organisations alike. The Australian Senate has recently passed a bill containing mandatory requirements to notify the privacy commissioner and consumers when data is at risk of causing serious harm in the case of a data breach occurring. Europe has also announced new laws that allow consumers more control over their data. These laws allow consumers to tell companies to erase any data held about them. These new laws will have a significant impact on organisations that store personal information. This talk will examine some of these legislative changes and how specific PostgreSQL features can assist organisations in meeting their obligations and avoid heavy fines associated with breaching them.

PGConf APAC 2018: PostgreSQL 10 - Replication goes LogicalPGConf APAC

While the physical replication in PostgreSQL is quite robust, however, it doesn’t fit well in the picture when: - You need partial replication only - You want to replicate between different major versions of PostgreSQL - You need to replicate multiple databases to the same target - Transformation of the data is needed - You want to replicate in order to upgrade without downtime The answer to these use cases is logical replication This talk will discuss and cover these use cases followed by a logical replication demo.

PGConf APAC 2018 - Lightening Talk #3: How To Contribute to PostgreSQLPGConf APAC

This document outlines various ways to contribute to the PostgreSQL open source database project. It discusses that PostgreSQL needs support from individuals and companies to continue developing and competing against commercial databases. Contributing provides benefits like being listed as a contributor or sponsor on PostgreSQL's website. The document then lists several contribution methods like making donations, participating in surveys, providing hardware/infrastructure, helping with documentation, answering user questions, reporting bugs, and writing code in the form of tools, extensions, or patches.

PGConf APAC 2018 - Lightening Talk #2 - Centralizing Authorization in PostgreSQLPGConf APAC

The document discusses implementing centralized authorization in PostgreSQL by synchronizing user roles and privileges with an LDAP server. It provides a step-by-step approach to setting up LDAP authentication in PostgreSQL and using scripts to synchronize user roles and privileges between the database and LDAP based on group membership. The synchronization scripts create roles for each LDAP user, grant privileges to roles based on mapping rules, and handle role inheritance.

Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...PGConf APAC

PGConf APAC 2018 - A PostgreSQL DBAs Toolbelt for 2018PGConf APAC

There's no need to re-invent the wheel! Dozens of people have already tried...and succeeded. This talk is a categorized and illustrated overview on most popular and/or useful PostgreSQL specific scripts, utilities and whole toolsets that DBAs should be aware of for solving daily tasks. Inlcuding - performance monitoring, logs management/analyzis, identifying/fixing most common adminstration problems around areas of general performance metrics, tuning, locking, indexing, bloat, leaving out high-availability topics. Covered are venerable oldies from wiki.postgresql.org as well as my newer favourites from Github.

PGConf APAC 2018 - Patroni: Kubernetes-native PostgreSQL companionPGConf APAC

Speaker: Alexander Kukushkin Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure. There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul. Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes. Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni

PGConf APAC 2018 - High performance json postgre-sql vs. mongodbPGConf APAC

Speakers: Dominic Dwyer & Wei Shan Ang This talk was presented in Percona Live Europe 2017. However, we did not have enough time to test against more scenario. We will be giving an updated talk with a more comprehensive tests and numbers. We hope to run it against citusDB and MongoRocks as well to provide a comprehensive comparison. https://www.percona.com/live/e17/sessions/high-performance-json-postgresql-vs-mongodb

PGConf APAC 2018 - Monitoring PostgreSQL at ScalePGConf APAC

Speaker: Lukas Fittl Your PostgreSQL database is one of the most important pieces of your architecture - yet the level of introspection available in Postgres is often hard to work with. Its easy to get very detailed information, but what should you really watch out for, send reports on and alert on? In this talk we'll discuss how query performance statistics can be made accessible to application developers, critical entries one should monitor in the PostgreSQL log files, how to collect EXPLAIN plans at scale, how to watch over autovacuum and VACUUM operations, and how to flag issues based on schema statistics. We'll also talk a bit about monitoring multi-server setups, first going into high availability and read standbys, logical replication, and then reviewing how monitoring looks like for sharded databases like Citus. The talk will primarily describe free/open-source tools and statistics views readily available from within Postgres.

PGConf APAC 2018 - Where's Waldo - Text Search and Pattern in PostgreSQLPGConf APAC

Speaker: Joe Conway There are many use cases for text search and pattern matching, and there are also a wide variety of techniques available in PostgreSQL to perform text search and pattern matching. Figuring out the best "match" between use case and technique can be confusing. This talk will review the possibilities and provide guidance regarding when to use what method, and especially how to properly deal with the related index methods to ensure speedy searches. This talk covers: * The primary available search methods * Examples illustrating when to use each * Extensive discussion of index use * Timing comparisons using realistic examples

PGConf APAC 2018 - Managing replication clusters with repmgr, Barman and PgBo...PGConf APAC

Speaker: Ian Barwick PostgreSQL and reliability go hand-in-hand - but your data is only truly safe with a solid and trusted backup system in place, and no matter how good your application is, it's useless if it can't talk to your database. In this talk we'll demonstrate how to set up a reliable replication cluster using open source tools closely associated with the PostgreSQL project. The talk will cover following areas: - how to set up and manage a replication cluster with `repmgr` - how to set up and manage reliable backups with `Barman` - how to manage failover and application connections with `repmgr` and `PgBouncer` Ian Barwick has worked for 2ndQuadrant since 2014, and as well as making various contributions to PostgreSQL itself, is lead `repmgr` developer. He lives in Tokyo, Japan.

PGConf APAC 2018 - PostgreSQL HA with Pgpool-II and whats been happening in P...PGConf APAC

Speaker: Muhammad Usama Pgpool-II has been around to complement PostgreSQL over a decade and provides many features like connection pooling, failover, query caching, load balancing, and HA. High Availability (HA) is very critical to most enterprise application, the clients needs the ability to automatically reconnect with a secondary node when the master nodes goes down. This is where Pgpool-II watchdog feature comes in, the core feature of Pgpool-II provides HA by eliminating the SPOF is the Watchdog. This watchdog feature has been around for a while but it went through major overhauling and enhancements in recent releases. This talk aims to explain the watchdog feature, the recent enhancements went into the watchdog and describe how it can be used to provide PostgreSQL HA and automatic failover. Their is rising trend of enterprise deployment shifting to cloud based environment, Pgpool II can be used in the cloud without any issues. In this talk we will give some ideas how Pgpool-II is used to provide PostgreSQL HA in cloud environment. Finally we will summarise the major features that have been added in the recent major release of Pgpool II and whats in the pipeline for the next major release.

PGConf APAC 2018 - PostgreSQL performance comparison in various cloudsPGConf APAC

Speaker: Oskari Saarenmaa Aiven PostgreSQL is available in five different public cloud providers' infrastructure in more than 60 regions around the world, including 18 in APAC. This has given us a unique opportunity to benchmark and compare performance of similar configurations in different environments. We'll share our benchmark methods and results, comparing various PostgreSQL configurations and workloads across different clouds.

Sponsored Talk @ PGConf APAC 2018 - Migrating Oracle to EDB Postgres Approach...PGConf APAC

This document discusses migrating Oracle databases to EDB Postgres. It outlines the steps to migrate, including assessing the database, preparing the environment, migrating database objects and data, porting applications, testing, integrating, and rolling out the migration. It then provides two case studies of large companies that migrated from Oracle to EDB Postgres to significantly lower costs while still meeting their business and technical requirements.

PGConf APAC 2018 - Tale from TrenchesPGConf APAC

About a year ago I was caught up in line-of-fire when a production system started behaving abruptly - A batch process which would finish in 15minutes started taking 1.5 hours - We started facing OLTP read queries on standby being cancelled - We faced a sudden slowness on the Primary server and we were forced to do a forceful switch to standby. We were able to figure out that some peculiarities of the application code and batch process were responsible for this. But we could not fix the application code (as it is packaged application). In this talk I would like to share more details of how we debugged, what was the problem we were facing and how we applied a work around for it. We also learnt that a query returning in 10minutes may not be as dangerous as a query returning in 10sec but executed 100s of times in an hour. I will share in detail- - How to map the process/top stats from OS with pg_stat_activity - How to get and read explain plan - How to judge if a query is costly - What tools helped us - A peculiar autovacuum/vacuum Vs Replication conflict we ran into - Various parameters to tune autvacuum and auto-analyze process - What we have done to work-around the problem - What we have put in place for better monitoring and information gathering

PGConf APAC 2018 Keynote: PostgreSQL goes elevenPGConf APAC

The document discusses PostgreSQL version 11 and future development. It provides a history of PostgreSQL and its predecessors, describing the development process and community. It summarizes key features committed to version 11, including improvements to partitioning, parallelization, performance and logical replication. It also outlines features proposed for future versions, with a focus on continued enhancements to partitioning and query planning.

Amazon (AWS) AuroraPGConf APAC

Go Faster With Native CompilationPGConf APAC