PostgreSQL replication from setup to advanced features.

Apr 27, 20160 likes597 views

The document discusses various PostgreSQL replication methods and tools, including streaming and logical replication options, as well as middleware solutions like pgpool, slony, and bucardo. It highlights key features such as index types, connection pooling, and maturity of different replication solutions. Additionally, it provides a link for further exploration of PostgreSQL replication and clustering resources.

PostgreSQL replication
Lviv, #pivorak, 22/04/16

PG vs. MySQL
● Indexes on Expressions
● Partial Indexes
● Combining Multiple
Indexes
● Index Types
● 50+ Data Types
● Pattern Matching
with RegExps
● Savepoints
● Different Auth
Providers

Terms
● Write Ahead Log (WAL)
● Streaming Replication (SR)
● Physical Log Streaming Replication (LLSR)
● Logical Log Streaming Replication (LLSR)
● Bi-Directional Replication (BDR)

Physical Log Streaming Replication
WAL
Master
Slave

Solutions
Program License Maturity Replication Method Sync Connection Pooling Load Balancing Query Partitioning
PgCluster BSD Stalled Master-Master Synchronous No Yes No
pgpool-I BSD Stable Statement-Based
Middleware
Synchronous Yes Yes No
Pgpool-II BSD Recent release Statement-Based
Middleware
Synchronous Yes Yes Yes
slony BSD Stable Master-Slave Asynchronous No No No
Bucardo BSD Stable Master-Master, Master-
Slave
Asynchronous No No No
Londiste BSD Stable Master-Slave Asynchronous No No No
Mammoth BSD Stalled Master-Slave Asynchronous No No No
rubyrep MIT Stalled Master-Master, Master-
Slave
Asynchronous No No No
BDR (Bi-Directional
Replication)
PostgreSQL (BSD) Stable Master-Master
(no triggers needed)
Asynchronous No No No
pg_shard LGPL Recent release Statement-based
Middleware (as an
extension)
Synchronous No Yes Yes
pglogical PostgreSQL Recent Release Master-Slave Asynchronous No No No
https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling

Let’s practice
Browse to
vova.github.io/pivorak.md

This document covers several key topics in C programming for Linux including preprocessing, compilation and linking; processes and process address spaces; stacks and scope; libraries; makefiles; gdb basics; and pointers. It discusses concepts like static and dynamic libraries, the stack and call stack, variables and scopes, linking to shared objects, makefile rules and variables, using gdb for debugging, and common pointer issues. The document appears to be notes for a training session on C programming fundamentals in Linux.

gRPCDharshana Ratnayake

The document compares REST and gRPC approaches to building APIs. It notes that while REST uses JSON over HTTP, gRPC uses protocol buffers over HTTP/2, allowing for benefits like binary encoding, multiplexing, and simpler parsing. An example shows a character request is 205 characters in JSON but only 44 bytes in protocol buffers. gRPC is also said to allow lower latency, better CPU utilization, and server push capabilities compared to REST for real-time use cases.

Scalability (NoSQL Inspired Topic)yibter

This document discusses scalable systems and NoSQL databases. It provides examples of companies that use scalable systems like Google with BigTable, Amazon with Dynamo and SimpleDB, and Twitter with Hadoop and Pig. It then defines what is meant by a scalable system, such as doubling resources leading to doubling performance. Finally, it lists some popular technologies used to build scalable systems, including distributed key-value stores, document databases, and MapReduce.

Graph databases in computational bioloby: case of neo4j and TitanDBAndrei KUCHARAVY

This document discusses graph databases and their use in computational biology. It introduces Neo4j and TitanDB as graph database options and describes how biological interaction networks and pathways can be modeled as graphs. Key advantages of graph databases over relational databases are also summarized, such as increased speed for graph queries and simpler programming. The document provides an overview of Neo4j and TitanDB, including their core abstractions, interfaces, and advantages/limitations for storing large biological network data. Examples are given of loading Reactome pathway data into Neo4j and performing graph queries.

Lab 01 03_16Hao Wu

The document discusses the status and plans for pipelines integrating T cell receptor (TCR) and B cell receptor (BCR) analysis. The current TCR pipeline involves preprocessing taking around 10 minutes, storage for 5 minutes, and analysis for half an hour, with a minor problem to fix. It plans to improve the TCR algorithm from mitcr to mixcr, which will support additional receptor types, and move the BCR pipeline from fasta to fastq format, which is almost complete. A framework is envisioned to handle both TCR and BCR pipelines where the downstream analysis would be similar but the upstream processes are very different currently.

KubeCon EU 2019 - P2P Docker Image Distribution in Hybrid Cloud Environment w...Yiran Wang

The document discusses the architecture and performance of Kraken, a P2P Docker image distribution system designed to optimize speed and efficiency in large-scale environments like Uber. It covers key concepts such as Docker images, registry APIs, caching strategies, and peer discovery mechanisms, while detailing challenges and solutions for handling large data transfers. Future plans include improving performance for tiny files and integrating more closely with Kubernetes and other registry systems.

From rest api to graph ql a 10 year journeyArno Schulz

The document chronicles the author's transition from REST APIs to GraphQL over a decade, highlighting key milestones and technologies encountered along the way. It emphasizes the advantages of GraphQL, such as improved resource discovery, server-side business logic, and the ability to combine numerous requests into a single query. The author also discusses best practices for choosing and implementing libraries in Go for GraphQL development, emphasizing the importance of adapting solutions to the language.

How to specify `frozen_string_literal: true`Kazuhiro Nishiyama

This document discusses specifying the frozen_string_literal: true option in Ruby files. Frozen string literals are faster and generate less objects but can cause errors if strings are modified. The presenter recommends using frozen_string_literal: true if possible but remaining compatible with frozen-string_literal by also allowing false. Thorough testing is important when making changes to ensure no modifications are overlooked.

Automata Invasionlucenerevolution

The document discusses the use of finite state automata (FSA) and finite state transducers (FST) within the context of the Lucene search library, highlighting their implementations and applications. It covers three main implementations: automaton, FST, and token stream, illustrating their functionalities and performance benefits. The document also outlines various applications of these models in processing and querying data, emphasizing their importance in optimizing Lucene's capabilities.

EVCache: Lowering Costs for a Low Latency Cache with RocksDBScott Mansfield

EVCache is a distributed, sharded, replicated key-value store optimized for Netflix's use cases on AWS. It is based on Memcached but uses RocksDB for persistent storage, lowering costs compared to storing all data in memory. Moneta is the next generation EVCache server, using Rend and Mnemonic libraries to intelligently manage data placement in RAM and SSD. This provides high performance for both volatile and batch workloads while reducing costs by 70% compared to the original Memcached-based design.

Text tagging with finite state transducerslucenerevolution

David Smiley discusses text tagging using finite state transducers (FSTs) in his presentation at Lucene/Solr Revolution 2013. He outlines the capabilities of the Opensextant project for geotagging unstructured text and compares various tagging implementations, highlighting Lucene's efficient FST approach which significantly reduces memory usage. The presentation emphasizes the potential applications of FSTs in improving text tagging performance while integrating seamlessly with Solr for scalability and ease of updating.

Kraken mesoscon 2018joeyzhang1989928

The document discusses the evolution of the Docker registry at Uber and the development of a peer-to-peer (p2p) solution called Kraken, aimed at handling growing demands on bandwidth and disk I/O. It details the architecture, performance challenges, and potential optimizations for efficiently managing container images in a rapidly scaling environment. The presentation also outlines the benefits of p2p solutions within data centers and future plans for open sourcing and integration with existing systems.

Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...PROIDEA

The document discusses centralized log management using Logstash and Kibana, addressing various problems and solutions related to log transport formats, architectures, and best practices. Key components include a log sender, router, collector, and a full-text search engine, with details on transport formats like Syslog and JSON. It emphasizes practices such as processing logs on the sender side, maintaining log systems, and utilizing Elasticsearch and Kibana for searching and analyzing logs.

Couchbase live 2016Pierre Mavro

Pierre Mavro from Criteo discussed Couchbase usage at their company. Criteo has over 100 Couchbase clusters storing over 90TB of data serving up to 25 million queries per second. They benchmarked Couchbase and found network bandwidth and replicas increased latency. To improve, Criteo monitored latency, split workloads across clusters, automated operations, and tuned Couchbase and systems. Their changes helped Couchbase scale for Criteo's large workload.

Cassandra ExplainedEric Evans

This document provides an overview and introduction to Cassandra, an open source distributed database management system designed to handle large amounts of data across many commodity servers. It discusses Cassandra's origins from influential papers on Bigtable and Dynamo, its properties including flexibility, scalability and high availability. The document also covers Cassandra's data model using keyspaces and column families, its consistency options, API including Thrift and language drivers, and provides examples of usage for an address book app and storing timeseries data.

PostgreSQL and Compressed Documents (pgconf.ru 2018)Aleksander Alekseev

This document discusses data compression techniques for PostgreSQL, including ZSON for compressing JSONB data and pg_protobuf for indexing Protobuf data. It provides overviews of how these extensions work, such as replacing common strings with codes for ZSON and avoiding shared dictionaries for pg_protobuf. The document also shares ideas for future projects, such as adding modification support to pg_protobuf or generating PL/pgSQL procedures for Protobuf fields. Finally, it invites the audience to learn more about these open source projects and announces job opportunities at Postgres Professional.

gRPC: Beyond RESTDomingo Suarez Torres

This document discusses gRPC, an open-source high performance RPC framework created by Google. It provides benefits over REST such as binary encoding which allows for higher performance, as well as a contract-first approach using protocol buffers to define services. Protocol buffers can generate code for multiple platforms and allow flexible evolution of structures. gRPC uses HTTP/2 for a single connection with header compression. While REST is still necessary for some use cases, gRPC provides higher performance for internal APIs and microservices.

Is It Faster to Go with Redpanda Transactions than Without Them?!ScyllaDB

Denis Rystsov, a staff engineer at Vectorized, discusses the performance of Redpanda transactions compared to traditional methods, highlighting the importance of optimizations such as eventual consistency and parallel commits. The document emphasizes strategies to improve efficiency, including minimizing disk I/O and leveraging algorithmic complexity. Key takeaways include embracing limitations and adopting a more optimistic approach to transaction handling.

Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicumScott Tsai

This document discusses the Capsicum capability model and its use in sandboxing systems in FreeBSD. Capsicum allows tight sandboxing by limiting access when a process enters capability mode using cap_enter(). It describes debugging capabilities like ktrace and enotcap. It also introduces Casper, which provides additional capabilities through delegation before entering capability mode. Casper allows easier process separation by creating services run with user privileges that processes can then securely access through APIs. The document outlines several system services implemented through Casper and future work integrating it more fully with libc and sandboxing additional system services.

Logging for OpenStack - Elasticsearch, Fluentd, Logstash, KibanaMd Safiyat Reza

erlang 101Gokhan Boranalp

Erlang 101 provides an overview of the Erlang programming language. It discusses Erlang's history and current status, key features like concurrency, distribution, and fault tolerance. These features make Erlang well-suited for building large, distributed, highly-available systems. Examples are given of companies using Erlang like Amazon, Facebook, Twitter, and of applications and frameworks built with Erlang like Apache CouchDB, RabbitMQ, and Riak. Resources for learning more about Erlang are also provided.

HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon

Toolchain Independent Distributed CompilationDietmar Hauser

This document discusses toolchain independent distributed compilation. It explores previous solutions like IncrediBuild and distcc, and proposes a plan to analyze source files to find dependencies and distribute compilation across servers while caching inputs and outputs. Key challenges include precompiled headers, preprocessor directives, and debug information storing absolute paths. The author details solutions using sandboxing software. Benchmarking shows speedups of up to 17x for large codebases. Future work could involve distributed linking and unifying with LLVM/Clang.

Cloud storage: the right way OSS EU 2018Orit Wasserman

1. The document discusses cloud object storage, describing its features like multipart uploads, versioning, and lifecycles. It provides examples of using object storage for media and documents. 2. Key aspects of object storage security are covered, including signatures, encryption, access control lists, and policies. Disaster recovery options like geo-replication are also summarized. 3. In the conclusion, the document emphasizes using object storage APIs to access advanced features, ensuring data safety, testing disaster recovery plans, and using Ceph for private cloud object storage.

Ceph Day Chicago: Using Ceph for Large Hadron Collider Data Ceph Community

Lincoln Bryant from the University of Chicago gave a presentation on using Ceph for Large Hadron Collider data storage and analysis. Key points: - Ceph is used to store reconstruction data from the ATLAS experiment at the LHC and for analysis datasets. - Ceph provides scalable storage through an erasure coded CephFS that is mounted by XRootD servers for data access. - Ceph allows efficient data transfers from CERN to the University of Chicago site for regional analysis. - Future evaluations include using librados directly with XRootD and running Ceph and analysis jobs on the same cluster nodes using cgroups for resource control.

Baker: Scaling OVN with Kubernetes API ServerHan Zhou

This document discusses using Kubernetes API server framework (Baker) to scale out OVN control plane. It aims to distribute OVN northd and central OVSDB cluster. Challenges include lack of clustering in OVSDB SB and distributed state management. Baker proposes distributed northd, scale-out central cluster backed by ETCD, and distributed agents to watch local objects. This would address connectivity issues in large cloud deployments. The document also discusses addressing segmentation vs connectivity, using ACLs and address sets for L3 segmentation, and evaluating port security and ACL processing on each hypervisor.

OVN Controller Incremental ProcessingHan Zhou

The document discusses the challenges and solutions for incremental processing in OVN Controller, highlighting issues such as CPU resource waste and high latency due to full recomputation. It introduces an incremental processing engine using a Directed Acyclic Graph (DAG) to handle dependencies and change management efficiently. The future direction includes leveraging Differential Datalog (ddlog) for built-in incremental processing and improved maintenance of features.

Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen

This document discusses how to manage PostgreSQL databases with Puppet. It describes using Puppet to deploy PostgreSQL binaries, configure files like postgresql.conf and pg_hba.conf, manage users and permissions, handle database initialization, configure backups with cron jobs, manage replication with Slony, and more. Templates, Hiera, and custom defined resources are used to configure PostgreSQL instances in a consistent and automated way.

MySQL Multi Master ReplicationMoshe Kaplan

This document summarizes and compares several solutions for multi-master replication in MySQL databases: Native MySQL replication, MySQL Cluster (NDB), Galera, and Tungsten. Native MySQL replication supports only limited topologies and has asynchronous replication. MySQL Cluster allows synchronous replication across two data centers but is limited to in-memory tables. Galera provides synchronous, row-based replication across multiple masters with automatic conflict resolution. Tungsten allows asynchronous multi-master replication to different database systems and automatic failover.

Logical replication with pglogicalUmair Shahid

This document discusses logical replication with pglogical. It begins by explaining that pglogical performs row-oriented replication and outputs replication data that can be used in various ways. It then covers the architectures of standalone PostgreSQL, physical replication, and logical replication. The rest of the document discusses key aspects of pglogical such as its output plugin, selective replication capabilities, performance and future plans, and examples of using the output with other applications.

More Related Content

What's hot (19)

Automata Invasionlucenerevolution

EVCache: Lowering Costs for a Low Latency Cache with RocksDBScott Mansfield

Text tagging with finite state transducerslucenerevolution

Kraken mesoscon 2018joeyzhang1989928

Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...PROIDEA

Couchbase live 2016Pierre Mavro

Cassandra ExplainedEric Evans

PostgreSQL and Compressed Documents (pgconf.ru 2018)Aleksander Alekseev

gRPC: Beyond RESTDomingo Suarez Torres

Is It Faster to Go with Redpanda Transactions than Without Them?!ScyllaDB

Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicumScott Tsai

Logging for OpenStack - Elasticsearch, Fluentd, Logstash, KibanaMd Safiyat Reza

erlang 101Gokhan Boranalp

HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon

Toolchain Independent Distributed CompilationDietmar Hauser

Cloud storage: the right way OSS EU 2018Orit Wasserman

Ceph Day Chicago: Using Ceph for Large Hadron Collider Data Ceph Community

Baker: Scaling OVN with Kubernetes API ServerHan Zhou

OVN Controller Incremental ProcessingHan Zhou

Automata Invasionlucenerevolution

EVCache: Lowering Costs for a Low Latency Cache with RocksDBScott Mansfield

Text tagging with finite state transducerslucenerevolution

Kraken mesoscon 2018joeyzhang1989928

Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...PROIDEA

Couchbase live 2016Pierre Mavro

Cassandra ExplainedEric Evans

PostgreSQL and Compressed Documents (pgconf.ru 2018)Aleksander Alekseev

gRPC: Beyond RESTDomingo Suarez Torres

Is It Faster to Go with Redpanda Transactions than Without Them?!ScyllaDB

Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicumScott Tsai

Logging for OpenStack - Elasticsearch, Fluentd, Logstash, KibanaMd Safiyat Reza

erlang 101Gokhan Boranalp

HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon

Toolchain Independent Distributed CompilationDietmar Hauser

Cloud storage: the right way OSS EU 2018Orit Wasserman

Ceph Day Chicago: Using Ceph for Large Hadron Collider Data Ceph Community

Baker: Scaling OVN with Kubernetes API ServerHan Zhou

OVN Controller Incremental ProcessingHan Zhou

Viewers also liked (20)

Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen

MySQL Multi Master ReplicationMoshe Kaplan

Logical replication with pglogicalUmair Shahid

On The Building Of A PostgreSQL ClusterSrihari Sriraman

The document discusses lessons learned from setting up and maintaining a PostgreSQL cluster for a data analytics platform. It describes four stories where problems arose: 1) Implementing automatic failover using Repmgr when the master node failed, 2) The disk filling up faster than expected due to PostgreSQL's MVCC implementation, 3) Being unable to add a new standby node due to missing WAL segments, and 4) Long running queries on the standby node causing conflicts with replication. The key lessons are around using the right tools like Repmgr for replication management, tuning autovacuum, archiving WALs, and addressing hardware limitations for analytics workloads.

Linux tuning to improve PostgreSQL performancePostgreSQL-Consulting

The document discusses tuning Linux settings to enhance PostgreSQL performance by focusing on CPU, memory, and storage configurations. Key areas include optimizing memory usage with huge pages, adjusting swap settings, and fine-tuning flushing processes, as well as scheduler configurations for improved throughput. It emphasizes the need for careful parameter adjustments to fully leverage the efficiency of the modern Linux kernel.

The Security-Enhanced PostgreSQL - "system wide" consistency in access controlKohei KaiGai

The Security-Enhanced PostgreSQL (SE-PostgreSQL) provides system-wide consistency in access controls between the operating system and database management system through a single unified security policy. It implements fine-grained mandatory access controls, including column- and row-level access controls. The goal of SE-PostgreSQL is to allow the database to be part of data flow control schemes and prevent information leakage or manipulation by malicious actors.

Technical Introduction to PostgreSQL and PPASAshnikbiz

The document provides a comprehensive overview of PostgreSQL and EDB's PostgreSQL Advanced Server, detailing its architecture, performance features, Oracle compatibility, and various DBA tools. It highlights PostgreSQL's strong community support, enterprise-class capabilities, and competitive advantages such as scalability and security features. The document also discusses real-world applications and user stories, emphasizing the database's cost-effectiveness and robust performance in high-demand environments.

Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenPostgresOpen

PostgresOpen 2013 A Comparison of PostgreSQL Encryption OptionsFaisal Akber

The document discusses various PostgreSQL encryption options, emphasizing the importance of encrypting sensitive data for compliance and security. It covers methods such as column encryption using pgcrypto, encrypting data partitions, and securing data in transit with SSL or SSH tunneling. Performance benchmarks and real-world use cases demonstrate the practical application of these encryption techniques in scenarios like e-commerce and patient information management.

Streaming Replication (Keynote @ PostgreSQL Conference 2009 Japan)Masao Fujii

This document discusses PostgreSQL's streaming replication feature, which allows for sending changes to a slave database to keep it current, contrasting it with other replication methods. It highlights key aspects such as failover, modes of synchronization, and architecture without a single point of failure, making it easier for installation and use. The proposal for built-in replication aims to simplify the process while ensuring reliability and community involvement in its development.

Streaming huge databases using logical decodingAlexander Shulgin

Enterprise PostgreSQL - EDB's answer to conventional DatabasesAshnikbiz

The document discusses the benefits of using EnterpriseDB's Postgres Plus Advanced Server, particularly focusing on its high performance, security features, and cost-effectiveness compared to proprietary databases like Oracle. It highlights various implementation success stories, showcasing improved scalability, reduced total cost of ownership, and enhanced reporting capabilities. The advanced server offers enterprise-class tools, support for replication, and performance optimization without hidden costs or vendor lock-in.

Ryan Jarvinen Open Shift Talk @ Postgres Open 2013PostgresOpen

Keith Fiske - When PostgreSQL Can't, You Can @ Postgres OpenPostgresOpen

This document summarizes several PostgreSQL extensions developed by Keith Fiske at OmniTI to add capabilities not natively available in PostgreSQL. The extensions discussed include PG Extractor for selective database dumps, PG Job Monitor for monitoring long-running functions, Mimeo for logical replication of individual tables, and PG Partition Manager for automated partitioning of tables. Links are provided for GitHub repositories containing code for each extension.

Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...PostgresOpen

Postgres is an open source relational database management system that provides developers and managers with several advantages over commercial RDBMSs. For developers, Postgres is highly reliable, secure, and extensible due to its liberal license, wide range of built-in features, and support for custom data types and functions. For managers, Postgres offers zero acquisition costs due to its free license, lower maintenance costs compared to commercial databases, and support from many consulting vendors.

Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres OpenPostgresOpen

David Keeney - SQL Database Server Requests from the Browser @ Postgres OpenPostgresOpen

This document describes a universal web application backend called RdbHost that allows writing SQL directly in browser-side JavaScript code. It provides various methods for making database queries from the client-side using JavaScript, including authentication options and ways to whitelist queries for security. The service aims to remove layers from the traditional web stack by handling data access and business logic directly in browser-side code.

Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres OpenPostgresOpen

PostgreSQL uses shared memory structures to coordinate access to data across multiple database processes. The main shared memory structures include shared buffers for caching data pages, a proc array for tracking server processes, lightweight locks for synchronizing access to shared resources, and lock hashes for coordinating locks on database objects. Other shared structures store information for multi-version concurrency control, two-phase commit, subtransactions, the write-ahead log, and background worker synchronization.

Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...PostgresOpen

The document discusses object-relational impedance mismatch that occurs when object-oriented programming languages are used to interact with relational databases. It provides examples from a Ruby on Rails application that uses an ORM called ActiveRecord to show how straightforward object mapping can lead to many database queries being executed to retrieve related data. The document argues this performance issue could be avoided by better integrating database operations directly into objects and methods.

Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Pos...PostgresOpen

Alembic is a schema migration tool that integrates with SQLAlchemy to help developers and database administrators (DBAs) safely manage schema changes in a controlled and repeatable way. It is important for DBAs to work with developers on schema changes to communicate the changes and process. When choosing a migration tool, it is important to consider how often the schema changes and if the tool allows changes to be tested before production. Alembic helps solve problems by enabling disciplined database and code review processes for schema changes and allowing changes to be rolled back if needed.