Distributed Postgres
1 like391 views
This document discusses distributed Postgres including multi-master replication, distributed transactions, and high availability/auto failover. It explores existing implementations like Postgres-XC and proposes a transaction manager API and time-stamp based approach to enable distributed transactions without a central bottleneck. The document also outlines a multimaster implementation built on logical replication, a transaction replay pool, and Raft-based storage for failure handling and distributed deadlocks. Performance is approximately half of standalone Postgres with the same read speeds and capabilities for node recovery and network partition handling.
1 of 17
Download to read offline
![More nodes, higher probability of failure in system.
Possible problems with nodes:
Node stopped (and will not be back);
Node was down small amount of time (and we should bring it
back to operation);
Network partitions (avoid split-brain).
If we want to survive network partitions than we can have not more
than [N/2] - 1 failures.
HA/autofailover
13](https://image.slidesharecdn.com/distributedpostgres-160518144919/85/Distributed-Postgres-13-320.jpg)
Ad
Recommended
Multimaster
MultimasterStas Kelvich This document discusses the objectives, implementation, testing, and roadmap for multi-master replication in Postgres. The key goals are fault tolerance, allowing writes to any node, and compatibility with standalone Postgres. It uses logical replication and a transaction manager based on ClockSI to allow transactions to commit on any node. Testing involves starting docker containers to inject failures like network partitions and verify automatic recovery works as expected. The roadmap includes releasing a public beta, contributing patches to upstream Postgres, and discussing replication of catalog content.
Postgres clusters
Postgres clustersStas Kelvich This document provides an overview of Postgres clustering solutions and distributed Postgres architectures. It discusses master-slave replication, Postgres-XC/XL, Greenplum, CitusDB, pg_shard, BDR, pg_logical, and challenges around distributed transactions, high availability, and multimaster replication. Key points include the tradeoffs of different approaches and an implementation of multimaster replication built on pg_logical and a timestamp-based distributed transaction manager (tsDTM) that provides partition tolerance and automatic failover.
Scylla Summit 2022: Making Schema Changes Safe with Raft
Scylla Summit 2022: Making Schema Changes Safe with RaftScyllaDB The document discusses making schema changes safe in Scylla using the Raft consensus protocol for enhanced consistency in distributed databases. It covers the limitations of schema modifications in Scylla, how Raft ensures state machine replication, and the special features of Scylla Raft, including resilience against network failures and support for large clusters. Additionally, it highlights the process of executing DDL statements and emphasizes the need for extensive testing and considerations regarding cluster topology.
Managing terabytes: When Postgres gets big
Managing terabytes: When Postgres gets bigSelena Deckelmann This document summarizes the challenges and solutions for maintaining large PostgreSQL databases at Emma, including:
- Maintaining terabytes of data across multiple clusters up to version 9.0
- Facing performance issues when the hardware load was pushed to its limits
- Dealing with huge catalogs containing millions of data points that caused slow performance
- Addressing problems like bloat, backups that took hours, system resource exhaustion, and transaction wraparound issues
- Implementing solutions such as scripts to clean up bloat, sharding to a Linux filesystem, and increasing autovacuum thresholds
OpenTSDB: HBaseCon2017
OpenTSDB: HBaseCon2017HBaseCon Chris Larsen, a maintainer of OpenTSDB, discusses the open-source time series database that can efficiently handle a massive volume of data with high precision. The document elaborates on the architecture, schema, use cases, and recent updates of OpenTSDB, including enhancements in query capabilities and support for different storage backends. Additionally, it addresses key challenges related to data aggregation and compactions within the time series database framework.
Ceph at Work in Bloomberg: Object Store, RBD and OpenStack
Ceph at Work in Bloomberg: Object Store, RBD and OpenStackRed_Hat_Storage The document discusses Bloomberg's cloud infrastructure, detailing the use of Ceph for object storage, block storage, and integration with OpenStack. It highlights various resource challenges, architectural components, advantages and disadvantages of Ceph versus ephemeral storage, as well as automation and testing strategies utilized by the organization. The document also outlines future plans for evolving their pod architecture and improving performance and security in their storage systems.
Eventually, Scylla Chooses Consistency
Eventually, Scylla Chooses ConsistencyScyllaDB Konstantin Osipov from ScyllaDB discusses the implementation of Raft log replication in Scylla, focusing on its applications for schema and topology changes, as well as the use of lightweight transactions. The Raft protocol provides strong consistency with reduced network round trips per write, allowing for durable changes with linearizability while maintaining high availability. Scylla plans to leverage Raft for efficient management of schema changes, topology changes, and tablet partitioning, ensuring a consistent state across nodes.
Kafka on ZFS: Better Living Through Filesystems
Kafka on ZFS: Better Living Through Filesystems confluent The document discusses the benefits of using ZFS with Kafka, highlighting that ZFS improves Kafka's performance and cost-effectiveness through enhanced filesystem caching and efficient use of I/O devices. It explains how Kafka operates differently from Redis and details various I/O modes and cache management strategies. The author emphasizes key ZFS features such as automatic checksumming, compression, and advanced caching mechanisms that contribute to optimized data handling for Kafka applications.
An intro to Ceph and big data - CERN Big Data Workshop
An intro to Ceph and big data - CERN Big Data WorkshopPatrick McGarry Ceph is an open-source distributed storage system that provides scalable object, block, and file storage in a single unified platform. It uses a technique called CRUSH to automatically distribute data across clusters of commodity servers and provides self-healing capabilities through data replication. Ceph's unified storage platform includes RADOS, an object store; RBD for block storage; CephFS for distributed file storage; and RADOSGW for cloud storage compatibility. It is designed for large-scale deployments of 10s to 10,000s of nodes using heterogeneous hardware.
Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase HBaseCon This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase UpdateHBaseCon OpenTSDB is an open source distributed time series database for storing large amounts of metrics data and performing fast queries. It is scalable and can store trillions of data points across multiple servers. Data is stored in HBase and queries are performed using a simple query language. OpenTSDB 2.x includes new features like salting to distribute writes across servers, compact storage formats using column appends, and downsampling to fill in missing data during aggregation.
Bluestore
BluestorePatrick McGarry This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and enables more efficient features like checksumming, compression, and cloning. BlueStore addresses consistency and performance problems that arose with previous approaches like FileStore and NewStore.
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community This document summarizes Noah Watkins' presentation on building a distributed shared log using Ceph. The key points are:
1) Noah discusses how shared logs are challenging to scale due to the need to funnel all writes through a total ordering engine. This bottlenecks performance.
2) CORFU is introduced as a shared log design that decouples I/O from ordering by striping the log across flash devices and using a sequencer to assign positions.
3) Noah then explains how the components of CORFU can be mapped onto Ceph, using RADOS object classes, librados, and striping policies to implement the shared log without requiring custom hardware interfaces.
4) ZLog is presented
Introduction to Cassandra: Replication and Consistency
Introduction to Cassandra: Replication and ConsistencyBenjamin Black The document discusses the replication and consistency features of Cassandra, highlighting its dynamo-like and bigtable-like architectures. It covers critical concepts such as the number of data copies, consistency levels, and data placement strategies in a distributed hash table system. Additional topics include partitioning and the importance of timestamps in managing data consistency.
Cassandra at teads
Cassandra at teadsRomain Hardouin The document outlines the architecture and operational strategies of Teads utilizing Apache Cassandra, detailing the company's structure, metrics, and performance optimizations. It discusses provisioning, monitoring, tuning, and the use of various tools to manage the Cassandra database effectively. Additionally, it addresses challenges faced and improvements made, including a light fork of Cassandra for specific enhancements.
Evolving Virtual Networking with IO Visor
Evolving Virtual Networking with IO VisorLarry Lang The document discusses the evolution of virtual networking with an emphasis on the io visor project and its integration with eBPF for high-performance virtual network monitoring and data planes. It highlights the need for advanced monitoring techniques in dynamic application environments and presents Huawei's approach to cloud-native NFV using micro data plane containers. Additionally, it addresses the challenges in creating competitive network I/O frameworks within the Linux kernel to enhance performance across various architectures.
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon The document discusses the challenges and strategies involved in upgrading HBase at Pinterest from version 0.94 to 1.2, addressing the impossibility of a live upgrade without downtime. The successful upgrade included building a new cluster, setting up replication, exporting snapshots, and ensuring compatibility between client versions. Performance improvements and solutions for data consistency and latency were implemented throughout the process.
Debug generic process
Debug generic processVipin Varghese This document discusses various Linux debugging tools including:
1. SIMD, cache monitoring, firmware checks, NUMA memory, interrupts using tools like lstopo, ethtool, lspci, and lshw.
2. Using GDB for debugging with features like breakpoints, disassembly, and core file generation.
3. Tools like strace, ltrace, nm, objdump, and readelf for system call tracing, library call tracing, symbol tables, and object file analysis.
4. Techniques like LD_PRELOAD, ulimit, and perf for custom debugging and performance analysis.
OpenTSDB 2.0
OpenTSDB 2.0HBaseCon OpenTSDB 2.0 is a distributed, scalable, open-source time series database designed to store trillions of data points without losing precision, leveraging HBase for scalability. The document outlines various use cases, including systems monitoring, sensor data, and application performance metrics from companies like OVH and Yahoo, detailing its architecture for writing and querying data. It also discusses improvements in UID assignment, schema design, and the future developments planned for OpenTSDB, such as enhanced query capabilities and integration with time series searching.
SignalFx: Making Cassandra Perform as a Time Series Database
SignalFx: Making Cassandra Perform as a Time Series DatabaseDataStax Academy The document discusses optimizing Cassandra to function as a time series database, focusing on improving performance through various versions. Key strategies include using buffered writes, packed writes, and a redo-log system to enhance data ingestion efficiency, resulting in a significant performance increase for write-heavy workloads. The findings emphasize that effective workload management can drastically improve database performance, achieving up to 4x improvement.
pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)
pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)Wei Shan Ang The document discusses alternatives to Pacemaker/Corosync for managing high availability PostgreSQL clusters. It introduces pg_bouncer for connection pooling, Linux's UCARP for virtual IP failover, and 2ndQuadrant's repmgr for managing replication and automatic failover. It notes the challenges with Pacemaker/Corosync and provides an agenda that includes an introduction, challenges with Pacemaker/Corosync, discussing pg_bouncer, UCARP, repmgr, and a demo. Logs from Pacemaker/Corosync are also included showing failed operations and failover.
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and Beyond
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and BeyondScyllaDB The document discusses the future of consensus mechanisms in ScyllaDB 5.0, focusing on improving fault tolerance and consistency through the integration of the Raft protocol for managing token metadata and topology changes. It highlights challenges with the current design, including eventual consistency issues and the need for automated topology management to relieve admin burdens. The proposed enhancements aim to ensure reliable database operations where consistency is maintained even under various stress conditions, enabling dynamic scaling and load balancing without manual intervention.
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devicesMichael Stack The document summarizes a presentation on using persistent memory technology with Apache HBase. It describes the current HBase model using volatile memory and write-ahead logs, as well as persistent memory technology capabilities. The presentation proposes using persistent memory with HBase to store data non-volatility, eliminate write-ahead logs for faster performance, and increase availability through replica regions in memory. It shows performance test results finding over 2x higher throughput and up to 104x lower maximum latency without write-ahead logs when using persistent memory with HBase.
Pgxc scalability pg_open2012
Pgxc scalability pg_open2012Ashutosh Bapat Postgres-XC is a shared-nothing PostgreSQL cluster that scales horizontally by distributing data across multiple nodes. It supports both replicated and distributed tables. Replicated tables store each row on all nodes, while distributed tables store each row on a single node according to the distribution strategy. The document discusses Postgres-XC's architecture, data distribution techniques, query processing, and provides an example of how to distribute the tables in the TPC-B benchmark schema for optimal performance.
Ceph data services in a multi- and hybrid cloud world
Ceph data services in a multi- and hybrid cloud worldSage Weil The document discusses the evolution and capabilities of Ceph data services in multi- and hybrid cloud environments, emphasizing unified storage solutions including block, file, and object storage. It highlights the challenges of data placement, mobility, and policy management in cloud-native applications, advocating for an automated and orchestration-focused approach to data management. Key features and priorities of Ceph are detailed, including its release schedule, data services, and the integration of Kubernetes for enhanced application orchestration.
CephFS update February 2016
CephFS update February 2016John Spray The document provides an overview and update on CephFS, the distributed file system implemented in Ceph. Key points include new capabilities in the Jewel release like improved scrub/repair functionality to handle metadata damage, fine-grained authorization using new MDS auth caps, and experimental support for multiple CephFS filesystems backed by a single RADOS storage cluster. It also discusses ongoing work integrating CephFS with OpenStack Manila for shared file system provisioning and new tools like CephFSVolumeClient.
Latest performance changes by Scylla - Project optimus / Nolimits
Latest performance changes by Scylla - Project optimus / Nolimits ScyllaDB The document discusses performance enhancements in Scylla, focusing on faster read performance from large partitions, improved data structures, and advancements in the I/O scheduler. It highlights solutions to latency spikes and better data handling, allowing for efficient operations even in large partition scenarios. Key metrics and improvements, such as increased query throughput and read amplification reduction, are also presented.
Tungsten University: Setup & Operate Tungsten Replicator
Tungsten University: Setup & Operate Tungsten ReplicatorContinuent The document provides an overview of Tungsten Replicator, a product by Continuent for clustering and replication of MySQL databases. It covers installation processes, commands for checking and altering replication configurations, and troubleshooting steps for common issues. Additional resources and community engagement opportunities are also mentioned.
Postgres-XC Write Scalable PostgreSQL Cluster
Postgres-XC Write Scalable PostgreSQL ClusterMason Sharp Postgres-XC is a PostgreSQL-based database cluster designed for write scalability, allowing seamless read and write operations across multiple coordinators without a master-slave configuration. It supports consistent database views and ACID properties while utilizing global transaction management to maintain data integrity. The document also discusses various components, data distribution strategies, and considerations regarding transaction management and high availability.
Flexible Indexing with Postgres
Flexible Indexing with PostgresEDB The document presents a comprehensive overview of indexing options in PostgreSQL, discussing traditional and advanced indexing methods such as expression, partial, and bitmap indexes. It details various non-B-tree index types like GIN, GiST, and SP-GiST, which cater to complex data types and queries. Additionally, the presentation highlights best practices for index usage to optimize database performance.
More Related Content
What's hot (20)
An intro to Ceph and big data - CERN Big Data Workshop
An intro to Ceph and big data - CERN Big Data WorkshopPatrick McGarry Ceph is an open-source distributed storage system that provides scalable object, block, and file storage in a single unified platform. It uses a technique called CRUSH to automatically distribute data across clusters of commodity servers and provides self-healing capabilities through data replication. Ceph's unified storage platform includes RADOS, an object store; RBD for block storage; CephFS for distributed file storage; and RADOSGW for cloud storage compatibility. It is designed for large-scale deployments of 10s to 10,000s of nodes using heterogeneous hardware.
Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase HBaseCon This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase UpdateHBaseCon OpenTSDB is an open source distributed time series database for storing large amounts of metrics data and performing fast queries. It is scalable and can store trillions of data points across multiple servers. Data is stored in HBase and queries are performed using a simple query language. OpenTSDB 2.x includes new features like salting to distribute writes across servers, compact storage formats using column appends, and downsampling to fill in missing data during aggregation.
Bluestore
BluestorePatrick McGarry This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and enables more efficient features like checksumming, compression, and cloning. BlueStore addresses consistency and performance problems that arose with previous approaches like FileStore and NewStore.
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community This document summarizes Noah Watkins' presentation on building a distributed shared log using Ceph. The key points are:
1) Noah discusses how shared logs are challenging to scale due to the need to funnel all writes through a total ordering engine. This bottlenecks performance.
2) CORFU is introduced as a shared log design that decouples I/O from ordering by striping the log across flash devices and using a sequencer to assign positions.
3) Noah then explains how the components of CORFU can be mapped onto Ceph, using RADOS object classes, librados, and striping policies to implement the shared log without requiring custom hardware interfaces.
4) ZLog is presented
Introduction to Cassandra: Replication and Consistency
Introduction to Cassandra: Replication and ConsistencyBenjamin Black The document discusses the replication and consistency features of Cassandra, highlighting its dynamo-like and bigtable-like architectures. It covers critical concepts such as the number of data copies, consistency levels, and data placement strategies in a distributed hash table system. Additional topics include partitioning and the importance of timestamps in managing data consistency.
Cassandra at teads
Cassandra at teadsRomain Hardouin The document outlines the architecture and operational strategies of Teads utilizing Apache Cassandra, detailing the company's structure, metrics, and performance optimizations. It discusses provisioning, monitoring, tuning, and the use of various tools to manage the Cassandra database effectively. Additionally, it addresses challenges faced and improvements made, including a light fork of Cassandra for specific enhancements.
Evolving Virtual Networking with IO Visor
Evolving Virtual Networking with IO VisorLarry Lang The document discusses the evolution of virtual networking with an emphasis on the io visor project and its integration with eBPF for high-performance virtual network monitoring and data planes. It highlights the need for advanced monitoring techniques in dynamic application environments and presents Huawei's approach to cloud-native NFV using micro data plane containers. Additionally, it addresses the challenges in creating competitive network I/O frameworks within the Linux kernel to enhance performance across various architectures.
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon The document discusses the challenges and strategies involved in upgrading HBase at Pinterest from version 0.94 to 1.2, addressing the impossibility of a live upgrade without downtime. The successful upgrade included building a new cluster, setting up replication, exporting snapshots, and ensuring compatibility between client versions. Performance improvements and solutions for data consistency and latency were implemented throughout the process.
Debug generic process
Debug generic processVipin Varghese This document discusses various Linux debugging tools including:
1. SIMD, cache monitoring, firmware checks, NUMA memory, interrupts using tools like lstopo, ethtool, lspci, and lshw.
2. Using GDB for debugging with features like breakpoints, disassembly, and core file generation.
3. Tools like strace, ltrace, nm, objdump, and readelf for system call tracing, library call tracing, symbol tables, and object file analysis.
4. Techniques like LD_PRELOAD, ulimit, and perf for custom debugging and performance analysis.
OpenTSDB 2.0
OpenTSDB 2.0HBaseCon OpenTSDB 2.0 is a distributed, scalable, open-source time series database designed to store trillions of data points without losing precision, leveraging HBase for scalability. The document outlines various use cases, including systems monitoring, sensor data, and application performance metrics from companies like OVH and Yahoo, detailing its architecture for writing and querying data. It also discusses improvements in UID assignment, schema design, and the future developments planned for OpenTSDB, such as enhanced query capabilities and integration with time series searching.
SignalFx: Making Cassandra Perform as a Time Series Database
SignalFx: Making Cassandra Perform as a Time Series DatabaseDataStax Academy The document discusses optimizing Cassandra to function as a time series database, focusing on improving performance through various versions. Key strategies include using buffered writes, packed writes, and a redo-log system to enhance data ingestion efficiency, resulting in a significant performance increase for write-heavy workloads. The findings emphasize that effective workload management can drastically improve database performance, achieving up to 4x improvement.
pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)
pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)Wei Shan Ang The document discusses alternatives to Pacemaker/Corosync for managing high availability PostgreSQL clusters. It introduces pg_bouncer for connection pooling, Linux's UCARP for virtual IP failover, and 2ndQuadrant's repmgr for managing replication and automatic failover. It notes the challenges with Pacemaker/Corosync and provides an agenda that includes an introduction, challenges with Pacemaker/Corosync, discussing pg_bouncer, UCARP, repmgr, and a demo. Logs from Pacemaker/Corosync are also included showing failed operations and failover.
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and Beyond
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and BeyondScyllaDB The document discusses the future of consensus mechanisms in ScyllaDB 5.0, focusing on improving fault tolerance and consistency through the integration of the Raft protocol for managing token metadata and topology changes. It highlights challenges with the current design, including eventual consistency issues and the need for automated topology management to relieve admin burdens. The proposed enhancements aim to ensure reliable database operations where consistency is maintained even under various stress conditions, enabling dynamic scaling and load balancing without manual intervention.
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devicesMichael Stack The document summarizes a presentation on using persistent memory technology with Apache HBase. It describes the current HBase model using volatile memory and write-ahead logs, as well as persistent memory technology capabilities. The presentation proposes using persistent memory with HBase to store data non-volatility, eliminate write-ahead logs for faster performance, and increase availability through replica regions in memory. It shows performance test results finding over 2x higher throughput and up to 104x lower maximum latency without write-ahead logs when using persistent memory with HBase.
Pgxc scalability pg_open2012
Pgxc scalability pg_open2012Ashutosh Bapat Postgres-XC is a shared-nothing PostgreSQL cluster that scales horizontally by distributing data across multiple nodes. It supports both replicated and distributed tables. Replicated tables store each row on all nodes, while distributed tables store each row on a single node according to the distribution strategy. The document discusses Postgres-XC's architecture, data distribution techniques, query processing, and provides an example of how to distribute the tables in the TPC-B benchmark schema for optimal performance.
Ceph data services in a multi- and hybrid cloud world
Ceph data services in a multi- and hybrid cloud worldSage Weil The document discusses the evolution and capabilities of Ceph data services in multi- and hybrid cloud environments, emphasizing unified storage solutions including block, file, and object storage. It highlights the challenges of data placement, mobility, and policy management in cloud-native applications, advocating for an automated and orchestration-focused approach to data management. Key features and priorities of Ceph are detailed, including its release schedule, data services, and the integration of Kubernetes for enhanced application orchestration.
CephFS update February 2016
CephFS update February 2016John Spray The document provides an overview and update on CephFS, the distributed file system implemented in Ceph. Key points include new capabilities in the Jewel release like improved scrub/repair functionality to handle metadata damage, fine-grained authorization using new MDS auth caps, and experimental support for multiple CephFS filesystems backed by a single RADOS storage cluster. It also discusses ongoing work integrating CephFS with OpenStack Manila for shared file system provisioning and new tools like CephFSVolumeClient.
Latest performance changes by Scylla - Project optimus / Nolimits
Latest performance changes by Scylla - Project optimus / Nolimits ScyllaDB The document discusses performance enhancements in Scylla, focusing on faster read performance from large partitions, improved data structures, and advancements in the I/O scheduler. It highlights solutions to latency spikes and better data handling, allowing for efficient operations even in large partition scenarios. Key metrics and improvements, such as increased query throughput and read amplification reduction, are also presented.
Tungsten University: Setup & Operate Tungsten Replicator
Tungsten University: Setup & Operate Tungsten ReplicatorContinuent The document provides an overview of Tungsten Replicator, a product by Continuent for clustering and replication of MySQL databases. It covers installation processes, commands for checking and altering replication configurations, and troubleshooting steps for common issues. Additional resources and community engagement opportunities are also mentioned.
Viewers also liked (20)
Postgres-XC Write Scalable PostgreSQL Cluster
Postgres-XC Write Scalable PostgreSQL ClusterMason Sharp Postgres-XC is a PostgreSQL-based database cluster designed for write scalability, allowing seamless read and write operations across multiple coordinators without a master-slave configuration. It supports consistent database views and ACID properties while utilizing global transaction management to maintain data integrity. The document also discusses various components, data distribution strategies, and considerations regarding transaction management and high availability.
Flexible Indexing with Postgres
Flexible Indexing with PostgresEDB The document presents a comprehensive overview of indexing options in PostgreSQL, discussing traditional and advanced indexing methods such as expression, partial, and bitmap indexes. It details various non-B-tree index types like GIN, GiST, and SP-GiST, which cater to complex data types and queries. Additionally, the presentation highlights best practices for index usage to optimize database performance.
Postgres-XC as a Key Value Store Compared To MongoDB
Postgres-XC as a Key Value Store Compared To MongoDBMason Sharp The document outlines the capabilities and advantages of Postgres-XC, a distributed key-value store based on PostgreSQL, highlighting its support for various data types like XML, HSTORE, and JSON. It compares Postgres-XC's performance with MongoDB in terms of insert, read, and update operations, indicating that while MongoDB excels in low concurrency inserts, Postgres-XC benefits from scalability and PostgreSQL's robust features. Overall, it advocates for using Postgres-XC for projects that require schema-less functionality and the reliability of ACID transactions.
How the Postgres Query Optimizer Works
How the Postgres Query Optimizer WorksEDB The document discusses the PostgreSQL query optimizer, which is essential for interpreting SQL queries and determining the most efficient execution methods. It emphasizes the use of the EXPLAIN command to analyze how the optimizer works, showcasing different scanning methods and join algorithms. The presentation also highlights PostgreSQL's strong adherence to standards and its extensive community support.
1
1Yahia Mahmoud يتناول المستند دورة تعليمية حول استخدام قواعد البيانات mysql، بدءاً من التحميل والتثبيت، إلى تعليمات لغة SQL الأساسية مثل إنشاء وإدارة قواعد البيانات والجداول. ويشرح كيفية التعامل مع الاستعلامات الأساسية مثل إدخال، حذف، تحديث البيانات، وكيفية تعديل كلمات المرور. كما يتضمن أمثلة عملية على التعليمات البرمجية المطلوبة وطرق التأكد من العمليات المنجزة.
Postgres-XC: Symmetric PostgreSQL Cluster
Postgres-XC: Symmetric PostgreSQL ClusterPavan Deolasee The document provides an overview of Postgres-XC, a symmetric PostgreSQL cluster solution designed to enhance scalability and performance using shared-nothing architecture. It covers the project's history, architecture, data distribution strategies, and the roles of major components like the global transaction manager and coordinators. Key features of Postgres-XC include global consistency, support for read/write scalability, and efficient query processing across multiple nodes.
Best Practices for Database Schema Design
Best Practices for Database Schema DesignIron Speed The document provides best practices for database schema design to optimize use with the Iron Speed Designer application development tool. It recommends normalizing data, using separate lookup tables, declaring primary and foreign keys, creating views and indexes, and using naming conventions. Following these practices results in Iron Speed Designer generating more sophisticated and easily maintained web applications from the database schema.
5 data storage_and_indexing
5 data storage_and_indexingUtkarsh De The document discusses file organization and indexing in relational database management systems (RDBMS), highlighting the importance of disk space management and access methods for efficient data retrieval. It covers various data storage structures such as fixed-length and variable-length records, along with organization methods including heap files, sorted files, and hashed files. Additionally, it explains the processes for inserting, deleting, and updating records, emphasizing the impact of file organization on performance.
Managing your tech career
Managing your tech careerGreg Jensen This document summarizes a talk on managing your tech career and tracking your tech skills. The talk covered finding your path in the industry, building your personal brand, evolving your mindset, and tracking emerging technologies. It discussed understanding the different roles and paths available, developing your career narrative, maintaining an online presence, participating in communities, and dealing with imposter syndrome. The talk emphasized the importance of continuous learning, building a technology radar to track new tools and platforms, and appreciating how your skills fit within the broader tech ecosystem.
1 introduction
1 introductionUtkarsh De The document provides an overview of database systems, defining a database as a collection of related data that supports the activities of an enterprise. It discusses the role of Database Management Systems (DBMS), which facilitate data organization, retrieval, and management while overcoming challenges in traditional file systems. The content also covers various data models, the three-schema architecture, and the development process of database systems, emphasizing the need for flexibility and independence between data and schema.
4 the sql_standard
4 the sql_standardUtkarsh De The document discusses the SQL standard and its components. It describes how SQL is used to define schemas, manipulate data, write queries involving single or multiple tables, and perform other operations. Key topics covered include data definition language, data manipulation language, data types, integrity constraints, queries, subqueries, and set operations in SQL. Examples of SQL commands for creating tables, inserting data, and writing various types of queries are also provided.
6 relational schema_design
6 relational schema_designUtkarsh De The document discusses the importance of database design and the criteria for evaluating schema quality, including the concept of normal forms to avoid design issues. It outlines various anomalies associated with poor schema designs and provides definitions and examples of functional dependencies, normal forms (1NF, 2NF, 3NF, BCNF), and inference rules. Key principles such as soundness and completeness of inference rules for deriving new functional dependencies are also highlighted.
Normalization
NormalizationRandy Riness @ South Puget Sound Community College Normalizing a database involves organizing data to:
1) Avoid duplicate values and inconsistent dependencies by separating data into multiple tables.
2) Ensure each table describes a single entity or "thing".
3) Achieve third normal form where data is organized into tables such that each non-key attribute is dependent only on the primary key.
Webinar: Build an Application Series - Session 2 - Getting Started
Webinar: Build an Application Series - Session 2 - Getting StartedMongoDB The document outlines a webinar series on MongoDB basics, focusing on schema design and application architecture using a sample application called 'mycms'. It covers topics including data modeling with documents, user interactions, and implementing a RESTful API using Python and Flask. Key highlights include flexible schema design, handling comments and user profiles, and considerations for scalability and performance.
3 relational model
3 relational modelUtkarsh De The document discusses the relational model of databases as proposed by Edgar F. Codd, highlighting its mathematical basis and the foundation it laid for database design and SQL. It explains key concepts such as relations, tuples, keys (including primary and foreign keys), integrity constraints, and basic operations in relational algebra. Additionally, the document covers various operations such as selection, projection, union, intersection, and joins, which are crucial for querying relational databases.
MySQL Replication: Pros and Cons
MySQL Replication: Pros and ConsRachel Li The document discusses different types of MySQL replication including asynchronous, semi-synchronous, and synchronous. It provides pros and cons of each type and describes how they handle transaction ordering, parallelism, flow control, and consistency. The key points are that asynchronous replication has potential for data loss and stale reads, semi-synchronous reduces but does not eliminate data loss risk, while SchoonerSQL's synchronous replication guarantees no data loss and failover without stalled transactions.
Week3 Lecture Database Design
Week3 Lecture Database DesignKevin Element The document discusses database design principles, noting that designers should understand technology limitations, gather all relevant data about their project, and map out the database design on paper before implementing it in a computer system. It also provides examples of attributes and tables to store different types of data, emphasizing the importance of planning the database design to account for all necessary data entities and attributes.
Database Design
Database Designlearnt The document discusses database design and the design process. It explains that database design involves determining the logical structure of tables and relationships between data elements. The design process consists of steps like determining relationships between data, dividing information into tables, specifying primary keys, and applying normalization rules. The document also covers entity-relationship diagrams and designing inputs and outputs, including input controls and designing report formats.
2 entity relationship_model
2 entity relationship_modelUtkarsh De The document discusses the entity-relationship (E/R) model, a conceptual framework for database design introduced by Peter P. Chen in the 1970s. It covers key concepts such as entities, attributes, relationships, and various types of participation and constraints, providing visual notation for each component. Additionally, it examines structural constraints and design choices that impact how relationships and entities are defined in database systems.
English gcse final tips
English gcse final tipsmrhoward12 This document provides tips for the English GCSE final exam. It recommends spending an hour on the writing section, with 25 minutes for the first question and 35 minutes for the second. Students are advised to plan their writing by identifying the Genre, Audience, and Purpose (GAP) before starting. Examples of genres include letters, newspaper articles, and blog posts. The document also provides language tips for writing effective openings and conclusions to writing tasks and for structuring paragraphs with topic sentences and connectives. It includes sample questions and concludes with a practice question to write a persuasive leaflet.
Ad
Similar to Distributed Postgres (10)
Introduction to Postrges-XC
Introduction to Postrges-XCAshutosh Bapat Postgres-XC is a write-scalable, synchronous multi-master PostgreSQL cluster that allows adding servers for write scaling. It provides a single consistent view across the cluster and immediate visibility of writes. The architecture includes coordinators that clients connect to, global transaction managers that coordinate transactions, and datanodes that store tables and execute queries in a loosely coupled, shared-nothing approach. The project is developing new features toward version 1.0 including subqueries and changing data distribution strategies.
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)Aleksander Alekseev This document summarizes Aleksander Alekseev's talk on PostgreSQL sharding and high availability. The talk introduces replication in PostgreSQL, including physical and logical replication. It discusses solutions for high availability like Stolon and PostgreSQL Multimaster, as well as existing solutions for sharding like manual sharding, Citus, Greenplum, and pg_shardman. It also briefly covers other databases like Amazon Aurora and CockroachDB that are inspired by PostgreSQL but with additional distributed functionality.
Supersized PostgreSQL: Postgres-XL for Scale-Out OLTP and Big Data Analytics
Supersized PostgreSQL: Postgres-XL for Scale-Out OLTP and Big Data Analyticsmason_s Postgres-XL is an open-source, scale-out relational database management system designed for massive parallel processing and high scalability in OLTP and big data analytics. It enables efficient data distribution across nodes with cluster-wide ACID properties, and supports various programming connectors and features such as multi-version concurrency control and geo-spatial data types. While it offers many advantages over traditional PostgreSQL, it currently lacks built-in high availability and requires external solutions for such features.
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Michael Renner PostgreSQL is a free and open-source relational database management system that provides high performance and reliability. It supports replication through various methods including log-based asynchronous master-slave replication, which the presenter recommends as a first option. The upcoming PostgreSQL 9.4 release includes improvements to replication such as logical decoding and replication slots. Future releases may add features like logical replication consumers and SQL MERGE statements. The presenter took questions at the end and provided additional resources on PostgreSQL replication.
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQLYugabyte The document discusses YugabyteDB, a transactional database designed for high-performance cloud services, founded by engineers with experience at companies like Facebook and Oracle. It covers its unique architecture, which features automatic sharding, fault tolerance, and a focus on cloud-native deployment, as well as the transaction processing capabilities, including single and distributed transactions. A workshop agenda is outlined, including practical exercises aimed at demonstrating YugabyteDB's capabilities and functionality.
Eventual Consitency with CRDTS
Eventual Consitency with CRDTSSamir Bessalah The document discusses the use of CRDTs (Convergent Replicated Data Types) to achieve eventual consistency in distributed systems without consensus. It describes the CAP theorem and challenges with achieving consistency in a distributed manner. CRDTs are introduced as a way to build datatypes that can automatically resolve conflicts as they propagate through replicas. Examples of commonly used CRDTs include registers, counters, sets and graphs. The document outlines some real-world implementations of CRDTs and notes their limitations.
The Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus StoryHanna Kelman The document discusses the challenges of distributing PostgreSQL databases, focusing on high availability, replication, and the complexities of distributed transactions. It highlights the Citus Data solution as a scalable extension to PostgreSQL that enables efficient database management through sharding and replication. Key aspects include methods for achieving high availability, handling deadlocks, and the architecture of distributed databases.
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...Citus Data The document discusses the challenges of distributing PostgreSQL databases, emphasizing high availability, replication methods, and the management of distributed transactions. It outlines the architecture of Citus, a PostgreSQL extension that enables horizontal scaling through sharding and replication approaches. The presentation also touches on concurrency, deadlock detection, and compares various methods for building distributed databases.
Blockchain meets database
Blockchain meets databaseYongraeJo 1) The document describes two approaches - Order-Execute (OX) and Execute-Order in parallel (XO) - for building a blockchain platform using an existing relational database.
2) In the OX approach, transactions are ordered before execution, while in the XO approach transactions are executed in parallel without prior knowledge of ordering.
3) The approaches are implemented in PostgreSQL with 4000 lines of code. Serializable snapshot isolation is achieved using heuristics like abort during commit to detect conflicts.
Redis Day TLV 2018 - 10 Reasons why Redis should be your Primary Database
Redis Day TLV 2018 - 10 Reasons why Redis should be your Primary DatabaseRedis Labs Redis should be the primary database for three key reasons:
1. It is extremely fast and scales linearly even as more nodes are added. Performance remains highly consistent across scaling tests.
2. It is highly available with techniques like quorum-based replication across nodes rather than shards, in-memory replication for faster performance, and watchdogs to monitor the cluster.
3. It uses CRDTs to provide strong eventual consistency for active-active replication across replicas in under 1 millisecond, solving the conflicts that arise much faster than other approaches.
Ad
Recently uploaded (20)
Step by step guide to install Flutter and Dart
Step by step guide to install Flutter and DartS Pranav (Deepu) Flutter is basically Google’s portable user
interface (UI) toolkit, used to build and
develop eye-catching, natively-built
applications for mobile, desktop, and web,
from a single codebase. Flutter is free, open-
sourced, and compatible with existing code. It
is utilized by companies and developers
around the world, due to its user-friendly
interface and fairly simple, yet to-the-point
commands.
DevOps for AI: running LLMs in production with Kubernetes and KubeFlow
DevOps for AI: running LLMs in production with Kubernetes and KubeFlowAarno Aukia Presented by Aarno Aukia on June 4th, 2025, at Kubernetes Community Days KCD New York
GDG Douglas - Google AI Agents: Your Next Intern?
GDG Douglas - Google AI Agents: Your Next Intern?felipeceotto Presentation done at the GDG Douglas event for June 2025.
A first look at Google's new Agent Development Kit.
Agent Development Kit is a new open-source framework from Google designed to simplify the full stack end-to-end development of agents and multi-agent systems.
Making significant Software Architecture decisions
Making significant Software Architecture decisionsBert Jan Schrijver Presented at the iSAQB Software Architecture Community NL meetup on 12-6-2025
Generative Artificial Intelligence and its Applications
Generative Artificial Intelligence and its ApplicationsSandeepKS52 The exploration of generative AI begins with an overview of its fundamental concepts, highlighting how these technologies create new content and ideas by learning from existing data. Following this, the focus shifts to the processes involved in training and fine-tuning models, which are essential for enhancing their performance and ensuring they meet specific needs. Finally, the importance of responsible AI practices is emphasized, addressing ethical considerations and the impact of AI on society, which are crucial for developing systems that are not only effective but also beneficial and fair.
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWSBradBedford3 Creating meaningful, real-time engagement across channels is essential to building lasting business relationships. Discover how AWS, in collaboration with Deloitte, set up one of Adobe's first instances of Journey Optimizer B2B Edition to revolutionize customer journeys for B2B audiences.
This session will share the use cases the AWS team has the implemented leveraging Adobe's Journey Optimizer B2B alongside Marketo Engage and Real-Time CDP B2B to deliver unified, personalized experiences and drive impactful engagement.
They will discuss how they are positioning AJO B2B in their marketing strategy and how AWS is imagining AJO B2B and Marketo will continue to work together in the future.
Whether you’re looking to enhance customer journeys or scale your B2B marketing efforts, you’ll leave with a clear view of what can be achieved to help transform your own approach.
Speakers:
Britney Young Senior Technical Product Manager, AWS
Erine de Leeuw Technical Product Manager, AWS
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...BradBedford3 Inspired by the Adobe Summit hands-on lab, Optimize Your Marketo Instance Performance, review the recording from June 5th to learn best practices that can optimize your smart campaign and smart list processing time, inefficient practices to try to avoid, and tips and tricks for keeping your instance running smooth!
You will learn:
How smart campaign queueing works, how flow steps are prioritized, and configurations that slow down smart campaign processing.
Best practices for smart list and smart campaign configurations that yield greater reliability and processing efficiencies.
Generally recommended timelines for reviewing instance performance: walk away from this session with a guideline of what to review in Marketo and how often to review it.
This session will be helpful for any Marketo administrator looking for opportunities to improve and streamline their instance performance. Be sure to watch to learn best practices and connect with your local Marketo peers!
wAIred_RabobankIgniteSession_12062025.pptx
wAIred_RabobankIgniteSession_12062025.pptxSimonedeGijt In today's world, artificial intelligence (AI) is transforming the way we learn.
This talk will explore how we can use AI tools to enhance our learning experiences, by looking at some (recent) research that has been done on the matter.
But as we embrace these new technologies, we must also ask ourselves:
Are we becoming less capable of thinking for ourselves?
Do these tools make us smarter, or do they risk dulling our critical thinking skills?
This talk will encourage us to think critically about the role of AI in our education. Together, we will discover how to use AI to support our learning journey while still developing our ability to think critically.
Application Modernization with Choreo - The AI-Native Internal Developer Plat...
Application Modernization with Choreo - The AI-Native Internal Developer Plat...WSO2 In this slide deck, we explore the challenges and best practices of application modernization. We also deep dive how an internal developer platform as a service like Choreo can fast track your modernization journey with AI capabilities and end-to-end workflow automation.
Plooma is a writing platform to plan, write, and shape books your way
Plooma is a writing platform to plan, write, and shape books your wayPlooma Plooma is your all in one writing companion, designed to support authors at every twist and turn of the book creation journey. Whether you're sketching out your story's blueprint, breathing life into characters, or crafting chapters, Plooma provides a seamless space to organize all your ideas and materials without the overwhelm. Its intuitive interface makes building rich narratives and immersive worlds feel effortless.
Packed with powerful story and character organization tools, Plooma lets you track character development and manage world building details with ease. When it’s time to write, the distraction-free mode offers a clean, minimal environment to help you dive deep and write consistently. Plus, built-in editing tools catch grammar slips and style quirks in real-time, polishing your story so you don’t have to juggle multiple apps.
What really sets Plooma apart is its smart AI assistant - analyzing chapters for continuity, helping you generate character portraits, and flagging inconsistencies to keep your story tight and cohesive. This clever support saves you time and builds confidence, especially during those complex, detail packed projects.
Getting started is simple: outline your story’s structure and key characters with Plooma’s user-friendly planning tools, then write your chapters in the focused editor, using analytics to shape your words. Throughout your journey, Plooma’s AI offers helpful feedback and suggestions, guiding you toward a polished, well-crafted book ready to share with the world.
With Plooma by your side, you get a powerful toolkit that simplifies the creative process, boosts your productivity, and elevates your writing - making the path from idea to finished book smoother, more fun, and totally doable.
Get Started here: https://www.plooma.ink/
Advanced Token Development - Decentralized Innovation
Advanced Token Development - Decentralized Innovationarohisinghas720 The world of blockchain is evolving at a fast pace, and at the heart of this transformation lies advanced token development. No longer limited to simple digital assets, today’s tokens are programmable, dynamic, and play a crucial role in driving decentralized applications across finance, governance, gaming, and beyond.
Wondershare PDFelement Pro 11.4.20.3548 Crack Free Download
Wondershare PDFelement Pro 11.4.20.3548 Crack Free DownloadPuppy jhon ➡ 🌍📱👉COPY & PASTE LINK👉👉👉 ➤ ➤➤ https://drfiles.net/
Wondershare PDFelement Professional is professional software that can edit PDF files. This digital tool can manipulate elements in PDF documents.
Reimagining Software Development and DevOps with Agentic AI
Reimagining Software Development and DevOps with Agentic AIMaxim Salnikov Key announcements about Developer Productivity from Microsoft Build 2025
Looking for a BIRT Report Alternative Here’s Why Helical Insight Stands Out.pdf
Looking for a BIRT Report Alternative Here’s Why Helical Insight Stands Out.pdfVarsha Nayak The search for an Alternative to BIRT Reports has intensified as companies face challenges with BIRT's steep learning curve, limited visualization capabilities, and complex deployment requirements. Organizations need reporting solutions that offer intuitive design interfaces, comprehensive analytics features, and seamless integration capabilities – all while maintaining the reliability and performance that enterprise environments demand.
How the US Navy Approaches DevSecOps with Raise 2.0
How the US Navy Approaches DevSecOps with Raise 2.0Anchore Join us as Anchore's solutions architect reveals how the U.S. Navy successfully approaches the shift left philosophy to DevSecOps with the RAISE 2.0 Implementation Guide to support its Cyber Ready initiative. This session will showcase practical strategies for defense application teams to pivot from a time-intensive compliance checklist and mindset to continuous cyber-readiness with real-time visibility.
Learn how to break down organizational silos through RAISE 2.0 principles and build efficient, secure pipeline automation that produces the critical security artifacts needed for Authorization to Operate (ATO) approval across military environments.
Migrating to Azure Cosmos DB the Right Way
Migrating to Azure Cosmos DB the Right WayAlexander (Alex) Komyagin In this session we cover the benefits of a migration to Cosmos DB, migration paths, common pain points and best practices. We share our firsthand experiences and customer stories. Adiom is the trusted partner for migration solutions that enable seamless online database migrations from MongoDB to Cosmos DB vCore, and DynamoDB to Cosmos DB for NoSQL.
Porting Qt 5 QML Modules to Qt 6 Webinar
Porting Qt 5 QML Modules to Qt 6 WebinarICS Have you upgraded your application from Qt 5 to Qt 6? If so, your QML modules might still be stuck in the old Qt 5 style—technically compatible, but far from optimal. Qt 6 introduces a modernized approach to QML modules that offers better integration with CMake, enhanced maintainability, and significant productivity gains.
In this webinar, we’ll walk you through the benefits of adopting Qt 6 style QML modules and show you how to make the transition. You'll learn how to leverage the new module system to reduce boilerplate, simplify builds, and modernize your application architecture. Whether you're planning a full migration or just exploring what's new, this session will help you get the most out of your move to Qt 6.
Distributed Postgres
- 1. Distributed postgres. XL, XTM, MultiMaster Stas Kelvich
- 2. Started about a year ago. Konstantin Knizhnik, Constantin Pan, Stas Kelvich Cluster group in PgPro 2
- 3. Started to playing with Postgres-XC. 2ndQuadrant also had project (finished now) to port XC to 9.5. Fork is painful; How can we bring functionality of XC in core? Cluster group in PgPro 3
- 4. Distributed transactions - nothing in-core; Distributed planner - fdw, pg_shard, greenplum planner (?); HA/Autofailover - can be built on top of logical decoding. Distributed postgres 4
- 5. Achieve proper isolation between tx for multi-node transactions. Now in postgres on write tx start: Aquire XID; Get list of running tx’s; Use that info in visibility checks. Distributed transactions 5
- 9. Aquire XID centrally (DTMd, arbiter); No local tx possible; DTMd is a bottleneck. XTM implementations GTM or snapshot sharing 9
- 10. Paper from SAP HANA team; Central daemon is needed, but only for multi-node tx; Snapshots -> Commit Sequence Number; DTMd is still a bottleneck. XTM implementations Incremental SI 10
- 11. XID/CSN are gathered from all nodes that participates in tx; No central service; local tx; possible to reduce communication by using time (Spanner, CockroachDB). XTM implementations Clock-SI or tsDTM 11
- 13. More nodes, higher probability of failure in system. Possible problems with nodes: Node stopped (and will not be back); Node was down small amount of time (and we should bring it back to operation); Network partitions (avoid split-brain). If we want to survive network partitions than we can have not more than [N/2] - 1 failures. HA/autofailover 13
- 14. Possible usage of such system: Multimaster replication; Tables with metainformation in sharded databases; Sharding with redundancy. HA/autofailover 14
- 15. By Multimaster we mean strongly coupled one, that acts as a single database. With proper isolation and no merge conflicts. Ways to build: Global order to XLOG (Postgres-R, MySQL Galera); Wrap each tx as distributed – allows parallelism while applying tx. Multimaster 15
- 16. Our implementation: Built on top of pg_logical; Make use of tsDTM; Pool of workers for tx replay; Raft-based storage for dealing with failures and distributed deadlock detection. Multimaster 16
- 17. Our implementation: Approximately half of a speed of standalone postgres; Same speed for reads; Deals with nodes autorecovery; Deals with network partitions (debugging right now). Can work as an extension (if community accept XTM API in core). Multimaster 17