An Introduction to Using PostgreSQL with Docker & Kubernetes
4 likes2,346 views
The document discusses the use of PostgreSQL with Docker and Kubernetes, highlighting the benefits and challenges of containerization for database management. It covers setting up PostgreSQL containers, using tools like pgAdmin4, and the advantages of Kubernetes for orchestration at scale. The author emphasizes the importance of expertise in managing databases within containerized environments and the potential for automation and flexibility in deployment.
1 of 35
Downloaded 55 times
Ad
Recommended
High Availability PostgreSQL on OpenShift...and more!
High Availability PostgreSQL on OpenShift...and more!Jonathan Katz The document discusses the advantages of using Crunchy Data's PostgreSQL on OpenShift, emphasizing its high availability, data management capabilities, and security features for enterprise applications. It highlights the need for Kubernetes operators to handle stateful workloads, automate administrative tasks, and ensure disaster recovery. Additionally, it outlines the functionalities of the Crunchy PostgreSQL operator that ease database management and deployment in various environments.
Operating PostgreSQL at Scale with Kubernetes
Operating PostgreSQL at Scale with KubernetesJonathan Katz The document discusses the implementation and management of PostgreSQL databases at scale using Kubernetes, highlighting the advantages and challenges of using containers. It covers the use of Crunchy Data's container suite for deployment, monitoring, and disaster recovery, and illustrates how operators can simplify administration tasks. The conclusion emphasizes the potential of PostgreSQL combined with containers and Kubernetes for creating efficient development environments and automated management solutions.
Using PostgreSQL With Docker & Kubernetes - July 2018
Using PostgreSQL With Docker & Kubernetes - July 2018Jonathan Katz The document discusses the integration of PostgreSQL with Docker and Kubernetes, highlighting the advantages and challenges of containerization for database management. It covers setup procedures, the use of Crunchy Data's container suite for PostgreSQL, and describes how Kubernetes can aid in orchestrating these containers for scalability and management. Ultimately, it emphasizes the benefits of containerized environments and the potential for automating database tasks through operators.
Building a Complex, Real-Time Data Management Application
Building a Complex, Real-Time Data Management ApplicationJonathan Katz The document discusses the construction of a real-time data management application utilizing PostgreSQL, focusing on managing room availability at a hotel. It highlights various PostgreSQL features such as range types and procedural languages, necessary for building applications that handle CRUD operations efficiently. Additionally, it explores the implementation of logical decoding for replicating changes and emphasizes the importance of optimizing the application's scalability and performance.
A guide of PostgreSQL on Kubernetes
A guide of PostgreSQL on Kubernetest8kobayashi StatefulSet is used to run PostgreSQL pods across Kubernetes nodes for high availability. When a pod fails, StatefulSet will restart the pod on the same node. However, if the entire node fails, the PostgreSQL pod will not failover to another node by default. To manually failover the pod, it needs to be force deleted and it will restart on a different ready node. However, manual failovers are not recommended for production use.
Realtime Analytics with Druid
Realtime Analytics with DruidSeungWoo Han 1. The document discusses realtime analytics with Druid and LeanStream.
2. It describes the history of technologies used including NiFi, Kafka, Druid and LeanStream and provides an overview of the realtime analytics flow.
3. Examples of realtime analytics queries are shown, including using dimensions like time and metrics like counts to analyze streaming data.
RENCI User Group Meeting 2017 - I Upgraded iRODS and I still have all my hair
RENCI User Group Meeting 2017 - I Upgraded iRODS and I still have all my hairJohn Constable The document summarizes the experience of upgrading a large iRODS installation from version 3.3.1 to 4.1.8 over the course of a year. Key aspects included developing unit tests using BATS, working closely with RENCI on bug fixes, and dealing with issues around large file uploads and full resources during the multi-stage upgrade process. Lessons learned included the importance of testing, configuration management, and working with users and the iRODS community.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein The document discusses real-time distributed and reactive systems utilizing Apache Kafka and Apache Accumulo, emphasizing their roles in asynchronous stream processing and complex event handling. It covers various components, such as the architecture of Kafka, including its producers and consumers, and the integration of frameworks like Apache Mesos for resource management. Additionally, the document provides quick start guides for implementing Kafka with different programming languages and highlights the potential applications of Accumulo in processing data streams.
Flexible compute
Flexible computePeter Clapham The document outlines the development and implementation of a flexible compute platform at Sanger, discussing the transition from traditional HPC to a hybrid cloud architecture using OpenStack. It highlights the infrastructure specifications, resource management strategies, and dynamic workflows aimed at improving application development and scaling services efficiently. The document also addresses the need for collaboration in cloud application development, mentioning early adopters and future opportunities for adaptation and feature requests.
Presto At Treasure Data
Presto At Treasure DataTaro L. Saito Presto is used to process 15 trillion rows per day for Treasure Data customers. Treasure Data developed tools to manage Presto performance and optimize queries. They collect Presto query logs to analyze performance bottlenecks and classify queries to set implicit service level objectives. Tools like Prestobase Proxy and Presto Stella storage optimizer were created to improve low-latency access and optimize storage partitioning. Workflows using DigDag and a new tabular data format called MessageFrame are being explored to split huge queries and support incremental processing.
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...Equnix Business Solutions The document discusses the integration of PostgreSQL with Kubernetes, outlining the capabilities and challenges of managing databases in a container orchestration environment. It covers aspects such as high availability, replication strategies, and the use of various Kubernetes operators for database management. Additionally, it touches on future considerations for cloud-native storage solutions and the evolving role of Kubernetes in database as a service (DBaaS).
MySQL Shell for Database Engineers
MySQL Shell for Database EngineersMydbops The document provides an overview of MySQL Shell, a command-line tool designed for database engineers, offering enhanced capabilities for database management and operations. It discusses utilities for upgrading, importing, exporting, and backing up databases, as well as support for complex architectures and high-availability solutions like InnoDB clusters and replicas. Additionally, it introduces custom shell plugins that facilitate troubleshooting and enhance functionality, along with resources and examples for practical application.
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...Spark Summit Mesos is an open source cluster manager that improves resource utilization. It allows Spark Streaming jobs to leverage Mesos fault tolerance features like driver supervision using Marathon. Backpressure is also supported in Spark Streaming to prevent scheduling delays from fast data arrival. Reactive Streams provide more direct backpressure control and are expected in future Spark versions.
Cassandra Introduction & Features
Cassandra Introduction & FeaturesPhil Peace Cassandra is an open source, distributed, decentralized, and fault-tolerant NoSQL database that is highly scalable and provides tunable consistency. It was created at Facebook based on Amazon's Dynamo and Google's Bigtable. Cassandra's key features include elastic scalability through horizontal partitioning, high availability with no single point of failure, tunable consistency levels, and a column-oriented data model with a CQL interface. Major companies like eBay, Netflix, and Apple use Cassandra for applications requiring large volumes of writes, geographical distribution, and evolving data models.
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein The document provides an overview of Apache Kafka, a distributed messaging system designed for real-time data processing, detailing its architecture, key components, and operational aspects. It highlights Joe's expertise in distributed systems and the importance of deploying Kafka in cloud environments, as well as various programming language clients available for Kafka. The document also discusses automated deployment and scaling solutions, operational benefits, and practical use cases for Kafka applications.
Data warehouse on Kubernetes - gentle intro to Clickhouse Operator, by Robert...
Data warehouse on Kubernetes - gentle intro to Clickhouse Operator, by Robert...Altinity Ltd The document provides an introduction to deploying ClickHouse data warehouses on Kubernetes, outlining the benefits such as portability, effective resource mapping, and fast deployment cycles. It details the architecture of ClickHouse, including its columnar storage, built-in sharding, and replication, while explaining how Kubernetes orchestrates container-based applications. Additionally, it covers the deployment processes, configuration best practices, and potential challenges associated with managing data warehouses in a Kubernetes environment.
Druid at naver.com - part 1
Druid at naver.com - part 1Jungsu Heo This document summarizes a presentation about Druid, an open-source distributed data store designed to handle real-time queries on large datasets. It discusses what Druid is, its architecture, and how it compares to other technologies. Specifically, it covers how Druid's TopN query works and is much faster than a GROUP BY query, though results can be unstable with high replication. It also provides examples of queries and performance comparisons between Druid, Elasticsearch, and Kudu+Impala.
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBKai Sasaki User defined partitioning is a new partitioning strategy in Treasure Data that allows users to specify which column to use for partitioning, in addition to the default "time" column. This provides more flexible partitioning that better fits customer data platform workloads. The user can define partitioning rules through Presto or Hive to improve query performance by enabling colocated joins and filtering data by the partitioning column.
PGConf.ASIA 2019 Bali - PostgreSQL on K8S at Zalando - Alexander Kukushkin
PGConf.ASIA 2019 Bali - PostgreSQL on K8S at Zalando - Alexander KukushkinEqunix Business Solutions Alexander Kukushkin discusses the implementation of PostgreSQL on Kubernetes at Zalando, focusing on the use of Spilo and Patroni for high availability and management of over 1000 PostgreSQL clusters. The presentation covers Kubernetes architecture, problem-solving strategies, and the features of Zalando's PostgreSQL operator to automate cluster management. Key challenges include AWS infrastructure limitations, disk space management, and ensuring efficient resource utilization in a cloud environment.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax The document discusses the implementation of running Apache Cassandra on Apache Mesos at Uber, emphasizing efficient cluster management, stateful service frameworks, and customizations for handling failures. It highlights the architecture, features, and tuning strategies employed for optimal performance, particularly in a multi-datacenter setup. Key statistics on latency and throughput illustrate the system's operational capabilities and reliability across clusters.
Introducing SciaaS @ Sanger
Introducing SciaaS @ SangerPeter Clapham The document outlines the Wellcome Trust's vision to enhance human and animal health by supporting biomedical research and medical humanities, with a focus on fostering innovative startup companies. It describes the Wellcome Genome Campus plan, which includes the establishment of bioinformatics incubators and the integration of cost-effective IT services to facilitate scientific research. Additionally, it discusses a hybrid cloud infrastructure for research, detailing its billing models and services offered to tenants.
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi This document summarizes a presentation about Presto, an open source distributed SQL query engine. It discusses Presto's distributed and plug-in architecture, query planning process, and cluster configuration options. For architecture, it explains that Presto uses coordinators, workers, and connectors to distribute queries across data sources. For query planning, it shows how SQL queries are converted into logical and physical query plans with stages, tasks, and splits. For configuration, it reviews single-server, multi-worker, and multi-coordinator cluster topologies. It also provides an overview of Presto's recent updates.
Mesosphere and Contentteam: A New Way to Run Cassandra
Mesosphere and Contentteam: A New Way to Run CassandraDataStax Academy The document introduces a framework for running Apache Cassandra using Apache Mesos, allowing developers to create scalable Cassandra clusters without extensive server setup. It details the Mesosphere DC/OS, showcasing its capabilities like a web UI, CLI tools, and a REST API for managing cluster tasks. Additionally, it highlights the benefits of the Cassandra-Mesos integration including flexible scaling, fault tolerance, and ease of administration.
Multi Master PostgreSQL Cluster on Kubernetes
Multi Master PostgreSQL Cluster on KubernetesOhyama Masanori The document discusses running PostgreSQL on Kubernetes using Crunchy PostgreSQL Containers. It describes how to run a simple PostgreSQL pod on OpenShift by writing a pod manifest file that defines the container image, ports, environment variables, volume mounts, and other settings. Pulling the PostgreSQL container image from DockerHub and defining the pod configuration in YAML allows running PostgreSQL as a stateful application on Kubernetes.
Terraform Modules Restructured
Terraform Modules RestructuredDoiT International Terraform modules provide reusable, composable infrastructure components. The document discusses restructuring infrastructure code into modules to make it more reusable, testable, and maintainable. Key points include:
- Modules should be structured in a three-tier hierarchy from primitive resources to generic services to specific environments.
- Testing modules individually increases confidence in changes.
- Storing module code and versions in Git provides versioning and collaboration.
- Remote state allows infrastructure to be shared between modules and deployments.
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraSadayuki Furuhashi Logging for Production Systems in The Container Era discusses how to effectively collect and analyze logs and metrics in microservices-based container environments. It introduces Fluentd as a centralized log collection service that supports pluggable input/output, buffering, and aggregation. Fluentd allows collecting logs from containers and routing them to storage systems like Kafka, HDFS and Elasticsearch. It also supports parsing, filtering and enriching log data through plugins.
Hbase Nosql
Hbase Nosqlelliando dias HBase is an open source, distributed, column-oriented database modeled after Google's Bigtable that runs on top of Hadoop. The presenter discusses HBase's architecture, performance improvements in version 0.20 including major gains from new file formats and compression, and Stumbleupon's extensive use of HBase including supporting over 9 billion rows in a single table with high import and read speeds.
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기NAVER D2 Presto generates Java bytecode at runtime to optimize query execution. Key query operations like filtering, projections, joins and aggregations are compiled into efficient Java methods using libraries like ASM and Fastutil. This bytecode generation improves performance by 30% through techniques like compiling row hashing for join lookups directly into machine instructions.
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKSCarlos Andrés García The document provides an overview of Crunchy Data's PostgreSQL for Kubernetes, a certified open-source solution that enhances data security and facilitates cloud-native PostgreSQL management. It highlights the Crunchy PostgreSQL Operator, which automates the provisioning and management of PostgreSQL clusters, while addressing high availability and scalability across various environments. Additionally, it outlines the operator's roadmap and benefits for enterprises, including faster database provisioning and a unified platform for diverse cloud deployments.
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKSVMware Tanzu This document discusses Crunchy Data's PostgreSQL solutions for Kubernetes, including the Crunchy Container Suite and Crunchy PostgreSQL Operator. The Container Suite provides PostgreSQL/PostGIS containers and utility containers for high availability, disaster recovery, and monitoring. The PostgreSQL Operator is an open source project that provisions and manages PostgreSQL clusters on Kubernetes through custom resource definitions and a REST API. It automates complex database administration tasks and allows elastic database provisioning. The Operator is being used successfully by Fortune 100 companies and federal governments to deploy thousands of databases quickly in a secure and auditable environment.
More Related Content
What's hot (20)
Flexible compute
Flexible computePeter Clapham The document outlines the development and implementation of a flexible compute platform at Sanger, discussing the transition from traditional HPC to a hybrid cloud architecture using OpenStack. It highlights the infrastructure specifications, resource management strategies, and dynamic workflows aimed at improving application development and scaling services efficiently. The document also addresses the need for collaboration in cloud application development, mentioning early adopters and future opportunities for adaptation and feature requests.
Presto At Treasure Data
Presto At Treasure DataTaro L. Saito Presto is used to process 15 trillion rows per day for Treasure Data customers. Treasure Data developed tools to manage Presto performance and optimize queries. They collect Presto query logs to analyze performance bottlenecks and classify queries to set implicit service level objectives. Tools like Prestobase Proxy and Presto Stella storage optimizer were created to improve low-latency access and optimize storage partitioning. Workflows using DigDag and a new tabular data format called MessageFrame are being explored to split huge queries and support incremental processing.
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...Equnix Business Solutions The document discusses the integration of PostgreSQL with Kubernetes, outlining the capabilities and challenges of managing databases in a container orchestration environment. It covers aspects such as high availability, replication strategies, and the use of various Kubernetes operators for database management. Additionally, it touches on future considerations for cloud-native storage solutions and the evolving role of Kubernetes in database as a service (DBaaS).
MySQL Shell for Database Engineers
MySQL Shell for Database EngineersMydbops The document provides an overview of MySQL Shell, a command-line tool designed for database engineers, offering enhanced capabilities for database management and operations. It discusses utilities for upgrading, importing, exporting, and backing up databases, as well as support for complex architectures and high-availability solutions like InnoDB clusters and replicas. Additionally, it introduces custom shell plugins that facilitate troubleshooting and enhance functionality, along with resources and examples for practical application.
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...Spark Summit Mesos is an open source cluster manager that improves resource utilization. It allows Spark Streaming jobs to leverage Mesos fault tolerance features like driver supervision using Marathon. Backpressure is also supported in Spark Streaming to prevent scheduling delays from fast data arrival. Reactive Streams provide more direct backpressure control and are expected in future Spark versions.
Cassandra Introduction & Features
Cassandra Introduction & FeaturesPhil Peace Cassandra is an open source, distributed, decentralized, and fault-tolerant NoSQL database that is highly scalable and provides tunable consistency. It was created at Facebook based on Amazon's Dynamo and Google's Bigtable. Cassandra's key features include elastic scalability through horizontal partitioning, high availability with no single point of failure, tunable consistency levels, and a column-oriented data model with a CQL interface. Major companies like eBay, Netflix, and Apple use Cassandra for applications requiring large volumes of writes, geographical distribution, and evolving data models.
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein The document provides an overview of Apache Kafka, a distributed messaging system designed for real-time data processing, detailing its architecture, key components, and operational aspects. It highlights Joe's expertise in distributed systems and the importance of deploying Kafka in cloud environments, as well as various programming language clients available for Kafka. The document also discusses automated deployment and scaling solutions, operational benefits, and practical use cases for Kafka applications.
Data warehouse on Kubernetes - gentle intro to Clickhouse Operator, by Robert...
Data warehouse on Kubernetes - gentle intro to Clickhouse Operator, by Robert...Altinity Ltd The document provides an introduction to deploying ClickHouse data warehouses on Kubernetes, outlining the benefits such as portability, effective resource mapping, and fast deployment cycles. It details the architecture of ClickHouse, including its columnar storage, built-in sharding, and replication, while explaining how Kubernetes orchestrates container-based applications. Additionally, it covers the deployment processes, configuration best practices, and potential challenges associated with managing data warehouses in a Kubernetes environment.
Druid at naver.com - part 1
Druid at naver.com - part 1Jungsu Heo This document summarizes a presentation about Druid, an open-source distributed data store designed to handle real-time queries on large datasets. It discusses what Druid is, its architecture, and how it compares to other technologies. Specifically, it covers how Druid's TopN query works and is much faster than a GROUP BY query, though results can be unstable with high replication. It also provides examples of queries and performance comparisons between Druid, Elasticsearch, and Kudu+Impala.
User Defined Partitioning on PlazmaDB
User Defined Partitioning on PlazmaDBKai Sasaki User defined partitioning is a new partitioning strategy in Treasure Data that allows users to specify which column to use for partitioning, in addition to the default "time" column. This provides more flexible partitioning that better fits customer data platform workloads. The user can define partitioning rules through Presto or Hive to improve query performance by enabling colocated joins and filtering data by the partitioning column.
PGConf.ASIA 2019 Bali - PostgreSQL on K8S at Zalando - Alexander Kukushkin
PGConf.ASIA 2019 Bali - PostgreSQL on K8S at Zalando - Alexander KukushkinEqunix Business Solutions Alexander Kukushkin discusses the implementation of PostgreSQL on Kubernetes at Zalando, focusing on the use of Spilo and Patroni for high availability and management of over 1000 PostgreSQL clusters. The presentation covers Kubernetes architecture, problem-solving strategies, and the features of Zalando's PostgreSQL operator to automate cluster management. Key challenges include AWS infrastructure limitations, disk space management, and ensuring efficient resource utilization in a cloud environment.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax The document discusses the implementation of running Apache Cassandra on Apache Mesos at Uber, emphasizing efficient cluster management, stateful service frameworks, and customizations for handling failures. It highlights the architecture, features, and tuning strategies employed for optimal performance, particularly in a multi-datacenter setup. Key statistics on latency and throughput illustrate the system's operational capabilities and reliability across clusters.
Introducing SciaaS @ Sanger
Introducing SciaaS @ SangerPeter Clapham The document outlines the Wellcome Trust's vision to enhance human and animal health by supporting biomedical research and medical humanities, with a focus on fostering innovative startup companies. It describes the Wellcome Genome Campus plan, which includes the establishment of bioinformatics incubators and the integration of cost-effective IT services to facilitate scientific research. Additionally, it discusses a hybrid cloud infrastructure for research, detailing its billing models and services offered to tenants.
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi This document summarizes a presentation about Presto, an open source distributed SQL query engine. It discusses Presto's distributed and plug-in architecture, query planning process, and cluster configuration options. For architecture, it explains that Presto uses coordinators, workers, and connectors to distribute queries across data sources. For query planning, it shows how SQL queries are converted into logical and physical query plans with stages, tasks, and splits. For configuration, it reviews single-server, multi-worker, and multi-coordinator cluster topologies. It also provides an overview of Presto's recent updates.
Mesosphere and Contentteam: A New Way to Run Cassandra
Mesosphere and Contentteam: A New Way to Run CassandraDataStax Academy The document introduces a framework for running Apache Cassandra using Apache Mesos, allowing developers to create scalable Cassandra clusters without extensive server setup. It details the Mesosphere DC/OS, showcasing its capabilities like a web UI, CLI tools, and a REST API for managing cluster tasks. Additionally, it highlights the benefits of the Cassandra-Mesos integration including flexible scaling, fault tolerance, and ease of administration.
Multi Master PostgreSQL Cluster on Kubernetes
Multi Master PostgreSQL Cluster on KubernetesOhyama Masanori The document discusses running PostgreSQL on Kubernetes using Crunchy PostgreSQL Containers. It describes how to run a simple PostgreSQL pod on OpenShift by writing a pod manifest file that defines the container image, ports, environment variables, volume mounts, and other settings. Pulling the PostgreSQL container image from DockerHub and defining the pod configuration in YAML allows running PostgreSQL as a stateful application on Kubernetes.
Terraform Modules Restructured
Terraform Modules RestructuredDoiT International Terraform modules provide reusable, composable infrastructure components. The document discusses restructuring infrastructure code into modules to make it more reusable, testable, and maintainable. Key points include:
- Modules should be structured in a three-tier hierarchy from primitive resources to generic services to specific environments.
- Testing modules individually increases confidence in changes.
- Storing module code and versions in Git provides versioning and collaboration.
- Remote state allows infrastructure to be shared between modules and deployments.
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraSadayuki Furuhashi Logging for Production Systems in The Container Era discusses how to effectively collect and analyze logs and metrics in microservices-based container environments. It introduces Fluentd as a centralized log collection service that supports pluggable input/output, buffering, and aggregation. Fluentd allows collecting logs from containers and routing them to storage systems like Kafka, HDFS and Elasticsearch. It also supports parsing, filtering and enriching log data through plugins.
Hbase Nosql
Hbase Nosqlelliando dias HBase is an open source, distributed, column-oriented database modeled after Google's Bigtable that runs on top of Hadoop. The presenter discusses HBase's architecture, performance improvements in version 0.20 including major gains from new file formats and compression, and Stumbleupon's extensive use of HBase including supporting over 9 billion rows in a single table with high import and read speeds.
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기NAVER D2 Presto generates Java bytecode at runtime to optimize query execution. Key query operations like filtering, projections, joins and aggregations are compiled into efficient Java methods using libraries like ASM and Fastutil. This bytecode generation improves performance by 30% through techniques like compiling row hashing for join lookups directly into machine instructions.
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...Equnix Business Solutions
Similar to An Introduction to Using PostgreSQL with Docker & Kubernetes (20)
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKSCarlos Andrés García The document provides an overview of Crunchy Data's PostgreSQL for Kubernetes, a certified open-source solution that enhances data security and facilitates cloud-native PostgreSQL management. It highlights the Crunchy PostgreSQL Operator, which automates the provisioning and management of PostgreSQL clusters, while addressing high availability and scalability across various environments. Additionally, it outlines the operator's roadmap and benefits for enterprises, including faster database provisioning and a unified platform for diverse cloud deployments.
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKSVMware Tanzu This document discusses Crunchy Data's PostgreSQL solutions for Kubernetes, including the Crunchy Container Suite and Crunchy PostgreSQL Operator. The Container Suite provides PostgreSQL/PostGIS containers and utility containers for high availability, disaster recovery, and monitoring. The PostgreSQL Operator is an open source project that provisions and manages PostgreSQL clusters on Kubernetes through custom resource definitions and a REST API. It automates complex database administration tasks and allows elastic database provisioning. The Operator is being used successfully by Fortune 100 companies and federal governments to deploy thousands of databases quickly in a secure and auditable environment.
Deploying PostgreSQL on Kubernetes
Deploying PostgreSQL on KubernetesJimmy Angelakos The document provides a comprehensive overview of deploying PostgreSQL on Kubernetes, emphasizing the importance of Kubernetes as a container orchestrator in a service-oriented architecture. It covers Kubernetes basics, including its architecture, key terms, and deployment strategies, followed by methods for deploying PostgreSQL using Helm charts and the Crunchy Data operator. Observations highlight the advantages and challenges of different deployment methods, advocating for tools like Helm and the Crunchy operator for ease of management.
Crunchy containers
Crunchy containersRenato Lucena This document provides a summary of the Crunchy Container Suite, including:
- An overview of the container images included in the suite for PostgreSQL databases, tools to manage databases, backup/restore, administration, performance monitoring, and connection pooling.
- Descriptions of the key features and benefits of the Crunchy Container Suite, such as compatibility with various platforms, cloud-agnostic deployment, open source, and certified builds.
- Instructions for users to set up their environment, install the Crunchy Container Suite, configure storage, and run examples to demonstrate the capabilities of the software.
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized WorldJignesh Shah This document discusses PostgreSQL high availability in a containerized environment. It begins with an overview of containers and their advantages like lower footprint and density. It then covers enterprise needs for high availability like recovery time objectives. Common approaches to PostgreSQL high availability are discussed like replication, shared storage, and using projects like Patroni and Stolon. Modern trends with containers are highlighted like separating data and binaries. Kubernetes is presented as a production-grade orchestrator that can provide horizontal scaling and self-healing capabilities. The discussion concludes with challenges of multi-region deployments and how service discovery with Consul can help address those challenges.
Cloud Native PostgreSQL
Cloud Native PostgreSQLEDB The document provides an overview of cloud-native PostgreSQL deployment using Kubernetes, highlighting the benefits of utilizing operators for management and automation of PostgreSQL clusters. It discusses key features including self-healing capabilities, storage management options, and security measures, along with performance benchmarks comparing local and network storage. The agenda emphasizes various technical aspects such as architectures, configuration, and capabilities related to using PostgreSQL in cloud environments.
Deployment of PostgreSQL inside of Kubernetes with High Availability
Deployment of PostgreSQL inside of Kubernetes with High AvailabilityEDB The document discusses the deployment of PostgreSQL in Kubernetes with high availability, led by Armin Nesiren, a database engineer at Cybertec. It covers container management, the benefits of Kubernetes like automated rollouts, failover handling, and the role of tools like Patroni for automatic failover and cluster management. Additionally, it highlights the simplification of PostgreSQL deployment through Kubernetes operators and the various backup methods available.
Cloud Native PostgreSQL - APJ
Cloud Native PostgreSQL - APJEDB The document discusses the integration of PostgreSQL with Kubernetes, outlining the capabilities and architectures involved in managing databases in a cloud-native environment. It introduces Gabriele Bartolini and highlights EDB's position as a leader in PostgreSQL innovation through its operator for Kubernetes, which automates deployment, configuration, and management of PostgreSQL clusters. Key features include self-healing capabilities, support for persistent volumes, and a focus on security and performance, emphasizing the benefits of using local storage for enhanced database performance.
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized WorldJignesh Shah This document discusses high availability for PostgreSQL in a containerized environment. It outlines typical enterprise requirements for high availability including recovery time objectives and recovery point objectives. Shared storage-based high availability is described as well as the advantages and disadvantages of PostgreSQL replication. The use of Linux containers and orchestration tools like Kubernetes and Consul for managing containerized PostgreSQL clusters is also covered. The document advocates for using PostgreSQL replication along with services and self-healing tools to provide highly available and scalable PostgreSQL deployments in modern container environments.
Postgres on Kubernetes - Dos and Donts.pdf
Postgres on Kubernetes - Dos and Donts.pdfChristoph Engelbert The document discusses the challenges and considerations of running databases, particularly PostgreSQL, on Kubernetes, highlighting that Kubernetes is not designed with databases in mind. It emphasizes the importance of separating databases from applications, using persistent volumes, ensuring high availability through connection pooling, and leveraging Kubernetes operators for management. Additionally, it addresses monitoring and security best practices to enhance database performance and reliability.
Containers Roadshow: How to Develop Containers for the Enterprise
Containers Roadshow: How to Develop Containers for the EnterpriseHonza Horák The document provides an overview on developing containers for enterprise applications, focusing on various technologies and methodologies used at Red Hat. Key topics include container basics, PostgreSQL container deployment, organizing standalone and GUI applications in containers, and using Ansible for container orchestration. The presentation emphasizes the importance of efficiency and automation in container management within organizational systems.
EDB Postgres with Containers
EDB Postgres with ContainersEDB Enterprisedb is a leader in operational databases, offering enterprise-class features and support for PostgreSQL. The document discusses containers as a transformative technology for application packaging and performance, detailing their benefits and key features. It emphasizes the ease of deploying stateful applications using containers and encourages exploring their use within existing architectures.
PostgreSQL and Linux Containers
PostgreSQL and Linux ContainersJignesh Shah This document provides an overview of Linux containers and Docker containers. It begins with definitions of containers and their advantages over virtual machines. It describes early implementations of containers like chroot, Jails, and Zones. It explains the underlying kernel technologies like cgroups and namespaces that enable Linux containers. It provides instructions for using LXC and systemd-nspawn to deploy basic containers. It then focuses on Docker containers, covering installation, images, volumes, and best practices for running applications like PostgreSQL in Docker containers.
Postgre sql linuxcontainers by Jignesh Shah
Postgre sql linuxcontainers by Jignesh ShahPivotalOpenSourceHub This document discusses Linux containers and PostgreSQL in Docker containers. It begins with an overview of containers, their advantages and disadvantages compared to virtual machines. It then discusses different implementations of containers like LXC and systemd-nspawn. A large portion of the document is dedicated to Docker containers - how to install, use images and volumes, and common commands. It concludes with best practices for running PostgreSQL in Docker containers, including mounting data volumes, linking containers, checking stats and processes.
Introduction to SQL Server in Containers
Introduction to SQL Server in ContainersGrant Fritchey The document outlines the use of SQL Server in containerized environments, detailing what containers are and how they differ from virtual machines. It explains the requirements for Docker installation on various operating systems and introduces orchestration tools like Kubernetes and Docker Swarm. Additionally, the document emphasizes the benefits of using containers in development, testing, and deployment for enhanced speed, consistency, and security.
Docker Containers- Data Engineers' Arsenal.pdf
Docker Containers- Data Engineers' Arsenal.pdfgr6336192 Leverage Docker containers to build, deploy, and scale data pipelines effortlessly. From reproducible environments to streamlined workflows, Docker is a game-changer in modern data engineering.
Speed up development, ensure reproducibility, and simplify deployment with Docker containers in your data engineering stack.
Get Started with Data Container Here https://bit.ly/3Hbd3wW
Cont0519
Cont0519Samuel Dratwa The document is an introduction to Docker and Kubernetes, discussing their components, lifecycle, and the benefits of using containers and microservices over traditional virtual machines. It covers the architecture of Docker, including the use of images, containers, and orchestration via Kubernetes, as well as hands-on examples and use cases in various environments. Additionally, it highlights the advantages of these technologies for developers and operations teams, aimed at improving deployment speed and system reliability.
YugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on KubernetesDoKC The document discusses the benefits and challenges of running a distributed SQL database, specifically YugabyteDB, on Kubernetes. It highlights key features such as better resource utilization, dynamic resource resizing, and the importance of automation and resilience in database operations. Additionally, it addresses potential pitfalls like pod failures and networking complexities while emphasizing YugabyteDB's scalability and compatibility across various cloud environments.
A Primer on Kubernetes and Google Container Engine
A Primer on Kubernetes and Google Container EngineRightScale The document provides a comprehensive overview of Kubernetes and Google Container Engine (GKE), detailing the evolution and significance of container technology in managing applications at scale. It highlights the features and architecture of Kubernetes, its community support, and integration with cloud services. Additionally, it discusses RightScale's approach to automating cloud management and orchestration for container deployments.
Docker for Ruby Developers
Docker for Ruby DevelopersAptible The document discusses the use of Docker for Ruby developers, highlighting its advantages over traditional virtual machines and emphasizing the importance of achieving development and production parity using Docker images and containers. It provides practical examples, including Docker commands for pulling images, running containers, and configuring services with Docker Compose, while also addressing deployment complexities in service-oriented architectures. Additionally, it touches on managing SSL infrastructure with Docker and integration testing during image builds to ensure security and reliability.
Ad
More from Jonathan Katz (11)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Jonathan Katz The document discusses PostgreSQL as an advanced open-source relational database, highlighting its support for JSON and the addition of features for vector storage with the pgvector extension. It elaborates on the challenges and strategies in using vectors for data storage and retrieval, including approximate nearest neighbor searches and the impact of PostgreSQL's TOAST mechanism on performance. Additionally, it provides best practices and insights on utilizing various indexing methods such as HNSW and IVFFlat for optimizing vector searches.
Vectors are the new JSON in PostgreSQL
Vectors are the new JSON in PostgreSQLJonathan Katz The document discusses the significance of vectors in various fields, emphasizing their role in artificial intelligence and machine learning. It compares vectors to JSON, suggesting they represent an evolution in data representation. Key applications include mathematics, physics, and mapping.
Looking ahead at PostgreSQL 15
Looking ahead at PostgreSQL 15Jonathan Katz The document discusses the features and enhancements of PostgreSQL 15, highlighting its compatibility with Amazon RDS and new functionalities such as conditional SQL and JSON support. It details the PostgreSQL release cycle, evolution of SQL features, and improvements in backup and recovery processes. Additionally, it covers logical replication capabilities and performance optimizations in data handling.
Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!Jonathan Katz The document presents a talk by Jonathan S. Katz on building a complex real-time data management application using PostgreSQL, focusing on managing availability for virtual rooms in an event platform. It covers various PostgreSQL features such as range types, recursive queries, and logical replication, emphasizing methods to enhance performance and scalability while avoiding trigger overhead. The document concludes with insights on PostgreSQL's capabilities and architectural patterns for effective application development.
Get Your Insecure PostgreSQL Passwords to SCRAM
Get Your Insecure PostgreSQL Passwords to SCRAMJonathan Katz Jonathan S. Katz discusses password management in PostgreSQL, emphasizing the shift towards using SCRAM-SHA-256 for enhanced security over previous methods like plaintext and MD5. The presentation details the vulnerabilities associated with older password mechanisms and explains how SCRAM provides a more secure authentication process by allowing mutual verification without password exposure. Attendees will learn how to transition systems to implement SCRAM-SHA-256 effectively.
Safely Protect PostgreSQL Passwords - Tell Others to SCRAM
Safely Protect PostgreSQL Passwords - Tell Others to SCRAMJonathan Katz Jonathan S. Katz gave a talk on safely protecting passwords in PostgreSQL. He discussed:
- The evolution of password management in PostgreSQL, from storing passwords in plain text to using md5 hashes to modern SCRAM authentication.
- How plain text and md5 password storage are insecure as passwords can be intercepted or cracked.
- The SCRAM authentication standard which allows two parties to verify they know a secret without exchanging the secret directly.
- How PostgreSQL implements SCRAM-SHA-256 to generate a secure verifier from the password and authenticate users with random salts and iterations to secure against brute force attacks.
Developing and Deploying Apps with the Postgres FDW
Developing and Deploying Apps with the Postgres FDWJonathan Katz This document summarizes Jonathan Katz's experience building a foreign data wrapper (FDW) between two PostgreSQL databases to enable an API for his company VenueBook. He created separate "app" and "api" databases, with the api database using FDWs to access tables in the app database. This allowed inserting and querying data across databases. However, he encountered permission errors and had to grant various privileges on the remote database to make it work properly, demonstrating the importance of permissions management with FDWs.
On Beyond (PostgreSQL) Data Types
On Beyond (PostgreSQL) Data TypesJonathan Katz El documento presenta una charla sobre los tipos de datos en PostgreSQL, destacando su flexibilidad y funcionalidad para manejar datos complejos. Se discuten diferentes tipos como geométricos, UUIDs y arreglos, incluyendo sus características y ejemplos de implementación. Además, se abordan técnicas de optimización del rendimiento utilizando índices para eficientizar consultas de datos espaciales y arreglos.
Accelerating Local Search with PostgreSQL (KNN-Search)
Accelerating Local Search with PostgreSQL (KNN-Search)Jonathan Katz The document discusses the use of PostgreSQL 9.1 for enhancing local search capabilities, especially through k-nearest neighbor (k-NN) queries and the application of distance metrics. It highlights the improvements in performance and indexing for geometric types and text similarity with the pg_trgm module. The presentation concludes with recommendations for further geometric support and potential applications of these indexing techniques.
Webscale PostgreSQL - JSONB and Horizontal Scaling Strategies
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz The document discusses the evolution of PostgreSQL's JSON and JSONB data types, highlighting their advantages over the traditional relational model. It explains the features, operators, and performance comparisons of JSON and JSONB, and the impact of indexing on query speed. Additionally, it emphasizes the necessity of validation and structure for JSON documents within a database context.
Indexing Complex PostgreSQL Data Types
Indexing Complex PostgreSQL Data TypesJonathan Katz The document is a presentation by Jonathan S. Katz on indexing complex PostgreSQL data types, covering various indexing methods including B-trees, GIST, and GIN. It explores their applications, performance characteristics, and practical examples using PostgreSQL for efficient data retrieval. The presentation also discusses production indexing strategies and the integration of full-text search functionalities with PostgreSQL through indexing techniques.
Ad
Recently uploaded (20)
Edge-banding-machines-edgeteq-s-200-en-.pdf
Edge-banding-machines-edgeteq-s-200-en-.pdfAmirStern2 מכונת קנטים המתאימה לנגריות קטנות או גדולות (כמכונת גיבוי).
מדביקה קנטים מגליל או פסים, עד עובי קנט – 3 מ"מ ועובי חומר עד 40 מ"מ. בקר ממוחשב המתריע על תקלות, ומנועים מאסיביים תעשייתיים כמו במכונות הגדולות.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdf
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdfMuhammad Rizwan Akram DC Inverter Air Conditioners are revolutionizing the cooling industry by delivering affordable,
energy-efficient, and environmentally sustainable climate control solutions. Unlike conventional
fixed-speed air conditioners, DC inverter systems operate with variable-speed compressors that
modulate cooling output based on demand, significantly reducing energy consumption and
extending the lifespan of the appliance.
These systems are critical in reducing electricity usage, lowering greenhouse gas emissions, and
promoting eco-friendly technologies in residential and commercial sectors. With advancements in
compressor control, refrigerant efficiency, and smart energy management, DC inverter air conditioners
have become a benchmark in sustainable climate control solutions
vertical-cnc-processing-centers-drillteq-v-200-en.pdf
vertical-cnc-processing-centers-drillteq-v-200-en.pdfAmirStern2 מכונות CNC קידוח אנכיות הן הבחירה הנכונה והטובה ביותר לקידוח ארונות וארגזים לייצור רהיטים. החלק נוסע לאורך ציר ה-x באמצעות ציר דיגיטלי מדויק, ותפוס ע"י צבת מכנית, כך שאין צורך לבצע setup (התאמות) לגדלים שונים של חלקים.
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...Safe Software The National Fuels Treatments Initiative (NFT) is transforming wildfire mitigation by creating a standardized map of nationwide fuels treatment locations across all land ownerships in the United States. While existing state and federal systems capture this data in diverse formats, NFT bridges these gaps, delivering the first truly integrated national view. This dataset will be used to measure the implementation of the National Cohesive Wildland Strategy and demonstrate the positive impact of collective investments in hazardous fuels reduction nationwide. In Phase 1, we developed an ETL pipeline template in FME Form, leveraging a schema-agnostic workflow with dynamic feature handling intended for fast roll-out and light maintenance. This was key as the initiative scaled from a few to over fifty contributors nationwide. By directly pulling from agency data stores, oftentimes ArcGIS Feature Services, NFT preserves existing structures, minimizing preparation needs. External mapping tables ensure consistent attribute and domain alignment, while robust change detection processes keep data current and actionable. Now in Phase 2, we’re migrating pipelines to FME Flow to take advantage of advanced scheduling, monitoring dashboards, and automated notifications to streamline operations. Join us to explore how this initiative exemplifies the power of technology, blending FME, ArcGIS Online, and AWS to solve a national business problem with a scalable, automated solution.
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptx
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptxFIDO Alliance FIDO Seminar: Authentication for a Billion Consumers - Amazon
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
MuleSoft for AgentForce : Topic Center and API Catalog
MuleSoft for AgentForce : Topic Center and API Catalogshyamraj55 This presentation dives into how MuleSoft empowers AgentForce with organized API discovery and streamlined integration using Topic Center and the API Catalog. Learn how these tools help structure APIs around business needs, improve reusability, and simplify collaboration across teams. Ideal for developers, architects, and business stakeholders looking to build a connected and scalable API ecosystem within AgentForce.
Bridging the divide: A conversation on tariffs today in the book industry - T...
Bridging the divide: A conversation on tariffs today in the book industry - T...BookNet Canada A collaboration-focused conversation on the recently imposed US and Canadian tariffs where speakers shared insights into the current legislative landscape, ongoing advocacy efforts, and recommended next steps. This event was presented in partnership with the Book Industry Study Group.
Link to accompanying resource: https://bnctechforum.ca/sessions/bridging-the-divide-a-conversation-on-tariffs-today-in-the-book-industry/
Presented by BookNet Canada and the Book Industry Study Group on May 29, 2025 with support from the Department of Canadian Heritage.
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Safe Software Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...SOFTTECHHUB AudGram changes everything by bridging the gap between your audio content and the visual engagement your audience craves. This cloud-based platform transforms your existing audio into scroll-stopping visual content that performs across all social media platforms.
Supporting the NextGen 911 Digital Transformation with FME
Supporting the NextGen 911 Digital Transformation with FMESafe Software Next Generation 911 involves the transformation of our 911 system from an old analog one to the new digital internet based architecture. The evolution of NG911 opens up a host of new opportunities to improve the system. This includes everything from device based location, to real time text. This can improve location accuracy dramatically as well as provide live updates from the citizen in need along with real time sensor updates. There is also the opportunity to provide multi-media attachments and medical records if the end user approves. This digital transformation and enhancements all require the support of new NENA and CRTC standards, along with integration across a variety of data streams.
This presentation will focus on how FME has supported NG911 transformations to date, and how we are positioning FME to support the enhanced capabilities to come. This session will be of interest to emergency services, municipalities and anyone who may be interested to know more about how emergency services are being improved to provide more accurate, localized information in order to improve the speed and relevance of emergency response and ultimately save more lives and provide better outcomes for those in need.
FME for Good: Integrating Multiple Data Sources with APIs to Support Local Ch...
FME for Good: Integrating Multiple Data Sources with APIs to Support Local Ch...Safe Software Have-a-skate-with-Bob (HASB-KC) is a local charity that holds two Hockey Tournaments every year to raise money in the fight against Pancreatic Cancer. The FME Form software is used to integrate and exchange data via API, between Google Forms, Google Sheets, Stripe payments, SmartWaiver, and the GoDaddy email marketing tools to build a grass-roots Customer Relationship Management (CRM) system for the charity. The CRM is used to communicate effectively and readily with the participants of the hockey events and most importantly the local area sponsors of the event. Communication consists of a BLOG used to inform participants of event details including, the ever-important team rosters. Funds raised by these events are used to support families in the local area to fight cancer and support PanCan research efforts to find a cure against this insidious disease. FME Form removes the tedium and error-prone manual ETL processes against these systems into 1 or 2 workbenches that put the data needed at the fingertips of the event organizers daily freeing them to work on outreach and marketing of the events in the community.
Providing an OGC API Processes REST Interface for FME Flow
Providing an OGC API Processes REST Interface for FME FlowSafe Software This presentation will showcase an adapter for FME Flow that provides REST endpoints for FME Workspaces following the OGC API Processes specification. The implementation delivers robust, user-friendly API endpoints, including standardized methods for parameter provision. Additionally, it enhances security and user management by supporting OAuth2 authentication. Join us to discover how these advancements can elevate your enterprise integration workflows and ensure seamless, secure interactions with FME Flow.
High Availability On-Premises FME Flow.pdf
High Availability On-Premises FME Flow.pdfSafe Software FME Flow is a highly robust tool for transforming data both automatically and by user-initiated workflows. At the Finnish telecommunications company Elisa, FME Flow serves processes and internal stakeholders that require 24/7 availability from underlying systems, while imposing limitations on the use of cloud based systems. In response to these business requirements, Elisa has implemented a high-availability on-premises setup of FME Flow, where all components of the system have been duplicated or clustered. The goal of the presentation is to provide insights into the architecture behind the high-availability functionality. The presentation will show in basic technical terms how the different parts of the system work together. Basic level understanding of IT technologies is required to understand the technical portion of the presentation, namely understanding the purpose of the following components: load balancer, FME Flow host nodes, FME Flow worker nodes, network file storage drives, databases, and external authentication services. The presentation will also outline our lessons learned from the high-availability project, both benefits and challenges to consider.
An Introduction to Using PostgreSQL with Docker & Kubernetes
- 1. An Introduction to Using PostgreSQL with Docker & Kubernetes Jonathan S. Katz DC PostgreSQL User Group April 2, 2018 1
- 2. About Crunchy Data • Leading provider of trusted open source PostgreSQL and PostgreSQL related technologies, support and training to enterprises. • We're Hiring! • crunchydata.com 2
- 3. About Me • Director of Communications & Customer Success, Crunchy Data • Longtime PostgreSQL community contributor • Director, PgUS • Co-Organizer, NYCPUG • @postgresql + .org content • Conference organization + speaking galore! • @jkatz05 !3
- 4. Outline • Containers: A Brief History • Containers + PostgreSQL • Setting up PostgreSQL with Containers • Deploying! - Container Orchestration • Look Ahead: Trends in the Container World !4
- 5. What are Containers? • Containers are processes that contain all the requirements to execute an application • Similar to virtual machines, Sandbox for applications similar to a virtual machine but with increased density on a single host 5 Source: Docker
- 6. Container Glossary • Image • The base filesystem that defines what to load in a container runtime. This is static. • Container • The runtime loads an image and instructions on how to set up the run time environment. File changes can be persisted to external mounts https://docs.docker.com/glossary/ !6
- 7. Why Containers? • Lightweight • Compared to VMs, use less disk, RAM, CPU • Ability to sandbox • Runtime is in an isolated environment • Portability • Can run containers on different platforms !7
- 8. Container Runtimes • Open Containers Initiative (OCI) • Developing standards for how containers should run and interface with platforms • containerd, Docker, runC, rkt !8
- 9. Container Example: Web Application Create Dockerfile FROM python:3 ENV PYTHONUNBUFFERED 1 RUN mkdir -p /path/to/app/code/ WORKDIR /path/to/app/code/ ADD requirements.txt /path/to/app/code/ RUN pip install -r requirements.txt ADD . /path/to/app/code/ !9 1.11.11 7.4
- 10. Container Example: Web Application Define and Execute the Container Runtime !10 /path/to/app/code/ 8000 1.11.11 8000 7.4
- 11. Container Example: Web Application Create multiple runtimes !11 /path/to/app/code/ 8001 1.11.11 8000 7.4 8000 1.11.11 8000 7.4 8002 1.11.11 8000 7.4
- 12. Container Example: Web Application Update Django Versions !12
- 13. Containers & PostgreSQL • Containers provide several advantages to running PostgreSQL: • Setup & distribution for developer environments • Ease of packaging extensions & minor upgrades • Separate out secondary applications (monitoring, administration) • Automation and scale for provisioning and creating replicas, backups Images sourced: docker.com & postgresql.org!13
- 14. Containers & PostgreSQL • Containers also introduce several challenges: • Administrator needs to understand and select appropriate storage options • Configuration for individual database specifications and user access • Managing 100s - 1000s of containers requires appropriate orchestration (more on that later) • However, these are challenges you will find in most database environments Images sourced: docker.com & postgresql.org!14
- 15. • We will use the Crunchy Container Suite • PostgreSQL (+ PostGIS): our favorite database; option to add our favorite geospatial extension • pgpool + pgbouncer: connection pooling, load balancing • pgbackrest: terabyte-scale backup management • Monitoring: Prometheus + export • Failover: "crunchy-watch" • pgadmin4: UX-driven management • Open source! • Apache 2.0 license • Support for Docker 1.12+, Kubernetes 1.5+ • Actively maintained and updated Getting Started With Containers + PostgreSQL https://github.com/CrunchyData/crunchy-containers!15
- 16. Getting Started With Containers + PostgreSQL https://github.com/CrunchyData/crunchy-containers !16
- 17. Getting Started with Containers & PostgreSQL mkdir postgres cd postgres docker volume create --driver local --name=pgvolume docker network create --driver bridge pgnetwork cat << EOF > pg-env.list PG_MODE=primary PG_PRIMARY_USER=postgres PG_PRIMARY_PASSWORD=password PG_DATABASE=whales PG_USER=jkatz PG_PASSWORD=password PG_ROOT_PASSWORD=password PG_PRIMARY_PORT=5432 EOF docker run --publish 5432:5432 --volume=pgvolume:/pgdata --env-file=pg-env.list --name="postgres" --hostname="postgres" --network="pgnetwork" --detach crunchydata/crunchy-postgres:centos7-10.3-1.8.2 Load the crunchy-postgres image !17
- 18. Demo !18
- 19. Adding pgadmin4 • pgadmin4 is an open source user interface for working with PostgreSQL • Desktop + Web support 19
- 20. Adding pgadmin4 Load the crunchy-pgadmin4 image docker volume create --driver local --name=pga4volume cat << EOF > pgadmin4-env.list [email protected] PGADMIN_SETUP_PASSWORD=securepassword SERVER_PORT=5050 EOF docker run --publish 5050:5050 --volume=pga4volume:/var/lib/pgadmin --env-file=pgadmin4-env.list --name="pgadmin4" --hostname="pgadmin4" --network="pgnetwork" --detach crunchydata/crunchy-pgadmin4:centos7-10.3-1.8.2 !20
- 21. Demo !21
- 22. Recap So Far • Explored what / why / how of containers • Set up a PostgreSQL 10 instance that can be managed with pgadmin4 in a development environment !22
- 23. How do I manage these things at scale? !23
- 24. Kubernetes: Container Orchestration • "Open-source system for automating deployment, scaling, and management of containerized applications." • Manage the full lifecycle of a container • Pods => Services • Nodes => Cluster • Volumes 24 Source: https://kubernetes.io
- 26. Kubernetes is *not* easy to set up !26
- 27. When to Use Kubernetes • Managing a few containers is very different than managing 100s, if not 1000s • Advanced “orchestration” required to manage this many containers 27
- 28. PostgreSQL in a Kubernetes World • Advantages for using Kubernetes with PostgreSQL • Your own "database-as-a-service" • Mass apply databases commands: • Updates • Backups / Restores • ACL rule changes • Scale up / down replicas, though... !28
- 29. PostgreSQL in a Kubernetes World • Kubernetes is "turnkey" for stateless applications • e.g. web servers • Databases do maintain state: permanent storage • Persistent Volumes (PV) • Persistent Volume Claims (PVC) !29
- 30. The "postgres operator" • (aka "Crunchy PostgreSQL for Kubernetes") • Allows an administrator to run PostgreSQL-specific commands to manage database clusters, including: • Creating / Deleting a cluster (your own DBaaS) • Scaling up / down replicas • Apply user policies to PostgreSQL instances • REST API !30 https://github.com/CrunchyData/postgres-operator
- 31. How The Operator Works !31
- 33. Looking Ahead • Containers are no longer "new" - orchestration technologies have matured • Big debate with containers + databases: storage & management • No different than virtual machines + databases • Databases are still databases: need expertise to manage • Database deployment automation flexibility • Deploy your architecture to any number of clouds • "Monitoring at scale" !33
- 34. Conclusions • Containers + PostgreSQL gives you: • Easy-to-setup development environments • Your own production database-as-a-service • Tools to automate management of over 1000s of instances in short-order !34