SlideShare a Scribd company logo
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.
Managing Postgres
At Scale
With Postgres Enterprise Manager
Dave Page
VP & Chief Software Architect, Tools & Installers
PostgreSQL Core Team
1
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.2
WHO AM I?
• EDB:
• VP & Chief Architect:
• Tools
• Cloud products
• Configuration management
• Packaging/distribution
• PostgreSQL:
• Core team member
• pgAdmin lead developer
• Web/sysadmin teams
• PGCAC/PGEU board member
• Dave Page
• @pgsnake
• dpage@pgadmin.org
• dave.page@enterprisedb.com
• https://pgsnake.blogspot.com/
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.3
WHAT IS PEM?
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.4
HIGH LEVEL OVERVIEW
• 24x7 Monitoring and alerting
• Performance visualisation
• Bulk management
• Diagnostics tools
• Database management
• GUI based on pgAdmin 4
MONITOR MANAGE DIAGNOSE
Only solution available that combines
all three tasks into one tool
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.5
ARCHITECTURE - SERVER
• EDB Postgres Advanced Server or PostgreSQL
database:
• Stores system configuration data
• Stores monitoring data
• System scheduler
• Primary PEM Agent:
• Alert processing
• SMTP/SNMP/Nagios spoolers
• Garbage collection
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.6
ARCHITECTURE - AGENTS
• Additional agents installed on managed servers
• Each agent can manage zero or more Postgres
instances
• Monitor local instances or remote (with limitations)
• Responsible for:
• Probe execution and metric collection
• Execution of scheduled tasks such as:
• Server configuration and management
• Software installation/updates
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.7
ARCHITECTURE - UI
• Responsive web based UI
• Based on pgAdmin 4, including all features
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.8
ARCHITECTURE – RESTFUL API
• Allows easy integration with other systems
• Enables scripting of common tasks
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.9
CORE FEATURES
• 24x7 monitoring
• Alerting
• Dashboards
• Custom charts &
dashboards
MONITOR DIAGNOSEMANAGE
• Logging/audit wizards
• Tuning wizard
• Schema management
• Query tool
• Wait State Analysis
• Postgres Expert
• Log Analysis Expert
• Capacity Manager
• SQL Profiler
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.10
MANAGEMENT AT SCALE
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.11
WHAT IS MANAGEMENT AT SCALE?
• Large numbers of Postgres Servers
• More than 3, as that’s when repeating tasks starts to get annoying for me.
• But realistically, let’s say over 10!
• Up to hundreds or more:
• Large organisations, or those using micro-services may have many servers.
• May run in different environments:
• Bare metal
• Virtualised
• Cloud
• Containers
• DBaaS
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.12
WHAT IS MANAGEMENT AT SCALE?
• Large numbers of staff or power users
• Large organisations or those “following the sun” may have lots of DBAs
or sysadmins.
• They may be organised in functional or departmental teams, possibly
with multiple shifts.
• Sysadmins, developers and DBAs
• Engineering, finance and administration, sales, support and services
• Day shift, evening shift, night shift
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.13
WHAT IS MANAGEMENT AT SCALE?
• Different users have different concerns and
requirements, necessitating RBAC (Role Based Access
Controls):
• DBAs can run database management tools
• Sysadmins can use system related tools
• Managers can run reporting tools
• Some users may deploy new databases
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.14
SCALING THE TEAM
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.15
MANAGEMENT BY EXCEPTION
• Human monitoring of everything isn’t feasible, but
computer monitoring is:
• Have the system monitor everything, and tell us (via alerts) when
something seems wrong.
• We can then act, and utilise all the collected data, information and tools
to get a holistic view of the problem and drill down to the root cause.
• Let the system do the hard work!
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.16
MANAGEMENT BY EXCEPTION
• Alerts can be sent via:
• Dashboards (which can be displayed as wall boards)
• Email
• SNMP
• Nagios integration
• User defined scripts
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.17
USERS
• Each user in PEM is a role in the underlying Postgres
data store
• Each user may be a member of one or more additional
roles
• Authentication is handled by Postgres:
• SCRAM
• Kerberos
• LDAP (common, but to be avoided in preference to Kerberos)
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.18
RBAC
• Access to functionality in PEM requires membership in
Postgres roles, e.g:
• pem_manage_alert – manage alert templates and definitions
• pem_config_alert – configure alerts on objects
• pem_comp_sqlprofiler – use the SQL Profiler
• pem_manage_efm – manage EDB Failover Manager
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.19
RBAC
• Role membership can be inherited:
• Don’t grant functional roles directly to user roles!
• Create (or use) group roles, such as:
• pem_super_admin – manage and use all aspects of PEM
• pem_admin – administer/manage/configure agents and servers
• pem_component – use the tools in PEM, e.g. SQL Profiler, Postgres
Expert
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.20
TEAM SUPPORT
• Organise users into teams, through role membership in
the underlying data store.
• Server and Agent properties can include a team (role
name):
• Objects with no team specified are visible to all users (though they may
still not be able to login to them).
• Objects with a team specified are only visible to users who are
members of the team (or superusers, who are members of all roles)
• Postgres’ RLS is used to secure the underlying data.
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.21
SHIFTS (AND ESCALATIONS)
• Email groups can be setup to receive different alerts:
• Time-based groups
• Different groups for different alerts
• Severity based routing
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.22
SCALING THE INFRASTRUCTURE ESTATE
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.23
DEPLOYMENT & UPGRADES
• Utilise native packaging for deployments:
• Better performance
• Fully integrated with the OS package management
• See Devrim’s talk at 1:15PM tomorrow in Back Bay B
• Utilise deployment tools for executing deployments:
• SaltStack
• Puppet
• Chef
• Ansible
• Except on Windows (sigh)!
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.24
DEPLOYMENT & UPGRADES
• Group Policy:
• Requires running on a Windows Domain
• Cannot schedule easily (or at least, couldn’t last time I used it)
• Doesn’t handle config file changes etc. – assumes the use of the Windows
Registry
• Works best with “pure” Windows software/packaging
• PEM:
• Package Management will monitor for updates to EDB-originating Windows
packages
• Can deploy updates and installations on machines with a PEM Agent
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.25
TREEVIEW ORGANISATION
• PEM’s primary navigation is through the
treeview browser on the left of the UI
• This lists all objects, with high level ones
including:
• Database Servers
• PEM Agents (zero or one per host)
• EDB Ark Servers
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.26
TREEVIEW ORGANISATION
• Organise objects into logical
groups to suit your needs:
• Development vs. Production
• AWS vs. Azure vs. Google Cloud
• Finance vs. HR vs. Engineering
• K8s vs. Openshift vs. Cloud vs. Metal
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.27
TREEVIEW ORGANISATION
• Use colouring to
distinguish different
servers or classes of
servers at a glance:
• Development vs. Production
• QA vs. Staging
• Finance vs. HR
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.28
DASHBOARDS
• Simple things can help a lot!
• Search/filtering menus
• Make it easy to work with many
servers and Postgres instances.
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.29
MANAGEMENT EN-MASSE
• Configuration and
management tools
that can operate on
multiple servers at
once
• Examples:
• Tuning Wizard
• Log Wizard
• Audit Wizard
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.30
CROSS-ESTATE REPORTING
• Generate reports to compare performance and other
metrics across multiple servers
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.31
BATCH ANALYTICS/DIAGNOSTICS
• Analyse
configuration and
performance on
multiple servers
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.32
LOG MONITORING
•Configuration
• Configure server and audit logs en-masse using the Audit & Log Wizards
• Deploy configuration at the desired time automatically
• Collection
• Regularly import logs into PEM
• Review logs on the dashboard
• Alerting
• Warn if specific strings are found in logs (e.g. ERROR)
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.33
LOG MONITORING
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.34
CONCLUSIONS
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.35
CONCLUSIONS
Managing at scale is about both scaling the infrastructure estate AND
scaling the management team:
• Management of
infrastructure en-masse:
• Handle many servers in different
environments
• Filtered views
• Bulk operations
• Bulk reporting
• Cross-estate reporting
• Scaling the team:
• Management by exception
• Multiple users and integration
with corporate authentication
• Role Based Access Controls
• Team and Group based
management
• Handling different shift and
escalation patterns
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.
QUESTIONS & DISCUSSION
36
© Copyright EnterpriseDB Corporation, 2019. All rights reserved.
THANK YOU
info@enterprisedb.com
www.enterprisedb.com
37

More Related Content

What's hot (20)

Riding the Second Wave: Open Source for Relational Databases, Enterprise Post...
Riding the Second Wave: Open Source for Relational Databases, Enterprise Post...
EDB
 
Running Highly Available Postgres Databases in Containers
Running Highly Available Postgres Databases in Containers
EDB
 
KT/KTDS Case Study: Open Source Database Adoption in Telecom
KT/KTDS Case Study: Open Source Database Adoption in Telecom
EDB
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data Center
EDB
 
60000 TPS: How many CPUs?, Enterprise Postgres Day
60000 TPS: How many CPUs?, Enterprise Postgres Day
EDB
 
Exploring 10 Postgres Tips and Tricks
Exploring 10 Postgres Tips and Tricks
EDB
 
No Time to Waste: Migrate from Oracle to Postgres in Minutes
No Time to Waste: Migrate from Oracle to Postgres in Minutes
EDB
 
New Enterprise Cloud Database Options for 2019
New Enterprise Cloud Database Options for 2019
EDB
 
Where Should You Deliver Database Services From?
Where Should You Deliver Database Services From?
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11
EDB
 
HTAP By Accident: Getting More From PostgreSQL Using Hardware Acceleration
HTAP By Accident: Getting More From PostgreSQL Using Hardware Acceleration
EDB
 
Automating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with Ansible
EDB
 
New Strategies for Database Modernization
New Strategies for Database Modernization
EDB
 
Hadoop Virtualization - Intel White Paper
Hadoop Virtualization - Intel White Paper
BlueData, Inc.
 
EDB Postgres Platform
EDB Postgres Platform
EDB
 
PostgreSQL to Accelerate Innovation
PostgreSQL to Accelerate Innovation
EDB
 
Why you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on Linux
EDB
 
IBM Lightning Talk
IBM Lightning Talk
MongoDB
 
No Time to Waste: Migrate from Oracle to EDB Postgres in Minutes
No Time to Waste: Migrate from Oracle to EDB Postgres in Minutes
EDB
 
Riding the Second Wave: Open Source for Relational Databases, Enterprise Post...
Riding the Second Wave: Open Source for Relational Databases, Enterprise Post...
EDB
 
Running Highly Available Postgres Databases in Containers
Running Highly Available Postgres Databases in Containers
EDB
 
KT/KTDS Case Study: Open Source Database Adoption in Telecom
KT/KTDS Case Study: Open Source Database Adoption in Telecom
EDB
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data Center
EDB
 
60000 TPS: How many CPUs?, Enterprise Postgres Day
60000 TPS: How many CPUs?, Enterprise Postgres Day
EDB
 
Exploring 10 Postgres Tips and Tricks
Exploring 10 Postgres Tips and Tricks
EDB
 
No Time to Waste: Migrate from Oracle to Postgres in Minutes
No Time to Waste: Migrate from Oracle to Postgres in Minutes
EDB
 
New Enterprise Cloud Database Options for 2019
New Enterprise Cloud Database Options for 2019
EDB
 
Where Should You Deliver Database Services From?
Where Should You Deliver Database Services From?
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11
EDB
 
HTAP By Accident: Getting More From PostgreSQL Using Hardware Acceleration
HTAP By Accident: Getting More From PostgreSQL Using Hardware Acceleration
EDB
 
Automating a PostgreSQL High Availability Architecture with Ansible
Automating a PostgreSQL High Availability Architecture with Ansible
EDB
 
New Strategies for Database Modernization
New Strategies for Database Modernization
EDB
 
Hadoop Virtualization - Intel White Paper
Hadoop Virtualization - Intel White Paper
BlueData, Inc.
 
EDB Postgres Platform
EDB Postgres Platform
EDB
 
PostgreSQL to Accelerate Innovation
PostgreSQL to Accelerate Innovation
EDB
 
Why you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on Linux
EDB
 
IBM Lightning Talk
IBM Lightning Talk
MongoDB
 
No Time to Waste: Migrate from Oracle to EDB Postgres in Minutes
No Time to Waste: Migrate from Oracle to EDB Postgres in Minutes
EDB
 

Similar to Managing Postgres at Scale With Postgres Enterprise Manager (20)

New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
EDB
 
Postgres Enterprise Manager 4.0 Overview
Postgres Enterprise Manager 4.0 Overview
EDB
 
Enterprise-class security with PostgreSQL - 2
Enterprise-class security with PostgreSQL - 2
Ashnikbiz
 
Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11
EDB
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
EDB
 
EnterpriseDB's Best Practices for Postgres DBAs
EnterpriseDB's Best Practices for Postgres DBAs
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Intro.to.pem webinar.slides-061913
Intro.to.pem webinar.slides-061913
EDB
 
EDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 Webinar
EDB
 
Webinar: Managing Postgres at Scale
Webinar: Managing Postgres at Scale
EDB
 
EDB: Power to Postgres
EDB: Power to Postgres
Ashnikbiz
 
5 Postgres DBA Tips
5 Postgres DBA Tips
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Which PostgreSQL is right for your multi cloud strategy? P2
Which PostgreSQL is right for your multi cloud strategy? P2
Ashnikbiz
 
EDB Database Servers and Tools
EDB Database Servers and Tools
Ashnikbiz
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database
EDB
 
10/ EnterpriseDB @ OPEN'16
10/ EnterpriseDB @ OPEN'16
Kangaroot
 
Minimize Headaches with Your Postgres Deployment
Minimize Headaches with Your Postgres Deployment
EDB
 
Tapping into New Postgres Resources with Remote DBAs
Tapping into New Postgres Resources with Remote DBAs
EDB
 
Best Practices for Becoming an Exceptional Postgres DBA
Best Practices for Becoming an Exceptional Postgres DBA
EDB
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
EDB
 
Postgres Enterprise Manager 4.0 Overview
Postgres Enterprise Manager 4.0 Overview
EDB
 
Enterprise-class security with PostgreSQL - 2
Enterprise-class security with PostgreSQL - 2
Ashnikbiz
 
Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11
EDB
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
EDB
 
EnterpriseDB's Best Practices for Postgres DBAs
EnterpriseDB's Best Practices for Postgres DBAs
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Intro.to.pem webinar.slides-061913
Intro.to.pem webinar.slides-061913
EDB
 
EDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 Webinar
EDB
 
Webinar: Managing Postgres at Scale
Webinar: Managing Postgres at Scale
EDB
 
EDB: Power to Postgres
EDB: Power to Postgres
Ashnikbiz
 
5 Postgres DBA Tips
5 Postgres DBA Tips
EDB
 
Remote DBA Service: Powering your DBA needs
Remote DBA Service: Powering your DBA needs
EDB
 
Which PostgreSQL is right for your multi cloud strategy? P2
Which PostgreSQL is right for your multi cloud strategy? P2
Ashnikbiz
 
EDB Database Servers and Tools
EDB Database Servers and Tools
Ashnikbiz
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database
EDB
 
10/ EnterpriseDB @ OPEN'16
10/ EnterpriseDB @ OPEN'16
Kangaroot
 
Minimize Headaches with Your Postgres Deployment
Minimize Headaches with Your Postgres Deployment
EDB
 
Tapping into New Postgres Resources with Remote DBAs
Tapping into New Postgres Resources with Remote DBAs
EDB
 
Best Practices for Becoming an Exceptional Postgres DBA
Best Practices for Becoming an Exceptional Postgres DBA
EDB
 
Ad

More from EDB (20)

Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
EDB
 
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
EDB
 
Migre sus bases de datos Oracle a la nube
Migre sus bases de datos Oracle a la nube
EDB
 
EFM Office Hours - APJ - July 29, 2021
EFM Office Hours - APJ - July 29, 2021
EDB
 
Benchmarking Cloud Native PostgreSQL
Benchmarking Cloud Native PostgreSQL
EDB
 
Las Variaciones de la Replicación de PostgreSQL
Las Variaciones de la Replicación de PostgreSQL
EDB
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQL
EDB
 
Is There Anything PgBouncer Can’t Do?
Is There Anything PgBouncer Can’t Do?
EDB
 
Data Analysis with TensorFlow in PostgreSQL
Data Analysis with TensorFlow in PostgreSQL
EDB
 
Practical Partitioning in Production with Postgres
Practical Partitioning in Production with Postgres
EDB
 
A Deeper Dive into EXPLAIN
A Deeper Dive into EXPLAIN
EDB
 
IOT with PostgreSQL
IOT with PostgreSQL
EDB
 
A Journey from Oracle to PostgreSQL
A Journey from Oracle to PostgreSQL
EDB
 
Psql is awesome!
Psql is awesome!
EDB
 
EDB 13 - New Enhancements for Security and Usability - APJ
EDB 13 - New Enhancements for Security and Usability - APJ
EDB
 
Comment sauvegarder correctement vos données
Comment sauvegarder correctement vos données
EDB
 
Cloud Native PostgreSQL - Italiano
Cloud Native PostgreSQL - Italiano
EDB
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Cloud Native PostgreSQL - APJ
Cloud Native PostgreSQL - APJ
EDB
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
EDB
 
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
EDB
 
Migre sus bases de datos Oracle a la nube
Migre sus bases de datos Oracle a la nube
EDB
 
EFM Office Hours - APJ - July 29, 2021
EFM Office Hours - APJ - July 29, 2021
EDB
 
Benchmarking Cloud Native PostgreSQL
Benchmarking Cloud Native PostgreSQL
EDB
 
Las Variaciones de la Replicación de PostgreSQL
Las Variaciones de la Replicación de PostgreSQL
EDB
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQL
EDB
 
Is There Anything PgBouncer Can’t Do?
Is There Anything PgBouncer Can’t Do?
EDB
 
Data Analysis with TensorFlow in PostgreSQL
Data Analysis with TensorFlow in PostgreSQL
EDB
 
Practical Partitioning in Production with Postgres
Practical Partitioning in Production with Postgres
EDB
 
A Deeper Dive into EXPLAIN
A Deeper Dive into EXPLAIN
EDB
 
IOT with PostgreSQL
IOT with PostgreSQL
EDB
 
A Journey from Oracle to PostgreSQL
A Journey from Oracle to PostgreSQL
EDB
 
Psql is awesome!
Psql is awesome!
EDB
 
EDB 13 - New Enhancements for Security and Usability - APJ
EDB 13 - New Enhancements for Security and Usability - APJ
EDB
 
Comment sauvegarder correctement vos données
Comment sauvegarder correctement vos données
EDB
 
Cloud Native PostgreSQL - Italiano
Cloud Native PostgreSQL - Italiano
EDB
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Cloud Native PostgreSQL - APJ
Cloud Native PostgreSQL - APJ
EDB
 
Best Practices in Security with PostgreSQL
Best Practices in Security with PostgreSQL
EDB
 
Ad

Recently uploaded (20)

Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Anish Kumar
 
Reducing Conflicts and Increasing Safety Along the Cycling Networks of East-F...
Reducing Conflicts and Increasing Safety Along the Cycling Networks of East-F...
Safe Software
 
FIDO Seminar: Perspectives on Passkeys & Consumer Adoption.pptx
FIDO Seminar: Perspectives on Passkeys & Consumer Adoption.pptx
FIDO Alliance
 
Viral>Wondershare Filmora 14.5.18.12900 Crack Free Download
Viral>Wondershare Filmora 14.5.18.12900 Crack Free Download
Puppy jhon
 
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
SOFTTECHHUB
 
“Why It’s Critical to Have an Integrated Development Methodology for Edge AI,...
“Why It’s Critical to Have an Integrated Development Methodology for Edge AI,...
Edge AI and Vision Alliance
 
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
biswajitbanerjee38
 
Kubernetes Security Act Now Before It’s Too Late
Kubernetes Security Act Now Before It’s Too Late
Michael Furman
 
The State of Web3 Industry- Industry Report
The State of Web3 Industry- Industry Report
Liveplex
 
Oracle Cloud Infrastructure Generative AI Professional
Oracle Cloud Infrastructure Generative AI Professional
VICTOR MAESTRE RAMIREZ
 
Crypto Super 500 - 14th Report - June2025.pdf
Crypto Super 500 - 14th Report - June2025.pdf
Stephen Perrenod
 
Your startup on AWS - How to architect and maintain a Lean and Mean account J...
Your startup on AWS - How to architect and maintain a Lean and Mean account J...
angelo60207
 
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
Edge AI and Vision Alliance
 
cnc-drilling-dowel-inserting-machine-drillteq-d-510-english.pdf
cnc-drilling-dowel-inserting-machine-drillteq-d-510-english.pdf
AmirStern2
 
Supporting the NextGen 911 Digital Transformation with FME
Supporting the NextGen 911 Digital Transformation with FME
Safe Software
 
High Availability On-Premises FME Flow.pdf
High Availability On-Premises FME Flow.pdf
Safe Software
 
Edge-banding-machines-edgeteq-s-200-en-.pdf
Edge-banding-machines-edgeteq-s-200-en-.pdf
AmirStern2
 
Analysis of the changes in the attitude of the news comments caused by knowin...
Analysis of the changes in the attitude of the news comments caused by knowin...
Matsushita Laboratory
 
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdf
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdf
Muhammad Rizwan Akram
 
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Alliance
 
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Anish Kumar
 
Reducing Conflicts and Increasing Safety Along the Cycling Networks of East-F...
Reducing Conflicts and Increasing Safety Along the Cycling Networks of East-F...
Safe Software
 
FIDO Seminar: Perspectives on Passkeys & Consumer Adoption.pptx
FIDO Seminar: Perspectives on Passkeys & Consumer Adoption.pptx
FIDO Alliance
 
Viral>Wondershare Filmora 14.5.18.12900 Crack Free Download
Viral>Wondershare Filmora 14.5.18.12900 Crack Free Download
Puppy jhon
 
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
SOFTTECHHUB
 
“Why It’s Critical to Have an Integrated Development Methodology for Edge AI,...
“Why It’s Critical to Have an Integrated Development Methodology for Edge AI,...
Edge AI and Vision Alliance
 
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
biswajitbanerjee38
 
Kubernetes Security Act Now Before It’s Too Late
Kubernetes Security Act Now Before It’s Too Late
Michael Furman
 
The State of Web3 Industry- Industry Report
The State of Web3 Industry- Industry Report
Liveplex
 
Oracle Cloud Infrastructure Generative AI Professional
Oracle Cloud Infrastructure Generative AI Professional
VICTOR MAESTRE RAMIREZ
 
Crypto Super 500 - 14th Report - June2025.pdf
Crypto Super 500 - 14th Report - June2025.pdf
Stephen Perrenod
 
Your startup on AWS - How to architect and maintain a Lean and Mean account J...
Your startup on AWS - How to architect and maintain a Lean and Mean account J...
angelo60207
 
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
Edge AI and Vision Alliance
 
cnc-drilling-dowel-inserting-machine-drillteq-d-510-english.pdf
cnc-drilling-dowel-inserting-machine-drillteq-d-510-english.pdf
AmirStern2
 
Supporting the NextGen 911 Digital Transformation with FME
Supporting the NextGen 911 Digital Transformation with FME
Safe Software
 
High Availability On-Premises FME Flow.pdf
High Availability On-Premises FME Flow.pdf
Safe Software
 
Edge-banding-machines-edgeteq-s-200-en-.pdf
Edge-banding-machines-edgeteq-s-200-en-.pdf
AmirStern2
 
Analysis of the changes in the attitude of the news comments caused by knowin...
Analysis of the changes in the attitude of the news comments caused by knowin...
Matsushita Laboratory
 
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdf
ENERGY CONSUMPTION CALCULATION IN ENERGY-EFFICIENT AIR CONDITIONER.pdf
Muhammad Rizwan Akram
 
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Alliance
 

Managing Postgres at Scale With Postgres Enterprise Manager

  • 1. © Copyright EnterpriseDB Corporation, 2019. All rights reserved. Managing Postgres At Scale With Postgres Enterprise Manager Dave Page VP & Chief Software Architect, Tools & Installers PostgreSQL Core Team 1
  • 2. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.2 WHO AM I? • EDB: • VP & Chief Architect: • Tools • Cloud products • Configuration management • Packaging/distribution • PostgreSQL: • Core team member • pgAdmin lead developer • Web/sysadmin teams • PGCAC/PGEU board member • Dave Page • @pgsnake • [email protected][email protected] • https://pgsnake.blogspot.com/
  • 3. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.3 WHAT IS PEM?
  • 4. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.4 HIGH LEVEL OVERVIEW • 24x7 Monitoring and alerting • Performance visualisation • Bulk management • Diagnostics tools • Database management • GUI based on pgAdmin 4 MONITOR MANAGE DIAGNOSE Only solution available that combines all three tasks into one tool
  • 5. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.5 ARCHITECTURE - SERVER • EDB Postgres Advanced Server or PostgreSQL database: • Stores system configuration data • Stores monitoring data • System scheduler • Primary PEM Agent: • Alert processing • SMTP/SNMP/Nagios spoolers • Garbage collection
  • 6. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.6 ARCHITECTURE - AGENTS • Additional agents installed on managed servers • Each agent can manage zero or more Postgres instances • Monitor local instances or remote (with limitations) • Responsible for: • Probe execution and metric collection • Execution of scheduled tasks such as: • Server configuration and management • Software installation/updates
  • 7. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.7 ARCHITECTURE - UI • Responsive web based UI • Based on pgAdmin 4, including all features
  • 8. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.8 ARCHITECTURE – RESTFUL API • Allows easy integration with other systems • Enables scripting of common tasks
  • 9. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.9 CORE FEATURES • 24x7 monitoring • Alerting • Dashboards • Custom charts & dashboards MONITOR DIAGNOSEMANAGE • Logging/audit wizards • Tuning wizard • Schema management • Query tool • Wait State Analysis • Postgres Expert • Log Analysis Expert • Capacity Manager • SQL Profiler
  • 10. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.10 MANAGEMENT AT SCALE
  • 11. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.11 WHAT IS MANAGEMENT AT SCALE? • Large numbers of Postgres Servers • More than 3, as that’s when repeating tasks starts to get annoying for me. • But realistically, let’s say over 10! • Up to hundreds or more: • Large organisations, or those using micro-services may have many servers. • May run in different environments: • Bare metal • Virtualised • Cloud • Containers • DBaaS
  • 12. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.12 WHAT IS MANAGEMENT AT SCALE? • Large numbers of staff or power users • Large organisations or those “following the sun” may have lots of DBAs or sysadmins. • They may be organised in functional or departmental teams, possibly with multiple shifts. • Sysadmins, developers and DBAs • Engineering, finance and administration, sales, support and services • Day shift, evening shift, night shift
  • 13. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.13 WHAT IS MANAGEMENT AT SCALE? • Different users have different concerns and requirements, necessitating RBAC (Role Based Access Controls): • DBAs can run database management tools • Sysadmins can use system related tools • Managers can run reporting tools • Some users may deploy new databases
  • 14. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.14 SCALING THE TEAM
  • 15. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.15 MANAGEMENT BY EXCEPTION • Human monitoring of everything isn’t feasible, but computer monitoring is: • Have the system monitor everything, and tell us (via alerts) when something seems wrong. • We can then act, and utilise all the collected data, information and tools to get a holistic view of the problem and drill down to the root cause. • Let the system do the hard work!
  • 16. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.16 MANAGEMENT BY EXCEPTION • Alerts can be sent via: • Dashboards (which can be displayed as wall boards) • Email • SNMP • Nagios integration • User defined scripts
  • 17. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.17 USERS • Each user in PEM is a role in the underlying Postgres data store • Each user may be a member of one or more additional roles • Authentication is handled by Postgres: • SCRAM • Kerberos • LDAP (common, but to be avoided in preference to Kerberos)
  • 18. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.18 RBAC • Access to functionality in PEM requires membership in Postgres roles, e.g: • pem_manage_alert – manage alert templates and definitions • pem_config_alert – configure alerts on objects • pem_comp_sqlprofiler – use the SQL Profiler • pem_manage_efm – manage EDB Failover Manager
  • 19. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.19 RBAC • Role membership can be inherited: • Don’t grant functional roles directly to user roles! • Create (or use) group roles, such as: • pem_super_admin – manage and use all aspects of PEM • pem_admin – administer/manage/configure agents and servers • pem_component – use the tools in PEM, e.g. SQL Profiler, Postgres Expert
  • 20. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.20 TEAM SUPPORT • Organise users into teams, through role membership in the underlying data store. • Server and Agent properties can include a team (role name): • Objects with no team specified are visible to all users (though they may still not be able to login to them). • Objects with a team specified are only visible to users who are members of the team (or superusers, who are members of all roles) • Postgres’ RLS is used to secure the underlying data.
  • 21. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.21 SHIFTS (AND ESCALATIONS) • Email groups can be setup to receive different alerts: • Time-based groups • Different groups for different alerts • Severity based routing
  • 22. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.22 SCALING THE INFRASTRUCTURE ESTATE
  • 23. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.23 DEPLOYMENT & UPGRADES • Utilise native packaging for deployments: • Better performance • Fully integrated with the OS package management • See Devrim’s talk at 1:15PM tomorrow in Back Bay B • Utilise deployment tools for executing deployments: • SaltStack • Puppet • Chef • Ansible • Except on Windows (sigh)!
  • 24. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.24 DEPLOYMENT & UPGRADES • Group Policy: • Requires running on a Windows Domain • Cannot schedule easily (or at least, couldn’t last time I used it) • Doesn’t handle config file changes etc. – assumes the use of the Windows Registry • Works best with “pure” Windows software/packaging • PEM: • Package Management will monitor for updates to EDB-originating Windows packages • Can deploy updates and installations on machines with a PEM Agent
  • 25. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.25 TREEVIEW ORGANISATION • PEM’s primary navigation is through the treeview browser on the left of the UI • This lists all objects, with high level ones including: • Database Servers • PEM Agents (zero or one per host) • EDB Ark Servers
  • 26. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.26 TREEVIEW ORGANISATION • Organise objects into logical groups to suit your needs: • Development vs. Production • AWS vs. Azure vs. Google Cloud • Finance vs. HR vs. Engineering • K8s vs. Openshift vs. Cloud vs. Metal
  • 27. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.27 TREEVIEW ORGANISATION • Use colouring to distinguish different servers or classes of servers at a glance: • Development vs. Production • QA vs. Staging • Finance vs. HR
  • 28. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.28 DASHBOARDS • Simple things can help a lot! • Search/filtering menus • Make it easy to work with many servers and Postgres instances.
  • 29. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.29 MANAGEMENT EN-MASSE • Configuration and management tools that can operate on multiple servers at once • Examples: • Tuning Wizard • Log Wizard • Audit Wizard
  • 30. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.30 CROSS-ESTATE REPORTING • Generate reports to compare performance and other metrics across multiple servers
  • 31. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.31 BATCH ANALYTICS/DIAGNOSTICS • Analyse configuration and performance on multiple servers
  • 32. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.32 LOG MONITORING •Configuration • Configure server and audit logs en-masse using the Audit & Log Wizards • Deploy configuration at the desired time automatically • Collection • Regularly import logs into PEM • Review logs on the dashboard • Alerting • Warn if specific strings are found in logs (e.g. ERROR)
  • 33. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.33 LOG MONITORING
  • 34. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.34 CONCLUSIONS
  • 35. © Copyright EnterpriseDB Corporation, 2019. All rights reserved.35 CONCLUSIONS Managing at scale is about both scaling the infrastructure estate AND scaling the management team: • Management of infrastructure en-masse: • Handle many servers in different environments • Filtered views • Bulk operations • Bulk reporting • Cross-estate reporting • Scaling the team: • Management by exception • Multiple users and integration with corporate authentication • Role Based Access Controls • Team and Group based management • Handling different shift and escalation patterns
  • 36. © Copyright EnterpriseDB Corporation, 2019. All rights reserved. QUESTIONS & DISCUSSION 36
  • 37. © Copyright EnterpriseDB Corporation, 2019. All rights reserved. THANK YOU [email protected] www.enterprisedb.com 37