PostgreSQL and Linux Containers
Download as pptx, pdf18 likes11,232 views
This document provides an overview of Linux containers and Docker containers. It begins with definitions of containers and their advantages over virtual machines. It describes early implementations of containers like chroot, Jails, and Zones. It explains the underlying kernel technologies like cgroups and namespaces that enable Linux containers. It provides instructions for using LXC and systemd-nspawn to deploy basic containers. It then focuses on Docker containers, covering installation, images, volumes, and best practices for running applications like PostgreSQL in Docker containers.
![Dockerfile – Custom Recipe
• Container images are created using “Dockerfile”
• Build an image using the recipe file
• Push to a public or private registry (hub account login or private registry needed)
FROM centos:centos7.0.1406
RUN yum install -y epel-release
RUN yum install -y nginx
EXPOSE 80
CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g "daemon off;""]
docker build –t jkshah/nginx –t .
docker push jkshah/nginx:latest](https://image.slidesharecdn.com/postgresqllinuxcontainersmeetup2015jan05-160106065412/85/PostgreSQL-and-Linux-Containers-18-320.jpg)
![Docker Images
• Docker Images are layered templates used by containers instances
• Container instance images are layered Copy on Write Images based on Docker images
• You can mutate your local container images (unless you use --read-only flag)
• Not good for fast and frequent changing data
• Can select a different underlying type using --storage-driver
docker push jkshah/nginx:latest
Do you really want to push to public registry? [y/n]: y
The push refers to a repository [docker.io/jkshah/nginx] (len: 1)
d498b5680966: Pushed
4b1d16518ce1: Pushed
3185bef36db4: Pushed
2f7013eef4b8: Pushed
539eca37bade: Pushed
f1b10cd84249: Pushed
latest: digest:
sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size:
10608](https://image.slidesharecdn.com/postgresqllinuxcontainersmeetup2015jan05-160106065412/85/PostgreSQL-and-Linux-Containers-19-320.jpg)
PostgreSQL and Linux Containers
- 1. PostgreSQL and Linux Containers Jignesh Shah Founding Team @appOrbit SF Bay Area PostgreSQL User Group – Jan 5th 2015
- 2. Disclaimer The views expressed in this presentation are my own and do not necessarily reflect the views of appOrbit.
- 3. About Jignesh (@jkshah) • appOrbit • My focus is on managing Application Data running in Containers • VMware • Lead and manage Postgres and Data Management teams at VMware for various products embedding PostgreSQL running in virtualized embedded instances • Sun Microsystems • Team Member of first published SpecJAppServer 2004 benchmark with PostgreSQL • Performance of PostgreSQL on Solaris/Sun Servers • Working with PostgreSQL community since 2005 • http://jkshah.blogspot.com/2005/04/profiling-postgresql-using-dtrace-on_22.html • Working with Container technologies (Solaris Zones) since 2004 • http://jkshah.blogspot.com/2004/08/db2-working-under-solaris-10-zones_30.html
- 4. Agenda • Containers • Definition • Early examples • Linux Containers • Underlying Technologies • LXC • Systemd-nspawn • Docker Containers • Installation • Images • Volumes • PostgreSQL in Docker Container • What it means • Best practices
- 5. What are Containers? • OS Level virtualization where kernel allows for multiple isolated user-space instances Operating System Bare Metal Server OS Bare Metal Server Hypervisor OS Operating System Bare Metal Server C C C C C OS Bare Metal Server Hypervisor OS C C C C
- 6. Advantages of Containers • Lower footprint • Very Quick Startup and Shutdown time • Density • Nesting
- 7. Disadvantages of Containers • Same Kernel version • Cannot run other OS natively • Security (to be improved)
- 8. Where to use container? • Recreate identical environment (cookie-cutter) • Resource Grouping of specific processes in heavily loaded server • Handling multiple versions of software applications • Ephemeral application instances (Dev/Test) • Many more
- 9. Implementations of Containers • Chroot circa 1982 • FreeBSD Jails circa 2000 • Solaris Zones circa 2004 • Meiosys – MetaClusters with Checkpoint/Restore 2004-05 • Linux OpenVZ circa 2005 (not in mainstream Linux) • AIX WPARs circa 2007 • LXC circa 2008 • Systemd-nspawn circa 2010-2013 • Docker circa 2013
- 10. What makes containers possible? • Process Group Isolation • Filesystem Isolation • Network Isolation • CPU Isolation • Memory Isolation • Example – Solaris Containers • Base implementation provided Process, Filesystem and Network Isolation • Resource pools consisting of CPU, memory was originally used • Branded Zones – Userland Library Isolation
- 11. What makes Linux containers possible ? • cgroups • Allows limitation and prioritization of resources (CPU, memory, block I/O, network, etc.) • Namespace isolation • Mount namespace • PID namespace • Network namespace • UTS (Allows changing hostname, domainname) • IPC Namespace • User namespace • LXC • Combines kernel’s cgroup and namespaces to provide an isolated environment
- 12. LXC • CentOS 7 + EPEL Repository set • Commands available • Quick Guide to use an LXC based container of busybox lxc-attach lxc-clone lxc-destroy lxc-ls lxc-stop lxc-usernsexec lxc-autostart lxc-config lxc-execute lxc-monitor lxc-top lxc-wait lxc-cgroup lxc-console lxc-freeze lxc-snapshot lxc-unfreeze lxc-checkconfig lxc-create lxc-info lxc-start lxc-unshare wget https://www.busybox.net/downloads/binaries/busybox-x86_64 -O busybox chmod a+x busybox PATH=$(pwd):$PATH lxc-create -t busybox -n mycontainer lxc-start -d -n mycontainer lxc-console –n mycontainer # (Use CTRL-A Q to exit console mode) lxc-stop -n mycontainer lxc-destroy -n mycontainer yum install epel-release yum install bridge-utils libvirt lxc lxc-templates
- 13. Systemd-nspawn • Systemd • Replacement of SysV init scripts • Systemd-nspawn • Used to run a command or OS in light weight namespace container • Installed on most newer distros by default • Commands available systemd-analyze systemd-delta systemd-nspawn systemd-ask-password systemd-detect-virt systemd-run systemd-cat systemd-cgls systemd-loginctl systemd-sysv-convert systemd-cgtop systemd-machine-id-setup systemd-coredumpctl systemd-notify systemd-tty-ask-password-agent systemd-inhibit systemd-stdio-bridge systemd-tmpfiles systemctl machinectl hostnamectl journalctl yum install systemd
- 14. Systemd-nspawn • Quick guide to a container deployment using systemd-nspawn • Useful tools machinectl status mycontainer systemd-cgls systemd-cgtop # Create an Image yum -y --releasever=7 --nogpg --installroot=/mycontainers/centos7 install systemd passwd yum fedora-release vim-minimal # Change the root password in the image systemd-nspawn -D /mycontainers/centos7 passwd exit # Start the container as if booting into the container image systemd-nspawn -D /mycontainers/centos7 –M mycontainer –b # Get into the container nsenter -m -u -i -n -p –t $PID machinectl login mycontainer
- 15. Trends of Container Technologies
- 16. Disruption of trends by Docker
- 17. Docker • Installation • Commands available with docker binary • Quick Guide to use a docker based container attach build commit cp create diff events exec export history images import info inspect kill load login logout logs pause port ps pull push rename restart rm rmi run save search start stats stop tag top unpause version wait docker run --name mycontainer -e POSTGRES_PASSWORD=mysecretpassword -d postgres docker exec -ti mycontainer psql -U postgres docker stop mycontainer docker rm mycontainer docker rmi postgres yum install docker systemctl start docker
- 18. Dockerfile – Custom Recipe • Container images are created using “Dockerfile” • Build an image using the recipe file • Push to a public or private registry (hub account login or private registry needed) FROM centos:centos7.0.1406 RUN yum install -y epel-release RUN yum install -y nginx EXPOSE 80 CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g "daemon off;""] docker build –t jkshah/nginx –t . docker push jkshah/nginx:latest
- 19. Docker Images • Docker Images are layered templates used by containers instances • Container instance images are layered Copy on Write Images based on Docker images • You can mutate your local container images (unless you use --read-only flag) • Not good for fast and frequent changing data • Can select a different underlying type using --storage-driver docker push jkshah/nginx:latest Do you really want to push to public registry? [y/n]: y The push refers to a repository [docker.io/jkshah/nginx] (len: 1) d498b5680966: Pushed 4b1d16518ce1: Pushed 3185bef36db4: Pushed 2f7013eef4b8: Pushed 539eca37bade: Pushed f1b10cd84249: Pushed latest: digest: sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size: 10608
- 20. Docker Volumes • Persists beyond the life of a Docker container • VOLUME command in Dockerfile or • Using –v using docker run command • Automatically created if not already present during docker run • Not part of docker push/pull operations • Can select a non-local directory using --volume-driver • Third party components required to get multi-host support (NFS, etc ) • On CentOS with SELinux enabled need to set security context • Different options using –v • -v /hostsrc/data:/opt/data:ro # for read only volumes (default rw) • -v /hostsrc/data:/opt/data:Z # Z – private volume, z – shared volume • -v /etc/nginx.conf:/etc/nginx.conf # for mounting a single file only • Volumes can be shared from another container using --volumes-from on same host • Docker 1.9 gives first class status to Docker Volumes chcon -Rt svirt_sandbox_file_t /hostpath/pgdata
- 21. PostgreSQL in Docker container • Quick Deployment: • Check Deployment: • Use inspect command to get more information about the container • To check PostgreSQL system logs docker run --name mycontainer -v /hostpath/pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword -d postgres docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8d720b31c1fe postgres "/docker-entrypoint.s" 5 minutes ago Up 5 minutes 5432/tcp mycontainer docker logs mycontainer The files belonging to this database system will be owned by user "postgres". This user must also own the server process. … LOG: database system was shut down at 2016-01-04 22:58:40 UTC LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started docker inspect mycontainer
- 22. PostgreSQL in docker container • Part of Postgres Dockerfile has • Impacts • Port is exposed only to other containers directly linking with this container • Data is persistent only for the life of container (docker rm) but not easily accessible outside the container • -v enables to access the data outside container • Linking to PostgreSQL Server container • Ports are exposed externally using –p option • Verify using docker run --name myapp --link mycontainer:ds2db -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=mysecretpassword -p 8080:80 -d jkshah/dvdstore2 EXPOSE 5432 VOLUME /var/lib/postgresql/data docker port myapp 80/tcp -> 0.0.0.0:8080
- 23. PostgreSQL in docker container • Check for statistics using • Check for top processes in a container docker stats mycontainer myapp CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O myapp 0.01% 51.51 MB/1.924 GB 2.68% 54.42 kB/7.576 MB mycontainer 0.01% 133.3 MB/1.924 GB 6.93% 7.577 MB/54.42 kB docker top mycontainer UID PID PPID C STIME TTY TIME CMD polkitd 25313 10698 0 16:58 ? 00:00:00 postgres polkitd 25385 25313 0 16:58 ? 00:00:00 postgres: checkpointer process polkitd 25386 25313 0 16:58 ? 00:00:00 postgres: writer process polkitd 25387 25313 0 16:58 ? 00:00:00 postgres: wal writer process polkitd 25388 25313 0 16:58 ? 00:00:00 postgres: autovacuum launcher process polkitd 25389 25313 0 16:58 ? 00:00:00 postgres: stats collector process docker top myapp UID PID PPID C STIME TTY TIME CMD root 30747 10698 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30796 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30797 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30798 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30799 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30800 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
- 24. PostgreSQL in docker container • Alternate way to see Process Tree using systemd-cgls tool systemd-cgls ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 ├─user.slice │ └─user-0.slice │ └─session-34.scope │ ├─25129 sshd: root@pts/0 │ ├─25131 -bash │ └─31397 systemd-cgls └─system.slice ├─docker-9457652d7f6ec24ebd95305e788fe39030b049deb22f240ee2b7383488d0c215.scope │ ├─30747 httpd -D FOREGROUND │ ├─30796 httpd -D FOREGROUND │ ├─30797 httpd -D FOREGROUND │ ├─30798 httpd -D FOREGROUND │ ├─30799 httpd -D FOREGROUND │ └─30800 httpd -D FOREGROUND ├─docker-8d720b31c1fe0de77d3cd89942c1a72902b67d466f821ce63ee1271561a36451.scope │ ├─25313 postgres │ ├─25385 postgres: checkpointer process │ ├─25386 postgres: writer process │ ├─25387 postgres: wal writer process │ ├─25388 postgres: autovacuum launcher process │ └─25389 postgres: stats collector process ├─docker.service │ ├─10698 /usr/bin/docker daemon --selinux-enabled │ └─30739 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -contain...
- 25. PostgreSQL in Docker container • Alternate way to see top containers using systemd-cgtop systemd-cgtop Path Tasks %CPU Memory Input/s Output/s / 100 0.9 1.5G - - /user.slice 3 0.9 70.7M - - /system.slice - 0.0 268.5M - - /system.slice/docker.service 2 0.0 31.9M - - /system.slice/nimbus.service 5 0.0 20.0K - - /system.slice/tuned.service 1 0.0 36.0K - - /system.slice/dock...49deb22f240ee2b7383488d0c215.scope 6 0.0 49.1M - - /system.slice/auditd.service 1 - 11.9M - - /system.slice/avahi-daemon.service 2 - 12.0K - - /system.slice/crond.service 1 - 92.0K - - /system.slice/dbus.service 1 - - - - /system.slice/dock...7d466f821ce63ee1271561a36451.scope 6 - 127.3M - - /system.slice/gssproxy.service 1 - - - - /system.slice/libvirtd.service 3 - - - - /system.slice/lvm2-lvmetad.service 1 - - - - /system.slice/polkit.service 1 - 16.0K - - /system.slice/postfix.service 3 - 3.5M - - /system.slice/rsyslog.service 1 - 2.1M - - /system.slice/sshd.service 1 - 3.0M - - /system.slice/system-getty.slice/[email protected] 1 - - - - /system.slice/systemd-journald.service 1 - 16.0M - - /system.slice/systemd-logind.service 1 - 20.0K - - /system.slice/systemd-udevd.service 1 - 520.0K - - /system.slice/xe-linux-distribution.service 2 - 1.0M - - /user.slice/user-0.slice/session-34.scope 3 - - - -
- 26. PostgreSQL as a database server container • Maybe you want a database server standalone • Not all database clients will be on docker containers in the same host • Need to limit memory usage • Need different layout of how files are distributed (separage XLOG ) • Use the –p option to make the port available even to non containers clients • Use –m to limit memory usage by the DB server (by default it can see and use all) • Note this does not set shared buffers automatically with the library image docker run --name mycontainer -m 4g -e POSTGRES_PASSWORD=mysecretpassword -v /hostpath/pgdata:/var/lib/postgresql/data -p 5432:5432 -d postgres
- 27. PostgreSQL in an enterprise environment • However for a real production use case we would need • Bigger shared memory configurations • Need different layout of how files are distributed (separage XLOG ) • Ability to backup the database • Ability to setup replication • etc • In short we need a more custom image of PostgreSQL
- 28. Best Practices for custom image • For production install customize the docker image • Allocate proper memory limits - example 8GB • All pagecache usage shows up as docker container memory usage • Bump up shared buffers and other parameters as required • Hint: use PostgreSQL 9.3 or later otherwise have to privileged containers • http://jkshah.blogspot.com/2015/09/is-it-privilege-to-run-container-in.html • Support multiple volumes in your image • Pg_xlog • PITR archives • Tablespaces as required • Full Backup directory • PostgreSQL Extensions • Setup replication support • Out of box replication setup • Monitoring Tool • Your favorite monitoring agent
- 29. References • http://www.freedesktop.org/wiki/Software/systemd/ • https://linuxcontainers.org/ • http://www.haifux.org/lectures/299/netLec7.pdf • http://haifux.org/lectures/320/netLec8_final.pdf
- 30. Revolutionary platform to encapsulate and manage both legacy and new micro-services based applications with data and configurations running on containers and virtual machines in a private, public or hybrid cloud. Put your DevOps on hyper-drive! Efficiency and Velocity for Apps & Infrastructure We are HIRING !!!