GitLab PostgresMortem: Lessons Learned
0 likes498 views
1. A PostgreSQL database outage occurred at GitLab on January 31st due to a combination of factors including an increase in load, replication lag, and the deletion of the database directory. 2. Lessons learned include monitoring replication, using tools like pg_basebackup properly, and having backups and disaster recovery processes in place. 3. Recommended preventative measures include setting sane configuration values, automated testing of backups, assigning an owner for data durability, and improving documentation.
GitLab PostgresMortem: Lessons Learned
- 2. dataegret.com 31 January Events Failure's key points Preventative measures https://goo.gl/GO5rYJ 02 03 01
- 4. 17:20 - an LVM snapshot of the production db was taken. 19:00 - database load increased due to spam. 23:00 - secondary's replication process started to lag behind. 23:30 - PostgreSQL database directory was wiped. 31 January events01 dataegret.com
- 6. 1.LVM snapshots and staging provisioning. 2.When a replica start to lag. 3.Do pg_basebackup properly – part 1. 4.max_wal_senders was exceeded, but how? 5.max_connections = 8000. 6.pg_basebackup «stuck» – do pg_basebackup properly – part 2. 7.strace: good thing in wrong place. 8.rm or not rm? 9.A bit about backup. 10.Different PG versions on the production. 11.Broken mail. 31 January events02 dataegret.com
- 7. Snapshot impact on underlying storage. Provisioning from backup. Staging based on LVM snapshots02 dataegret.com
- 8. Re-initialize the standby. Monitoring with pg_stat_replication. Use wal_keep_segments while troubleshooting. Use WAL archive. When a replica started to lag02 dataegret.com
- 9. Do pg_basebackup into clean directory. Remove «unnecessary» directory. Use mv instead of rm. Do pg_basebackup properly. Part 102 dataegret.com
- 10. There was only one standby (which was failed). Increase max_wal_senders. Check who has stolen connections. The limit was exceeded by concurrent pg_basebackups. max_wal_senders was exceeded.02 dataegret.com
- 11. More than 500 is bad idea. Use pgbouncer to reduce the number of server connections. max_connections = 800002 dataegret.com
- 12. Don't run more than one pg_basebackups. It didn't stuck, it waited for the checkpoint. Use «-c» option to make fast checkpoint. Do pg_basebackup properly. Part 202 dataegret.com
- 13. Strace isn't a good tool in that case. Use strace for system errors tracing. Check stack trace from /proc/<pid>/stack or GDB. Good things in wrong place.02 dataegret.com
- 14. Data directory was cleaned with rm. Use mv instead of rm. rm or not rm02 dataegret.com
- 15. Daily pg_dump. Daily LVM snapshot. Daily Azure snapshot. PostgreSQL streaming replication. Basebackup with WAL archive. A bit about backup02 dataegret.com
- 16. Clean out old packages after major upgrade. Different versions on a production02 dataegret.com
- 17. Setup cron, but forgot notifications. Use reliable notification systems. Different versions on a production02 dataegret.com
- 19. 1. Update PS1 across all hosts to more clearly differentiate between hosts and environments. 2. Prometheus monitoring for backups. 3. Set PostgreSQL's max_connections to a sane value. 4. Investigate Point in time recovery & continuous archiving for PostgreSQL. 5. Hourly LVM snapshots of the production databases. 6. Azure disk snapshots of production databases. 7. Move staging to the ARM environment. 8. Recover production replica(s). 9. Automated testing of recovering PostgreSQL database backups. 10.Improve PostgreSQL replication documentation/runbooks. 11.Investigate pgbarman for creating PostgreSQL backups. 12.Investigate using WAL-E as a means of Database Backup and Realtime Replication. 13.Build Streaming Database Restore. 14.Assign an owner for data durability. Different versions on a production03 dataegret.com
- 20. 1. Update PS1 across all hosts. Looks OK. 2. Prometheus monitoring for backups. Size, number, age and recovery status. 3. Set PostgreSQL's max_connections to a sane value. Better use pgbouncer. 4. Investigate PITR & continuous archiving for PostgreSQL. Yes, as the part of the backup. Preventative measures03 dataegret.com
- 21. 5. Hourly LVM snapshots of the production databases. Looks unnecessary. 6. Azure disk snapshots of production databases. Looks unnecessary. 7. Move staging to the ARM environment. Very and very suspicious. 8. Recover production replica(s). Do that asap. Preventative measures03 dataegret.com
- 22. 9. Automated testing of recovering database backups. YES! 10. Improve documentation/runbooks. You need a bureaucrat. 11. Investigate pgbarman. Looks OK, Barman is stable and reliable. 12. Investigate using WAL-E. Looks OK, WAL-E is the «setup and forget». Preventative measures03 dataegret.com
- 23. 13. Build Streaming Database Restore. Corresponds with p.9. 14. Assign an owner for data durability. Hire a DBA. Preventative measures03 dataegret.com
- 24. Check and monitor backups. Create an emergency instructions. Learn to use tools properly. Lessons learned03 dataegret.com
- 25. Postmortem of database outage of January 31 https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ PostgreSQL Statistics Collector: pg_stat_replication view https://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-REPLICATION-VIEW pg_basebackup utility https://www.postgresql.org/docs/current/static/app-pgbasebackup.html PostgreSQL Replication https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html PgBouncer https://pgbouncer.github.io/ https://wiki.postgresql.org/wiki/PgBouncer Barman http://www.pgbarman.org/ Links03 dataegret.com