Advanced WAL File Management With OmniPITR

Advanced WAL File Management
With OmniPITR

Robert Treat, JDCon 2011 #pgeast

/ Presentation

Thursday, March 31, 2011

Who Am I?

• Robert Treat
• OmniTI
• Database Management and Consulting
• We’re Hiring!
• Postgres
• Major Contributor, Web, Advocacy, Random
• xzilla.net
• @robtreat2


What is PITR?


What is PITR?

•Postgres emits WAL ﬁles


Digression


Digression

•WAL Logs?

•Write Ahead Log Logs?

•OTOH, they are stored in pg_xlog...


What is PITR?


•Send the WAL ﬁle to another server


What is PITR?



•The other server can replay the WAL


What is PITR?

Use combination of
data ﬁles and WAL
to make more postgrezes


What is PITR?


Use combination of
data ﬁles and WAL


What is PITR?



Use combination of
data ﬁles and WAL


What is PITR?



•The other server can replay the WAL

Use combination of
data ﬁles and WAL


A Brief History of PITR



•8.1 (really)



•8.1 (really)

•8.2 (warm standby)



•8.1 (really)


•9.0 (hot standby)



•8.1 (really)


•9.0 (hot standby)

•9.0 (streaming replication)


Ghetto Style

archive_command =
'rsync %p sdb2:/data/pgsql/sdb1/84/walarchive/% f'


Ghetto Style

•Simple, Better Than You’d Think

archive_command =
'rsync %p sdb2:/data/pgsql/sdb1/84/walarchive/% f'


Early Version Of A PITR Script

archive_command = ‘cp -i %p /data/pgsql/82_walarchives/%f </dev/null’


•Kind of Hacky




opendir(DIR, ".");
my @wals2move = grep {
-f $_ &&
(stat(_))[9] + $AGE < time() &&
(
# 000000010000007200000031
( /^[0-9A-F]{24}$/ && (stat(_))[7] == $WALSIZE ) ||
# 000000010000007200000031.0012C968.backup
( /^[0-9A-F]{24}.[0-9A-F]{8}.backup$/ )
) &&
( $mtime{$_} = (stat(_))[9] ) # Always true (for later)
} readdir(DIR);
closedir(DIR);


•Kind of Hacky

opendir(DIR, ".");
my @wals2move = grep {
-f $_ &&
(stat(_))[9] + $AGE < time() &&
(
# 000000010000007200000031
( /^[0-9A-F]{24}$/ && (stat(_))[7] == $WALSIZE ) ||
# 000000010000007200000031.0012C968.backup
( /^[0-9A-F]{24}.[0-9A-F]{8}.backup$/ )
) &&
( $mtime{$_} = (stat(_))[9] ) # Always true (for later)
} readdir(DIR);
closedir(DIR);




@wals2move = sort { $mtime{$a} <=> $mtime{$b} } @wals2move;
if($TARGET) {
foreach (@wals2move) {
if($NOOP) {
print "$RSYNC -essh --rsync-path=$REMOTE_RSYNC -a $_
$TARGET/$_n";
print "mv $_ $_.slavedn";
} else {
if(system("$RSYNC -essh --rsync-path=$REMOTE_RSYNC -a $_
$TARGET/$_") != 0) {
print STDERR "Error moving wal to target: $?n";
last;
}
if(!rename("$_", "$_.slaved")) {
print STDERR "Error renaming $_ to $_.slavedn";
last;
}
}
}
}


•Kind of Hacky

@wals2move = sort { $mtime{$a} <=> $mtime{$b} } @wals2move;
if($TARGET) {
foreach (@wals2move) {
if($NOOP) {
print "$RSYNC -essh --rsync-path=$REMOTE_RSYNC -a $_
$TARGET/$_n";
print "mv $_ $_.slavedn";
} else {
if(system("$RSYNC -essh --rsync-path=$REMOTE_RSYNC -a $_
$TARGET/$_") != 0) {
print STDERR "Error moving wal to target: $?n";
last;
}
if(!rename("$_", "$_.slaved")) {
print STDERR "Error renaming $_ to $_.slavedn";
last;
}
}
}
}


Customizations


Customizations

•Multiple Destinations


Customizations


•WAL File Delay


Customizations


•WAL File Delay

•Archive Storage (gzip & friends)


Customizations


•WAL File Delay

•Archive Storage (gzip & friends)

•Better Backups


ENTER: OmniPITR


ENTER: OmniPITR

•Consolidate various scripts


ENTER: OmniPITR


•Deploy across ‘Nixes


ENTER: OmniPITR



•Reusable


ENTER: OmniPITR



•Reusable

•Complete Solution


OmniPITR-Archive


OmniPITR-Archive

•Lots of options


OmniPITR-Archive

•Lots of options

•compression


OmniPITR-Archive

•Lots of options

•compression

•custom log ﬁle


OmniPITR-Archive

•Lots of options

•compression

•custom log ﬁle

•special paths


OmniPITR-Archive

•Lots of options

•compression

•custom log ﬁle

•special paths

•multiple destinations


OmniPITR-Archive Example

archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'



archive_command =
-t /var/tmp/
"%p"'

Location of OmniPITR



archive_command =
-t /var/tmp/
"%p"'

Custom Log File Name/Location



archive_command =
-t /var/tmp/
"%p"'

state-directory to handle errors when sending wal to
multiple destinations



archive_command =
-t /var/tmp/
"%p"'

Where To Create Temp Files



archive_command =
-t /var/tmp/
-dr gzip=dbx:/mnt/db/prod/walstorage/
"%p"'

send to remote server (dbx) compressed (gzip)



archive_command =
-t /var/tmp/
"%p"'

send to remote server (db4) uncompressed



archive_command =
-t /var/tmp/
"%p"'

Where to store xlogs when building a backup



archive_command =
-t /var/tmp/
"%p"'

The xlog ﬁle :-)


OmniPITR-Restore


OmniPITR-Restore

•Lots of options (though not as many)


OmniPITR-Restore


•compressed ﬁles?


OmniPITR-Restore



•custom log ﬁle


OmniPITR-Restore



•custom log ﬁle

•special paths


OmniPITR-Restore Example

restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'

location of omnipitr-restore program



restore_command = '
-v
-r
%f %p'

custom log ﬁle format



restore_command = '
-v
-r
%f %p'

Source of WAL Files to Use For Restore



restore_command = '
-v
-r
%f %p'

pause xlog removal if this ﬁle exists
(foreshadow: used for making backups)



restore_command = '
-v
-r
%f %p'

verbose



restore_command = '
-v
-r
%f %p'

Remove Xlogs Files Which Are No Longer Needed



restore_command = '
-v
-r
%f %p'

standard macros for “basename of segment” and
“destination of recovery”


OmniPITR-Master-Backup



•Goal



•Goal

•simple “tarball” for backup



•Goal


•one for $PGDATA and xlogs



•Goal


•one for $PGDATA and xlogs

•simple to blow open and use


OmniPITR-Master-Backup Example

/opt/OMNIpitr/bin/omnipitr-backup-master
-v
-x /var/lib/pgsql/omnipitr/backup.xlogs
-dr gzip=db4:/mnt/db/prod/backups/
-t /var/tmp/
-pp /usr/pgsql-9.0/bin/psql
-D /pg_data/90data

location of OmniPITR-backup-master



-v
-t /var/tmp/
-D /pg_data/90data

verbose



-v
-t /var/tmp/
-D /pg_data/90data

custom log ﬁle



-v
-t /var/tmp/
-D /pg_data/90data

directory to store xlogs during backup creation



-v
-t /var/tmp/
-D /pg_data/90data

copy backup ﬁle to remote server, compressed



-v
-t /var/tmp/
-D /pg_data/90data

location for temporary ﬁles whilst working



-v
-t /var/tmp/
-D /pg_data/90data

path to psql



-v
-t /var/tmp/
-D /pg_data/90data

$PGDATA


Programmers Say:

“I love it when people use
my software to do cool
things that I didn’t expect”


Treat’s Maxim?

“Try not to let the way
software was designed get
in the way of how you
want to use it”


OmniPITR-Slave-Backup



•On MySQL, you could dump on slave
for a long time



for a long time

•On ZFS, we use snapshots to make
slaves on the backups



for a long time


•While “we” prefer to use Postgres,
some of us prefer to use Linux



for a long time


•While “we” prefer to use Postgres,
some of us prefer to use Linux

•But, you still don’t want to pay the
price for backups on master


OmniPITR-Slave-Backup Example

/opt/OMNIpitr/bin/omnipitr-backup-slave
-v
-l "/var/log/omnipitr/omnipitr-slave-backup-^Y-^m-^d.log"
-s /mnt/db/prod/db4-walarchive
-p /var/lib/pgsql/omnipitr/pause.removal
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
-dl gzip=/mnt/db/prod/backups/
-pp /usr/pgsql-9.0/bin/pg_controldata

Location of omni-pitr-slave command



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

verbose



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

custom log location / format



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

source of wal ﬁles



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
ﬁle to pause removal of xlogs no longer needed for
restore but still needed by backup



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

$PGDATA



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

location for temp ﬁles whilst working



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

location of our “gzip”



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

local storage, compressed



-v
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz

path to pg_controldata


Production Use?


Production Use?

•Versions 8.2, 8.3, 8.4, 9.0


Production Use?

•Versions 8.2, 8.3, 8.4, 9.0

•Linux, Solaris


Production Use?

•Versions 8.2, 8.3, 8.4, 9.0

•Linux, Solaris

•TB+ Databases, with multiple
tablespaces


Production Use?

•Versions 8.2, 8.3, 8.4, 9.0

•Linux, Solaris

tablespaces

•Thousands of txn/sec


Production Use?

•Versions 8.2, 8.3, 8.4, 9.0

•Linux, Solaris

tablespaces

•Thousands of txn/sec

•Cross Datacenter


Where’s The Code?

https://labs.omniti.com/labs/pgtreats

svn co https://labs.omniti.com/pgtreats/trunk/omnipitr/


Where’s The Code?

•Currently in “PGTreats” Repo




Where’s The Code?


•Available Via SVN Pull




Where’s The Code?


•Available Via SVN Pull

•“BSD” Licensed




TODO


TODO

•Better Monitoring


TODO


•Parallel Multi-Destination


TODO


•Parallel Multi-Destination

•Multiple-Source Restores


BUGS

Failover of a streaming replication
based slave created from a slave
based backup ﬁle against a stream
replication based slave doesn’t
work


BUGS

work, sometimes,


BUGS

work, sometimes, when under load


BUGS

work, sometimes, when under load

We think it’s a ﬂaw in Postgres,
but hard to say for sure


THE END

Thanks!
pgeast, pgus

Slides
http://www.xzilla.net/


Advanced WAL File Management With OmniPITR

More Related Content

Viewers also liked (20)

Similar to Advanced WAL File Management With OmniPITR (20)

More from Robert Treat (14)

Recently uploaded (20)

Advanced WAL File Management With OmniPITR