PostgreSQL: meet your queue

PostgreSQL:
meet your Queue

/ Presentation / Theo Schlossnagle

1

PostgreSQL is Awesome

• Fast.

• Extensible.

• Tablespaces.

• Robust data types.

• Partitioning (albeit fake).

• Partial and functional indexes.

• Extremely supportive community.

• Extremely compliant with database standards.

2

PostgreSQL is not the “world”

• Inevitably, we must interact with the rest of the world.

• “non-SQL” components:

• nosql systems

• caching systems

• search systems (solr/lucene)

• management processes

3

Appropiare Typicalis

• Enforce in the application:

• the application code that updates the price or description of a
product in the products table;

• the application submits the updates to the search index system.

• The flaw:

• psql# UPDATE products
SET description =
REPLACE(description, ‘behaviour’, ‘behavior’);

• Administrative fixes like that require out-of-band dependency
handling.

4

A Solution

• Ideally, the database would notify all of these systems.

• The most common case I see: memcached.

• app: pull from memcached user::jesus@omniti.com
if not found:
select * from users where email=‘jesus@omniti.com‘
put row in memcached at user::jesus@omniti.com

• app: update users set mood=‘happy‘
where email=‘jesus@omniti.com‘
(a) purge memcached record
(b) get full row and replace in memcached

• hence: pgmemcache

• Problem:

• need a Postgres module for each remote component

5

Enter Queueing

• Queueing?

• A generic message bus that allows PostgreSQL to communicate
with other components in the architecture.

• Enter AMQP: “Advanced Message Queueing Protocol”

• Why not STOMP?

• Why not Starling?

• AMQP has been around the block, and the specification is quite
complete.

• Almost every “real” message broker implementation supports
AMQP

6

Setups: Installing

• svn export
https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp

• cd pg_amqp

• make USE_PGXS=1

• make install

• add to postgresql.conf:
shared_preload_libraries = 'pg_amqp.so'

• (re)start postgres

• load the pg_amqp.sql file into your database.

7

Setup: configuring your broker

INSERT INTO amqp.broker (host,port,vhost,username,password)
VALUES (‘localhost’,5672,NULL,‘guest’,‘guest’)
RETURNING broker_id

8

Setup: declaring an exchange

• This can often be done outside of the AMQP client

• using an AMQP management process
(that is just, in fact, an AMQP client)

• Often, another component has already created the exchange.

• If you really need to do it within PostgreSQL:

SELECT amqp.declare_exchange(broker_id,
exchange_name, exchange_type,
passive, durable, auto_delete)

9

Usage: sending messages

• How do I connect?

• It’s implicit... you don’t need to.

• How do I disconnect?

• Broker connections are cached and live across transactions.

• If you want to explicitly disconnect:

SELECT amqp.disconnect(broker_id);

10

Usage: sending messages for real

• Sending messages as a part of my transaction:

SELECT amqp.publish(broker_id, exchange, routing_key, message);

• This will publish the “message” over the specified “exchange” using
the specified “routing_key,” but only on transaction commit within
Postgres.

• Sending messages NOW:

SELECT amqp.autonomous_publish(broker_id, exchange,
routing_key, message);

• Publish the same message, but do it immediately.

11

A dark side: unsafe? WTF?

• Currently, pg_amqp uses the AMQP 0.8 specification.

• The AMQP 1.0 specification introduces formal 2PC.

• It is possible in the current implementation to have AMQP transaction
fail when we commit it (in postgres’s commit txn hook) in a fashion
that we cannot act on. Ouch.

• This only happens when the AMQP broker crashes between the last
in-transaction publish call and the COMMIT on the database side.

• Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC.

12

Thank you for listening.
https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/

/ Presentation

13

PostgreSQL: meet your queue

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to PostgreSQL: meet your queue (20)

More from Theo Schlossnagle (20)

Recently uploaded (20)

PostgreSQL: meet your queue