Programming with Python
    and PostgreSQL
            Peter Eisentraut
          peter@eisentraut.org

            F-Secure Corporation



    PostgreSQL Conference East 2011



                                      CC-BY
Partitioning



   • Part I: Client programming (60 min)
   • Part II: PL/Python (30 min)
Why Python?
Why Python?
 Pros:
   • widely used
   • easy
   • strong typing
   • scripting, interactive use
   • good PostgreSQL support
   • client and server (PL) interfaces
   • open source, community-based
Why Python?
 Pros:
   • widely used
   • easy
   • strong typing
   • scripting, interactive use
   • good PostgreSQL support
   • client and server (PL) interfaces
   • open source, community-based
 Pros:
   • no static syntax checks, must rely on test coverage
   • Python community has varying interest in RDBMS
Part I

Client Programming
Example
 import psycopg2

 dbconn = psycopg2.connect('dbname=dellstore2')
 cursor = dbconn.cursor()
 cursor.execute("""
 SELECT firstname, lastname
 FROM customers
 ORDER BY 1, 2
 LIMIT 10
 """)
 for row in cursor.fetchall():
     print "Name: %s %s" % (row[0], row[1])
 cursor.close()
 db.close()
Drivers

  Name             License   Platforms     Py Versions
  Psycopg          LGPL      Unix, Win     2.4–3.2
  PyGreSQL         BSD       Unix, Win     2.3–2.6
  ocpgdb           BSD       Unix          2.3–2.6
  py-postgresql    BSD       pure Python   3.0+
  bpgsql (alpha)   LGPL      pure Python   2.3–2.6
  pg8000           BSD       pure Python   2.5–3.0+
Drivers

   Name             License   Platforms     Py Versions
   Psycopg          LGPL      Unix, Win     2.4–3.2
   PyGreSQL         BSD       Unix, Win     2.3–2.6
   ocpgdb           BSD       Unix          2.3–2.6
   py-postgresql    BSD       pure Python   3.0+
   bpgsql (alpha)   LGPL      pure Python   2.3–2.6
   pg8000           BSD       pure Python   2.5–3.0+
  More details
    • http://wiki.postgresql.org/wiki/Python
    • http://wiki.python.org/moin/PostgreSQL
DB-API 2.0


   • the standard Python database API
   • all mentioned drivers support it
   • defined in PEP 249
   • discussions: db-sig@python.org
   • very elementary (from a PostgreSQL perspective)
   • outdated relative to Python language development
   • lots of extensions and incompatibilities possible
Higher-Level Interfaces



   • Zope
   • SQLAlchemy
   • Django
Psycopg Facts

   • Main authors: Federico Di Gregorio, Daniele Varrazzo
   • License: LGPLv3+
   • Web site: http://initd.org/psycopg/
        • Documentation: http://initd.org/psycopg/docs/
        • Git, Gitweb
   • Mailing list: psycopg@postgresql.org
   • Twitter: @psycopg
   • Latest version: 2.4 (February 27, 2011)
Using the Driver



  import psycopg2

  dbconn = psycopg2.connect(...)
  ...
Driver Independence?



 import psycopg2

 dbconn = psycopg2.connect(...)   # hardcodes driver name
Driver Independence?



 import psycopg2 as dbdriver

 dbconn = dbdriver.connect(...)
Driver Independence?


 dbtype = 'psycopg2'   # e.g. from config file
 dbdriver = __import__(dbtype,
                       globals(), locals(),
                       [], -1)

 dbconn = dbdriver.connect(...)
Connecting
 # libpq-like connection string
 dbconn = psycopg2.connect('dbname=dellstore2
     host=localhost port=5432')

 # same
 dbconn = psycopg2.connect(dsn='dbname=dellstore2
     host=localhost port=5432')

 # keyword arguments
 # (not all possible libpq options supported)
 dbconn = psycopg2.connect(database='dellstore2',
                           host='localhost',
                           port='5432')

 DB-API 2.0 says: arguments database dependent
“Cursors”



  cursor = dbconn.cursor()

    • not a real database cursor, only an API abstraction
    • think “statement handle”
Server-Side Cursors



  cursor = dbconn.cursor(name='mycursor')

    • a real database cursor
    • use for large result sets
Executing
 # queries
 cursor.execute("""
 SELECT firstname, lastname
 FROM customers
 ORDER BY 1, 2
 LIMIT 10
 """)

 # updates
 cursor.execute("UPDATE customers SET password = NULL")
 print "%d rows updated" % cursor.rowcount

 # or anything else
 cursor.execute("ANALYZE customers")
Fetching Query Results

  cursor.execute("SELECT firstname, lastname FROM ...")
  cursor.fetchall()

  [('AABBKO',   'DUTOFRPLOK'),
   ('AABTSI',   'ZFCKMPRVVJ'),
   ('AACOHS',   'EECCQPVTIW'),
   ('AACVVO',   'CLSXSGZYKS'),
   ('AADVMN',   'MEMQEWYFYE'),
   ('AADXQD',   'GLEKVVLZFV'),
   ('AAEBUG',   'YUOIINRJGE')]
Fetching Query Results



  cursor.execute("SELECT firstname, lastname FROM ...")
  for row in cursor.fetchall():
      print "Name: %s %s" % (row[0], row[1])
Fetching Query Results



  cursor.execute("SELECT firstname, lastname FROM ...")
  for row in cursor.fetchall():
      print "Name: %s %s" % (row[0], row[1])

  Note: field access only by number
Fetching Query Results



  cursor.execute("SELECT firstname, lastname FROM ...")
  row = cursor.fetchone()
  if row is not None:
      print "Name: %s %s" % (row[0], row[1])
Fetching Query Results



  cursor.execute("SELECT firstname, lastname FROM ...")
  for row in cursor:
      print "Name: %s %s" % (row[0], row[1])
Fetching Query Results in Batches


  cursor = dbconn.cursor(name='mycursor')
  cursor.arraysize = 500   # default: 1
  cursor.execute("SELECT firstname, lastname FROM ...")
  while True:
      batch = cursor.fetchmany()
      break if not batch
      for row in batch:
          print "Name: %s %s" % (row[0], row[1])
Fetching Query Results in Batches



  cursor = dbconn.cursor(name='mycursor')
  cursor.execute("SELECT firstname, lastname FROM ...")
  cursor.itersize = 2000   # default
  for row in cursor:
      print "Name: %s %s" % (row[0], row[1])
Getting Query Metadata

 cursor.execute("SELECT DISTINCT state, zip FROM
     customers")
 print cursor.description[0].name
 print cursor.description[0].type_code
 print cursor.description[1].name
 print cursor.description[1].type_code

 state
 1043    # == psycopg2.STRING
 zip
 23      # == psycopg2.NUMBER
Passing Parameters



 cursor.execute("""
 UPDATE customers
     SET password = %s
     WHERE customerid = %s
 """, ["sekret", 37])
Passing Parameters


 Not to be confused with (totally evil):
 cursor.execute("""
 UPDATE customers
     SET password = '%s'
     WHERE customerid = %d
 """ % ["sekret", 37])
Passing Parameters

 cursor.execute("INSERT INTO foo VALUES (%s)",
                "bar")    # WRONG

 cursor.execute("INSERT INTO foo VALUES (%s)",
                ("bar")) # WRONG

 cursor.execute("INSERT INTO foo VALUES (%s)",
                ("bar",)) # correct

 cursor.execute("INSERT INTO foo VALUES (%s)",
                ["bar"]) # correct

 (from Psycopg documentation)
Passing Parameters



 cursor.execute("""
 UPDATE customers
     SET password = %(pw)s
     WHERE customerid = %(id)s
 """, {'id': 37, 'pw': "sekret"})
Passing Many Parameter Sets


 cursor.executemany("""
 UPDATE customers
     SET password = %s
     WHERE customerid = %s
 """, [["ahTh4oip", 100],
       ["Rexahho7", 101],
       ["Ee1aetui", 102]])
Calling Procedures



  cursor.callproc('pg_start_backup', 'label')
Data Types


 from decimal import Decimal
 from psycopg2 import Date

 cursor.execute("""
 INSERT INTO orders (orderdate, customerid,
                     netamount, tax, totalamount)
 VALUES (%s, %s, %s, %s, %s)""",
 [Date(2011, 03, 23), 12345,
  Decimal("899.95"), 8.875, Decimal("979.82")])
Mogrify
  from decimal import Decimal
  from psycopg2 import Date

  cursor.mogrify("""
  INSERT INTO orders (orderdate, customerid,
                      netamount, tax, totalamount)
  VALUES (%s, %s, %s, %s, %s)""",
  [Date(2011, 03, 23), 12345,
   Decimal("899.95"), 8.875, Decimal("979.82")])

  Result:
  "nINSERT INTO orders (orderdate, customerid,n
      netamount, tax, totalamount)nVALUES
      ('2011-03-23'::date, 12345, 899.95, 8.875, 979.82)"
Data Types


 cursor.execute("""
 SELECT * FROM orders WHERE customerid = 12345
 """)

 Result:
 (12002, datetime.date(2011, 3, 23), 12345,
     Decimal('899.95'), Decimal('8.88'),
     Decimal('979.82'))
Nulls

  Input:
  cursor.mogrify("SELECT %s", [None])

  'SELECT NULL'

  Output:
  cursor.execute("SELECT NULL")
  cursor.fetchone()

  (None,)
Booleans



 cursor.mogrify("SELECT %s, %s", [True, False])

 'SELECT true, false'
Binary Data
  Standard way:
  from psycopg2 import Binary
  cursor.mogrify("SELECT %s", [Binary("foo")])

  "SELECT E'x666f6f'::bytea"
Binary Data
  Standard way:
  from psycopg2 import Binary
  cursor.mogrify("SELECT %s", [Binary("foo")])

  "SELECT E'x666f6f'::bytea"

  Other ways:
  cursor.mogrify("SELECT %s", [buffer("foo")])

  "SELECT E'x666f6f'::bytea"

  cursor.mogrify("SELECT %s",
                 [bytearray.fromhex(u"deadbeef")])

  "SELECT E'xdeadbeef'::bytea"

  There are more. Check the documentation. Check the versions.
Date/Time

 Standard ways:
 from psycopg2 import Date, Time, Timestamp

 cursor.mogrify("SELECT %s, %s, %s",
                [Date(2011, 3, 23),
                 Time(9, 0, 0),
                 Timestamp(2011, 3, 23, 9, 0, 0)])

 "SELECT '2011-03-23'::date, '09:00:00'::time,
     '2011-03-23T09:00:00'::timestamp"
Date/Time
 Other ways:
 import datetime

 cursor.mogrify("SELECT %s, %s, %s, %s",
                [datetime.date(2011, 3, 23),
                 datetime.time(9, 0, 0),
                 datetime.datetime(2011, 3, 23, 9, 0),
                 datetime.timedelta(minutes=90)])

 "SELECT '2011-03-23'::date, '09:00:00'::time,
     '2011-03-23T09:00:00'::timestamp, '0 days
     5400.000000 seconds'::interval"

 mx.DateTime   also supported
Arrays


  foo = [1, 2, 3]
  bar = [datetime.time(9, 0), datetime.time(10, 30)]

  cursor.mogrify("SELECT %s, %s",
                 [foo, bar])

  "SELECT ARRAY[1, 2, 3], ARRAY['09:00:00'::time,
      '10:30:00'::time]"
Tuples


 foo = (1, 2, 3)

 cursor.mogrify("SELECT * FROM customers WHERE
     customerid IN %s",
                [foo])

 'SELECT * FROM customers WHERE customerid IN (1, 2, 3)'
Hstore

 import psycopg2.extras

 psycopg2.extras.register_hstore(cursor)

 x = {'a': 'foo', 'b': 'bar'}

 cursor.mogrify("SELECT %s",
                [x])

 "SELECT hstore(ARRAY[E'a', E'b'], ARRAY[E'foo',
     E'bar'])"
Unicode Support


 Cause all result strings to be returned as Unicode strings:
 psycopg2.extensions.register_type(psycopg2.extensions.
     UNICODE)
 psycopg2.extensions.register_type(psycopg2.extensions.
     UNICODEARRAY)
Transaction Control


  Transaction blocks are used by default. Must use
  dbconn.commit()

  or
  dbconn.rollback()
Transaction Control: Autocommit


  import psycopg2.extensions

  dbconn.set_isolation_level(psycopg2.extensions.
      ISOLATION_LEVEL_AUTOCOMMIT)

  cursor = dbconn.cursor()
  cursor.execute("VACUUM")
Transaction Control: Isolation Mode


  import psycopg2.extensions

  dbconn.set_isolation_level(psycopg2.extensions.
      ISOLATION_LEVEL_SERIALIZABLE) # or other level

  cursor = dbconn.cursor()
  cursor.execute(...)
  ...
  dbconn.commit()
Exception Handling

  StandardError
  |__ Warning
  |__ Error
      |__ InterfaceError
      |__ DatabaseError
          |__ DataError
          |__ OperationalError
          |   |__ psycopg2.extensions.QueryCanceledError
          |   |__ psycopg2.extensions.TransactionRollbackError
          |__ IntegrityError
          |__ InternalError
          |__ ProgrammingError
          |__ NotSupportedError
Error Messages



 try:
     cursor.execute("boom")
 except Exception, e:
     print e.pgerror
Error Codes

 import psycopg2.errorcodes

 while True:
     try:
         cursor.execute("UPDATE something ...")
         cursor.execute("UPDATE otherthing ...")
         break
     except Exception, e:
         if e.pgcode == 
                 psycopg2.errorcodes.SERIALIZATION_FAILURE:
             continue
         else:
             raise
Connection and Cursor Factories

  Want: accessing result columns by name
  Recall:
  dbconn = psycopg2.connect(dsn='...')
  cursor = dbconn.cursor()
  cursor.execute("""
  SELECT firstname, lastname
  FROM customers
  ORDER BY 1, 2
  LIMIT 10
  """)
  for row in cursor.fetchall():
      print "Name: %s %s" % (row[0], row[1])   # stupid :(
Connection and Cursor Factories
  Solution 1: Using DictConnection:
  import psycopg2.extras

  dbconn = psycopg2.connect(dsn='...',
      connection_factory=psycopg2.extras.DictConnection)
  cursor = dbconn.cursor()
  cursor.execute("""
  SELECT firstname, lastname
  FROM customers
  ORDER BY 1, 2
  LIMIT 10
  """)
  for row in cursor.fetchall():
      print "Name: %s %s" % (row['firstname'], # or row[0]
                             row['lastname']) # or row[1]
Connection and Cursor Factories
  Solution 2: Using RealDictConnection:
  import psycopg2.extras

  dbconn = psycopg2.connect(dsn='...',
      connection_factory=psycopg2.extras.RealDictConnection)
  cursor = dbconn.cursor()
  cursor.execute("""
  SELECT firstname, lastname
  FROM customers
  ORDER BY 1, 2
  LIMIT 10
  """)
  for row in cursor.fetchall():
      print "Name: %s %s" % (row['firstname'],
                             row['lastname'])
Connection and Cursor Factories
  Solution 3: Using NamedTupleConnection:
  import psycopg2.extras

  dbconn = psycopg2.connect(dsn='...',
      connection_factory=psycopg2.extras.NamedTupleConnection)
  cursor = dbconn.cursor()
  cursor.execute("""
  SELECT firstname, lastname
  FROM customers
  ORDER BY 1, 2
  LIMIT 10
  """)
  for row in cursor.fetchall():
      print "Name: %s %s" % (row.firstname,    # or row[0]
                             row.lastname)     # or row[1]
Connection and Cursor Factories
  Alternative: Using
  DictCursor/RealDictCursor/NamedTupleCursor:

  import psycopg2.extras

  dbconn = psycopg2.connect(dsn='...')
  cursor = dbconn.cursor(cursor_factory=psycopg2.extras.
      DictCursor/RealDictCursor/NameTupleCursor)
  cursor.execute("""
  SELECT firstname, lastname
  FROM customers
  ORDER BY 1, 2
  LIMIT 10
  """)
  for row in cursor.fetchall():
      print "Name: %s %s" % (row['firstname'],
                             row['lastname'])
      # (resp. row.firstname, row.lastname)
Supporting New Data Types



 Only a finite list of types is supported by default: Date, Binary,
 etc.
   • map new PostgreSQL data types into Python
   • map new Python data types into PostgreSQL
Mapping New PostgreSQL Types Into
Python
 import psycopg2
 import psycopg2.extensions

 def cast_oidvector(value, _cursor):
     """Convert oidvector to Python array"""
     if value is None:
         return None
     return map(int, value.split(' '))


 OIDVECTOR = psycopg2.extensions.new_type((30,),
     'OIDVECTOR', cast_oidvector)
 psycopg2.extensions.register_type(OIDVECTOR)
Mapping New Python Types into
PostgreSQL
 from psycopg2.extensions import adapt,
     register_adapter, AsIs

 class Point(object):
     def __init__(self, x, y):
         self.x = x
         self.y = y

 def adapt_point(point):
     return AsIs("'(%s, %s)'" % (adapt(point.x),
         adapt(point.y)))

 register_adapter(Point, adapt_point)

 cur.execute("INSERT INTO atable (apoint) VALUES (%s)",
             (Point(1.23, 4.56),))

 (from Psycopg documentation)
Connection Pooling With Psycopg
 from psycopg2.pool import SimpleConnectionPool

 pool = SimpleConnectionPool(1, 20, dsn='...')
 dbconn = pool.getconn()
 ...
 pool.putconn(dbconn)
 pool.closeall()
Connection Pooling With Psycopg
 for non-threaded applications:
 from psycopg2.pool import SimpleConnectionPool

 pool = SimpleConnectionPool(1, 20, dsn='...')
 dbconn = pool.getconn()
 ...
 pool.putconn(dbconn)
 pool.closeall()

 for non-threaded applications:
 from psycopg2.pool import ThreadedConnectionPool

 pool = ThreadedConnectionPool(1, 20, dsn='...')
 dbconn = pool.getconn()
 cursor = dbconn.cursor()
 ...
 pool.putconn(dbconn)
 pool.closeall()
Connection Pooling With DBUtils


  import psycopg2
  from DBUtils.PersistentDB import PersistentDB

  dbconn = PersistentDB(psycopg2, dsn='...')
  cursor = dbconn.cursor()
  ...

  see http://pypi.python.org/pypi/DBUtils/
The Other Stuff

   • thread safety: can share connections, but not cursors
   • COPY support: cursor.copy_from(), cursor.copy_to()
   • large object support: connection.lobject()
   • 2PC: connection.xid(), connection.tpc_begin(), . . .
   • query cancel: dbconn.cancel()
   • notices: dbconn.notices
   • notifications: dbconn.notifies
   • asynchronous communication
   • coroutine support
   • logging cursor
Part II

PL/Python
Setup


   • included with PostgreSQL
        • configure --with-python
        • apt-get/yum install postgresql-plpython
   • CREATE LANGUAGE plpythonu;
   • Python 3: CREATE LANGUAGE plpython3u;
   • “untrusted”, superuser only
Basic Examples
 CREATE FUNCTION add(a int, b int) RETURNS int
 LANGUAGE plpythonu
 AS $$
 return a + b
 $$;

 CREATE FUNCTION longest(a text, b text) RETURNS text
 LANGUAGE plpythonu
 AS $$
 if len(a) > len(b):
     return a
 elif len(b) > len(a):
     return b
 else:
     return None
 $$;
Using Modules


 CREATE FUNCTION json_to_array(j text) RETURNS text[]
 LANGUAGE plpythonu
 AS $$
 import json

 return json.loads(j)
 $$;
Database Calls


 CREATE FUNCTION clear_passwords() RETURNS int
 LANGUAGE plpythonu
 AS $$
 rv = plpy.execute("UPDATE customers SET password =
     NULL")
 return rv.nrows
 $$;
Database Calls With Parameters


 CREATE FUNCTION set_password(username text, password
     text) RETURNS boolean
 LANGUAGE plpythonu
 AS $$
 plan = plpy.prepare("UPDATE customers SET password = $1
     WHERE username= $2", ['text', 'text'])
 rv = plpy.execute(plan, [username, password])
 return rv.nrows == 1
 $$;
Avoiding Prepared Statements

 CREATE FUNCTION set_password(username text, password
     text) RETURNS boolean
 LANGUAGE plpythonu
 AS $$
 rv = plpy.execute("UPDATE customers SET password = %s
     WHERE username= %s" %
     (plpy.quote_nullable(username),
     plpy.quote_literal(password)))
 return rv.nrows == 1
 $$;

 (available in 9.1-to-be)
Caching Plans

 CREATE FUNCTION set_password2(username text, password
     text) RETURNS boolean
 LANGUAGE plpythonu
 AS $$
 if 'myplan' in SD:
     plan = SD['myplan']
 else:
     plan = plpy.prepare("UPDATE customers SET password
         = $1 WHERE username= $2", ['text', 'text'])
     SD['myplan'] = plan
 rv = plpy.execute(plan, [username, password])
 return rv.nrows == 1
 $$;
Processing Query Results

 CREATE FUNCTION get_customer_name(username text)
     RETURNS boolean
 LANGUAGE plpythonu
 AS $$
 plan = plpy.prepare("SELECT firstname || ' ' ||
     lastname AS ""name"" FROM customers WHERE username =
     $1", ['text'])
 rv = plpy.execute(plan, [username], 1)
 return rv[0]['name']
 $$;
Compare: PL/Python vs. DB-API

 PL/Python:
 plan = plpy.prepare("SELECT ...")
 for row in plpy.execute(plan, ...):
     plpy.info(row["fieldname"])

 DB-API:
 dbconn = psycopg2.connect(...)
 cursor = dbconn.cursor()
 cursor.execute("SELECT ...")
 for row in cursor.fetchall() do:
     print row[0]
Set-Returning and Table Functions


  CREATE FUNCTION get_customers(id int) RETURNS SETOF
      customers
  LANGUAGE plpythonu
  AS $$
  plan = plpy.prepare("SELECT * FROM customers WHERE
      customerid = $1", ['int'])
  rv = plpy.execute(plan, [id])
  return rv
  $$;
Triggers

  CREATE FUNCTION delete_notifier() RETURNS trigger
  LANGUAGE plpythonu
  AS $$
  if TD['event'] == 'DELETE':
      plpy.notice("one row deleted from table %s" %
          TD['table_name'])
  $$;

  CREATE TRIGGER customers_delete_notifier AFTER DELETE
      ON customers FOR EACH ROW EXECUTE PROCEDURE
      delete_notifier();
Exceptions


 CREATE FUNCTION test() RETURNS text
 LANGUAGE plpythonu
 AS $$
 try:
     rv = plpy.execute("SELECT ...")
 except plpy.SPIError, e:
     plpy.notice("something went wrong")

 The transaction is still aborted in < 9.1.
New in PostgreSQL 9.1

   • SPI calls wrapped in subtransactions
   • custom SPI exceptions: subclass per SQLSTATE,
    .sqlstate    attribute
   • plpy.subtransaction() context manager
   • support for OUT parameters
   • quoting functions
   • validator
   • lots of internal improvements
The End

More Related Content

PPTX
Psycopg2 - Connect to PostgreSQL using Python Script
PPTX
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
PPTX
Apache HBase™
PPT
NOSQL Database: Apache Cassandra
PDF
Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)
PDF
Get to know PostgreSQL!
PDF
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
PPTX
PostgreSQL- An Introduction
Psycopg2 - Connect to PostgreSQL using Python Script
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
Apache HBase™
NOSQL Database: Apache Cassandra
Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)
Get to know PostgreSQL!
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
PostgreSQL- An Introduction

What's hot (20)

PDF
[pgday.Seoul 2022] PostgreSQL with Google Cloud
PDF
Topic Modeling
PPTX
Introduction to Graph Databases
PPTX
Introduction to Redis
PDF
PySpark in practice slides
ODP
Introduction to PostgreSQL
PPTX
Map Reduce
PDF
Postgresql database administration volume 1
PPTX
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
PDF
PySpark Best Practices
PPTX
Real World Event Sourcing and CQRS
PPTX
Hive, Presto, and Spark on TPC-DS benchmark
ODP
OpenGurukul : Database : PostgreSQL
PPTX
The Basics of MongoDB
PDF
Introducing DataFrames in Spark for Large Scale Data Science
PDF
Apache Superset at Airbnb
PDF
Apache BookKeeper: A High Performance and Low Latency Storage Service
PDF
Introduction to PySpark
PDF
Amazon Redshift의 이해와 활용 (김용우) - AWS DB Day
PPTX
Airflow를 이용한 데이터 Workflow 관리
[pgday.Seoul 2022] PostgreSQL with Google Cloud
Topic Modeling
Introduction to Graph Databases
Introduction to Redis
PySpark in practice slides
Introduction to PostgreSQL
Map Reduce
Postgresql database administration volume 1
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
PySpark Best Practices
Real World Event Sourcing and CQRS
Hive, Presto, and Spark on TPC-DS benchmark
OpenGurukul : Database : PostgreSQL
The Basics of MongoDB
Introducing DataFrames in Spark for Large Scale Data Science
Apache Superset at Airbnb
Apache BookKeeper: A High Performance and Low Latency Storage Service
Introduction to PySpark
Amazon Redshift의 이해와 활용 (김용우) - AWS DB Day
Airflow를 이용한 데이터 Workflow 관리
Ad

Viewers also liked (20)

PDF
"PostgreSQL and Python" Lightning Talk @EuroPython2014
PDF
Socket Programming In Python
PDF
Programming with Python - Basic
PDF
Functional programming in Python
PDF
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
PDF
PPTX
Golang iran - tutorial go programming language - Preliminary
PDF
Golang #5: To Go or not to Go
PDF
Www Kitebird Com Articles Pydbapi Html Toc 1
ODP
Rethink db with Python
PPTX
Succumbing to the Python in Financial Markets
PDF
Massively Parallel Processing with Procedural Python (PyData London 2014)
ODP
Matando o Java e Mostrando o Python
PPTX
Relational Database Access with Python
PDF
Postgresql + Python = Power!
PDF
MySQL User Conference 2009: Python and MySQL
PDF
Scaling mysql with python (and Docker).
PDF
Oracle Deep Internal 1 (ver.2)
PDF
Oracle Deep Internal 3 (ver.2)
PPTX
Relational Database Access with Python ‘sans’ ORM
"PostgreSQL and Python" Lightning Talk @EuroPython2014
Socket Programming In Python
Programming with Python - Basic
Functional programming in Python
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
Golang iran - tutorial go programming language - Preliminary
Golang #5: To Go or not to Go
Www Kitebird Com Articles Pydbapi Html Toc 1
Rethink db with Python
Succumbing to the Python in Financial Markets
Massively Parallel Processing with Procedural Python (PyData London 2014)
Matando o Java e Mostrando o Python
Relational Database Access with Python
Postgresql + Python = Power!
MySQL User Conference 2009: Python and MySQL
Scaling mysql with python (and Docker).
Oracle Deep Internal 1 (ver.2)
Oracle Deep Internal 3 (ver.2)
Relational Database Access with Python ‘sans’ ORM
Ad

Similar to Programming with Python and PostgreSQL (20)

PDF
.gradle 파일 정독해보기
PDF
Pdxpugday2010 pg90
PDF
The Ring programming language version 1.2 book - Part 32 of 84
PDF
The Ring programming language version 1.6 book - Part 46 of 189
PDF
Emerging Languages: A Tour of the Horizon
PPTX
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
PDF
Go Web Development
PDF
Refactoring to Macros with Clojure
PDF
The Ring programming language version 1.7 book - Part 48 of 196
PDF
The Ring programming language version 1.5.2 book - Part 43 of 181
PDF
OrientDB - The 2nd generation of (multi-model) NoSQL
PDF
Graph Algorithms: Analytics for Understanding Data Relationships
PDF
Norikra: SQL Stream Processing In Ruby
PDF
The Ring programming language version 1.5 book - Part 8 of 31
PDF
2 BytesC++ course_2014_c3_ function basics&parameters and overloading
PDF
Benchy, python framework for performance benchmarking of Python Scripts
PDF
The Ring programming language version 1.4.1 book - Part 13 of 31
PDF
Managing Large-scale Networks with Trigger
PDF
The Ring programming language version 1.9 book - Part 53 of 210
PDF
GraphTour - Utilizing Powerful Extensions for Analytics & Operations
.gradle 파일 정독해보기
Pdxpugday2010 pg90
The Ring programming language version 1.2 book - Part 32 of 84
The Ring programming language version 1.6 book - Part 46 of 189
Emerging Languages: A Tour of the Horizon
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
Go Web Development
Refactoring to Macros with Clojure
The Ring programming language version 1.7 book - Part 48 of 196
The Ring programming language version 1.5.2 book - Part 43 of 181
OrientDB - The 2nd generation of (multi-model) NoSQL
Graph Algorithms: Analytics for Understanding Data Relationships
Norikra: SQL Stream Processing In Ruby
The Ring programming language version 1.5 book - Part 8 of 31
2 BytesC++ course_2014_c3_ function basics&parameters and overloading
Benchy, python framework for performance benchmarking of Python Scripts
The Ring programming language version 1.4.1 book - Part 13 of 31
Managing Large-scale Networks with Trigger
The Ring programming language version 1.9 book - Part 53 of 210
GraphTour - Utilizing Powerful Extensions for Analytics & Operations

More from Peter Eisentraut (20)

PDF
Getting Started with PL/Proxy
PDF
Linux distribution for the cloud
PDF
Most Wanted: Future PostgreSQL Features
ODP
Porting Applications From Oracle To PostgreSQL
PDF
Porting Oracle Applications to PostgreSQL
PDF
PostgreSQL and XML
PDF
XML Support: Specifications and Development
PDF
PostgreSQL: Die Freie Datenbankalternative
PDF
The Road to the XML Type: Current and Future Developments
PDF
Access ohne Access: Freie Datenbank-Frontends
PDF
PostgreSQL and PL/Java
PDF
Replication Solutions for PostgreSQL
PDF
PostgreSQL News
PDF
PostgreSQL News
PDF
Access ohne Access: Freie Datenbank-Frontends
PDF
Docbook: Textverarbeitung mit XML
PDF
Collateral Damage: Consequences of Spam and Virus Filtering for the E-Mail Sy...
PDF
Collateral Damage: Consequences of Spam and Virus Filtering for the E-Mail S...
PDF
Spaß mit PostgreSQL
PDF
The Common Debian Build System (CDBS)
Getting Started with PL/Proxy
Linux distribution for the cloud
Most Wanted: Future PostgreSQL Features
Porting Applications From Oracle To PostgreSQL
Porting Oracle Applications to PostgreSQL
PostgreSQL and XML
XML Support: Specifications and Development
PostgreSQL: Die Freie Datenbankalternative
The Road to the XML Type: Current and Future Developments
Access ohne Access: Freie Datenbank-Frontends
PostgreSQL and PL/Java
Replication Solutions for PostgreSQL
PostgreSQL News
PostgreSQL News
Access ohne Access: Freie Datenbank-Frontends
Docbook: Textverarbeitung mit XML
Collateral Damage: Consequences of Spam and Virus Filtering for the E-Mail Sy...
Collateral Damage: Consequences of Spam and Virus Filtering for the E-Mail S...
Spaß mit PostgreSQL
The Common Debian Build System (CDBS)

Recently uploaded (20)

PDF
STKI Israel Market Study 2025 version august
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
PDF
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
PPT
Geologic Time for studying geology for geologist
PPT
What is a Computer? Input Devices /output devices
PDF
Hybrid model detection and classification of lung cancer
PDF
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
PDF
WOOl fibre morphology and structure.pdf for textiles
PDF
DP Operators-handbook-extract for the Mautical Institute
PDF
A comparative study of natural language inference in Swahili using monolingua...
PPTX
Modernising the Digital Integration Hub
PDF
Getting started with AI Agents and Multi-Agent Systems
PDF
Zenith AI: Advanced Artificial Intelligence
PPTX
Benefits of Physical activity for teenagers.pptx
PDF
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
PDF
Assigned Numbers - 2025 - Bluetooth® Document
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
PDF
Enhancing emotion recognition model for a student engagement use case through...
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
PDF
Developing a website for English-speaking practice to English as a foreign la...
STKI Israel Market Study 2025 version august
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Geologic Time for studying geology for geologist
What is a Computer? Input Devices /output devices
Hybrid model detection and classification of lung cancer
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
WOOl fibre morphology and structure.pdf for textiles
DP Operators-handbook-extract for the Mautical Institute
A comparative study of natural language inference in Swahili using monolingua...
Modernising the Digital Integration Hub
Getting started with AI Agents and Multi-Agent Systems
Zenith AI: Advanced Artificial Intelligence
Benefits of Physical activity for teenagers.pptx
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
Assigned Numbers - 2025 - Bluetooth® Document
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
Enhancing emotion recognition model for a student engagement use case through...
Univ-Connecticut-ChatGPT-Presentaion.pdf
Developing a website for English-speaking practice to English as a foreign la...

Programming with Python and PostgreSQL

  • 1. Programming with Python and PostgreSQL Peter Eisentraut [email protected] F-Secure Corporation PostgreSQL Conference East 2011 CC-BY
  • 2. Partitioning • Part I: Client programming (60 min) • Part II: PL/Python (30 min)
  • 4. Why Python? Pros: • widely used • easy • strong typing • scripting, interactive use • good PostgreSQL support • client and server (PL) interfaces • open source, community-based
  • 5. Why Python? Pros: • widely used • easy • strong typing • scripting, interactive use • good PostgreSQL support • client and server (PL) interfaces • open source, community-based Pros: • no static syntax checks, must rely on test coverage • Python community has varying interest in RDBMS
  • 7. Example import psycopg2 dbconn = psycopg2.connect('dbname=dellstore2') cursor = dbconn.cursor() cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row[0], row[1]) cursor.close() db.close()
  • 8. Drivers Name License Platforms Py Versions Psycopg LGPL Unix, Win 2.4–3.2 PyGreSQL BSD Unix, Win 2.3–2.6 ocpgdb BSD Unix 2.3–2.6 py-postgresql BSD pure Python 3.0+ bpgsql (alpha) LGPL pure Python 2.3–2.6 pg8000 BSD pure Python 2.5–3.0+
  • 9. Drivers Name License Platforms Py Versions Psycopg LGPL Unix, Win 2.4–3.2 PyGreSQL BSD Unix, Win 2.3–2.6 ocpgdb BSD Unix 2.3–2.6 py-postgresql BSD pure Python 3.0+ bpgsql (alpha) LGPL pure Python 2.3–2.6 pg8000 BSD pure Python 2.5–3.0+ More details • http://wiki.postgresql.org/wiki/Python • http://wiki.python.org/moin/PostgreSQL
  • 10. DB-API 2.0 • the standard Python database API • all mentioned drivers support it • defined in PEP 249 • discussions: [email protected] • very elementary (from a PostgreSQL perspective) • outdated relative to Python language development • lots of extensions and incompatibilities possible
  • 11. Higher-Level Interfaces • Zope • SQLAlchemy • Django
  • 12. Psycopg Facts • Main authors: Federico Di Gregorio, Daniele Varrazzo • License: LGPLv3+ • Web site: http://initd.org/psycopg/ • Documentation: http://initd.org/psycopg/docs/ • Git, Gitweb • Mailing list: [email protected] • Twitter: @psycopg • Latest version: 2.4 (February 27, 2011)
  • 13. Using the Driver import psycopg2 dbconn = psycopg2.connect(...) ...
  • 14. Driver Independence? import psycopg2 dbconn = psycopg2.connect(...) # hardcodes driver name
  • 15. Driver Independence? import psycopg2 as dbdriver dbconn = dbdriver.connect(...)
  • 16. Driver Independence? dbtype = 'psycopg2' # e.g. from config file dbdriver = __import__(dbtype, globals(), locals(), [], -1) dbconn = dbdriver.connect(...)
  • 17. Connecting # libpq-like connection string dbconn = psycopg2.connect('dbname=dellstore2 host=localhost port=5432') # same dbconn = psycopg2.connect(dsn='dbname=dellstore2 host=localhost port=5432') # keyword arguments # (not all possible libpq options supported) dbconn = psycopg2.connect(database='dellstore2', host='localhost', port='5432') DB-API 2.0 says: arguments database dependent
  • 18. “Cursors” cursor = dbconn.cursor() • not a real database cursor, only an API abstraction • think “statement handle”
  • 19. Server-Side Cursors cursor = dbconn.cursor(name='mycursor') • a real database cursor • use for large result sets
  • 20. Executing # queries cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) # updates cursor.execute("UPDATE customers SET password = NULL") print "%d rows updated" % cursor.rowcount # or anything else cursor.execute("ANALYZE customers")
  • 21. Fetching Query Results cursor.execute("SELECT firstname, lastname FROM ...") cursor.fetchall() [('AABBKO', 'DUTOFRPLOK'), ('AABTSI', 'ZFCKMPRVVJ'), ('AACOHS', 'EECCQPVTIW'), ('AACVVO', 'CLSXSGZYKS'), ('AADVMN', 'MEMQEWYFYE'), ('AADXQD', 'GLEKVVLZFV'), ('AAEBUG', 'YUOIINRJGE')]
  • 22. Fetching Query Results cursor.execute("SELECT firstname, lastname FROM ...") for row in cursor.fetchall(): print "Name: %s %s" % (row[0], row[1])
  • 23. Fetching Query Results cursor.execute("SELECT firstname, lastname FROM ...") for row in cursor.fetchall(): print "Name: %s %s" % (row[0], row[1]) Note: field access only by number
  • 24. Fetching Query Results cursor.execute("SELECT firstname, lastname FROM ...") row = cursor.fetchone() if row is not None: print "Name: %s %s" % (row[0], row[1])
  • 25. Fetching Query Results cursor.execute("SELECT firstname, lastname FROM ...") for row in cursor: print "Name: %s %s" % (row[0], row[1])
  • 26. Fetching Query Results in Batches cursor = dbconn.cursor(name='mycursor') cursor.arraysize = 500 # default: 1 cursor.execute("SELECT firstname, lastname FROM ...") while True: batch = cursor.fetchmany() break if not batch for row in batch: print "Name: %s %s" % (row[0], row[1])
  • 27. Fetching Query Results in Batches cursor = dbconn.cursor(name='mycursor') cursor.execute("SELECT firstname, lastname FROM ...") cursor.itersize = 2000 # default for row in cursor: print "Name: %s %s" % (row[0], row[1])
  • 28. Getting Query Metadata cursor.execute("SELECT DISTINCT state, zip FROM customers") print cursor.description[0].name print cursor.description[0].type_code print cursor.description[1].name print cursor.description[1].type_code state 1043 # == psycopg2.STRING zip 23 # == psycopg2.NUMBER
  • 29. Passing Parameters cursor.execute(""" UPDATE customers SET password = %s WHERE customerid = %s """, ["sekret", 37])
  • 30. Passing Parameters Not to be confused with (totally evil): cursor.execute(""" UPDATE customers SET password = '%s' WHERE customerid = %d """ % ["sekret", 37])
  • 31. Passing Parameters cursor.execute("INSERT INTO foo VALUES (%s)", "bar") # WRONG cursor.execute("INSERT INTO foo VALUES (%s)", ("bar")) # WRONG cursor.execute("INSERT INTO foo VALUES (%s)", ("bar",)) # correct cursor.execute("INSERT INTO foo VALUES (%s)", ["bar"]) # correct (from Psycopg documentation)
  • 32. Passing Parameters cursor.execute(""" UPDATE customers SET password = %(pw)s WHERE customerid = %(id)s """, {'id': 37, 'pw': "sekret"})
  • 33. Passing Many Parameter Sets cursor.executemany(""" UPDATE customers SET password = %s WHERE customerid = %s """, [["ahTh4oip", 100], ["Rexahho7", 101], ["Ee1aetui", 102]])
  • 34. Calling Procedures cursor.callproc('pg_start_backup', 'label')
  • 35. Data Types from decimal import Decimal from psycopg2 import Date cursor.execute(""" INSERT INTO orders (orderdate, customerid, netamount, tax, totalamount) VALUES (%s, %s, %s, %s, %s)""", [Date(2011, 03, 23), 12345, Decimal("899.95"), 8.875, Decimal("979.82")])
  • 36. Mogrify from decimal import Decimal from psycopg2 import Date cursor.mogrify(""" INSERT INTO orders (orderdate, customerid, netamount, tax, totalamount) VALUES (%s, %s, %s, %s, %s)""", [Date(2011, 03, 23), 12345, Decimal("899.95"), 8.875, Decimal("979.82")]) Result: "nINSERT INTO orders (orderdate, customerid,n netamount, tax, totalamount)nVALUES ('2011-03-23'::date, 12345, 899.95, 8.875, 979.82)"
  • 37. Data Types cursor.execute(""" SELECT * FROM orders WHERE customerid = 12345 """) Result: (12002, datetime.date(2011, 3, 23), 12345, Decimal('899.95'), Decimal('8.88'), Decimal('979.82'))
  • 38. Nulls Input: cursor.mogrify("SELECT %s", [None]) 'SELECT NULL' Output: cursor.execute("SELECT NULL") cursor.fetchone() (None,)
  • 39. Booleans cursor.mogrify("SELECT %s, %s", [True, False]) 'SELECT true, false'
  • 40. Binary Data Standard way: from psycopg2 import Binary cursor.mogrify("SELECT %s", [Binary("foo")]) "SELECT E'x666f6f'::bytea"
  • 41. Binary Data Standard way: from psycopg2 import Binary cursor.mogrify("SELECT %s", [Binary("foo")]) "SELECT E'x666f6f'::bytea" Other ways: cursor.mogrify("SELECT %s", [buffer("foo")]) "SELECT E'x666f6f'::bytea" cursor.mogrify("SELECT %s", [bytearray.fromhex(u"deadbeef")]) "SELECT E'xdeadbeef'::bytea" There are more. Check the documentation. Check the versions.
  • 42. Date/Time Standard ways: from psycopg2 import Date, Time, Timestamp cursor.mogrify("SELECT %s, %s, %s", [Date(2011, 3, 23), Time(9, 0, 0), Timestamp(2011, 3, 23, 9, 0, 0)]) "SELECT '2011-03-23'::date, '09:00:00'::time, '2011-03-23T09:00:00'::timestamp"
  • 43. Date/Time Other ways: import datetime cursor.mogrify("SELECT %s, %s, %s, %s", [datetime.date(2011, 3, 23), datetime.time(9, 0, 0), datetime.datetime(2011, 3, 23, 9, 0), datetime.timedelta(minutes=90)]) "SELECT '2011-03-23'::date, '09:00:00'::time, '2011-03-23T09:00:00'::timestamp, '0 days 5400.000000 seconds'::interval" mx.DateTime also supported
  • 44. Arrays foo = [1, 2, 3] bar = [datetime.time(9, 0), datetime.time(10, 30)] cursor.mogrify("SELECT %s, %s", [foo, bar]) "SELECT ARRAY[1, 2, 3], ARRAY['09:00:00'::time, '10:30:00'::time]"
  • 45. Tuples foo = (1, 2, 3) cursor.mogrify("SELECT * FROM customers WHERE customerid IN %s", [foo]) 'SELECT * FROM customers WHERE customerid IN (1, 2, 3)'
  • 46. Hstore import psycopg2.extras psycopg2.extras.register_hstore(cursor) x = {'a': 'foo', 'b': 'bar'} cursor.mogrify("SELECT %s", [x]) "SELECT hstore(ARRAY[E'a', E'b'], ARRAY[E'foo', E'bar'])"
  • 47. Unicode Support Cause all result strings to be returned as Unicode strings: psycopg2.extensions.register_type(psycopg2.extensions. UNICODE) psycopg2.extensions.register_type(psycopg2.extensions. UNICODEARRAY)
  • 48. Transaction Control Transaction blocks are used by default. Must use dbconn.commit() or dbconn.rollback()
  • 49. Transaction Control: Autocommit import psycopg2.extensions dbconn.set_isolation_level(psycopg2.extensions. ISOLATION_LEVEL_AUTOCOMMIT) cursor = dbconn.cursor() cursor.execute("VACUUM")
  • 50. Transaction Control: Isolation Mode import psycopg2.extensions dbconn.set_isolation_level(psycopg2.extensions. ISOLATION_LEVEL_SERIALIZABLE) # or other level cursor = dbconn.cursor() cursor.execute(...) ... dbconn.commit()
  • 51. Exception Handling StandardError |__ Warning |__ Error |__ InterfaceError |__ DatabaseError |__ DataError |__ OperationalError | |__ psycopg2.extensions.QueryCanceledError | |__ psycopg2.extensions.TransactionRollbackError |__ IntegrityError |__ InternalError |__ ProgrammingError |__ NotSupportedError
  • 52. Error Messages try: cursor.execute("boom") except Exception, e: print e.pgerror
  • 53. Error Codes import psycopg2.errorcodes while True: try: cursor.execute("UPDATE something ...") cursor.execute("UPDATE otherthing ...") break except Exception, e: if e.pgcode == psycopg2.errorcodes.SERIALIZATION_FAILURE: continue else: raise
  • 54. Connection and Cursor Factories Want: accessing result columns by name Recall: dbconn = psycopg2.connect(dsn='...') cursor = dbconn.cursor() cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row[0], row[1]) # stupid :(
  • 55. Connection and Cursor Factories Solution 1: Using DictConnection: import psycopg2.extras dbconn = psycopg2.connect(dsn='...', connection_factory=psycopg2.extras.DictConnection) cursor = dbconn.cursor() cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row['firstname'], # or row[0] row['lastname']) # or row[1]
  • 56. Connection and Cursor Factories Solution 2: Using RealDictConnection: import psycopg2.extras dbconn = psycopg2.connect(dsn='...', connection_factory=psycopg2.extras.RealDictConnection) cursor = dbconn.cursor() cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row['firstname'], row['lastname'])
  • 57. Connection and Cursor Factories Solution 3: Using NamedTupleConnection: import psycopg2.extras dbconn = psycopg2.connect(dsn='...', connection_factory=psycopg2.extras.NamedTupleConnection) cursor = dbconn.cursor() cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row.firstname, # or row[0] row.lastname) # or row[1]
  • 58. Connection and Cursor Factories Alternative: Using DictCursor/RealDictCursor/NamedTupleCursor: import psycopg2.extras dbconn = psycopg2.connect(dsn='...') cursor = dbconn.cursor(cursor_factory=psycopg2.extras. DictCursor/RealDictCursor/NameTupleCursor) cursor.execute(""" SELECT firstname, lastname FROM customers ORDER BY 1, 2 LIMIT 10 """) for row in cursor.fetchall(): print "Name: %s %s" % (row['firstname'], row['lastname']) # (resp. row.firstname, row.lastname)
  • 59. Supporting New Data Types Only a finite list of types is supported by default: Date, Binary, etc. • map new PostgreSQL data types into Python • map new Python data types into PostgreSQL
  • 60. Mapping New PostgreSQL Types Into Python import psycopg2 import psycopg2.extensions def cast_oidvector(value, _cursor): """Convert oidvector to Python array""" if value is None: return None return map(int, value.split(' ')) OIDVECTOR = psycopg2.extensions.new_type((30,), 'OIDVECTOR', cast_oidvector) psycopg2.extensions.register_type(OIDVECTOR)
  • 61. Mapping New Python Types into PostgreSQL from psycopg2.extensions import adapt, register_adapter, AsIs class Point(object): def __init__(self, x, y): self.x = x self.y = y def adapt_point(point): return AsIs("'(%s, %s)'" % (adapt(point.x), adapt(point.y))) register_adapter(Point, adapt_point) cur.execute("INSERT INTO atable (apoint) VALUES (%s)", (Point(1.23, 4.56),)) (from Psycopg documentation)
  • 62. Connection Pooling With Psycopg from psycopg2.pool import SimpleConnectionPool pool = SimpleConnectionPool(1, 20, dsn='...') dbconn = pool.getconn() ... pool.putconn(dbconn) pool.closeall()
  • 63. Connection Pooling With Psycopg for non-threaded applications: from psycopg2.pool import SimpleConnectionPool pool = SimpleConnectionPool(1, 20, dsn='...') dbconn = pool.getconn() ... pool.putconn(dbconn) pool.closeall() for non-threaded applications: from psycopg2.pool import ThreadedConnectionPool pool = ThreadedConnectionPool(1, 20, dsn='...') dbconn = pool.getconn() cursor = dbconn.cursor() ... pool.putconn(dbconn) pool.closeall()
  • 64. Connection Pooling With DBUtils import psycopg2 from DBUtils.PersistentDB import PersistentDB dbconn = PersistentDB(psycopg2, dsn='...') cursor = dbconn.cursor() ... see http://pypi.python.org/pypi/DBUtils/
  • 65. The Other Stuff • thread safety: can share connections, but not cursors • COPY support: cursor.copy_from(), cursor.copy_to() • large object support: connection.lobject() • 2PC: connection.xid(), connection.tpc_begin(), . . . • query cancel: dbconn.cancel() • notices: dbconn.notices • notifications: dbconn.notifies • asynchronous communication • coroutine support • logging cursor
  • 67. Setup • included with PostgreSQL • configure --with-python • apt-get/yum install postgresql-plpython • CREATE LANGUAGE plpythonu; • Python 3: CREATE LANGUAGE plpython3u; • “untrusted”, superuser only
  • 68. Basic Examples CREATE FUNCTION add(a int, b int) RETURNS int LANGUAGE plpythonu AS $$ return a + b $$; CREATE FUNCTION longest(a text, b text) RETURNS text LANGUAGE plpythonu AS $$ if len(a) > len(b): return a elif len(b) > len(a): return b else: return None $$;
  • 69. Using Modules CREATE FUNCTION json_to_array(j text) RETURNS text[] LANGUAGE plpythonu AS $$ import json return json.loads(j) $$;
  • 70. Database Calls CREATE FUNCTION clear_passwords() RETURNS int LANGUAGE plpythonu AS $$ rv = plpy.execute("UPDATE customers SET password = NULL") return rv.nrows $$;
  • 71. Database Calls With Parameters CREATE FUNCTION set_password(username text, password text) RETURNS boolean LANGUAGE plpythonu AS $$ plan = plpy.prepare("UPDATE customers SET password = $1 WHERE username= $2", ['text', 'text']) rv = plpy.execute(plan, [username, password]) return rv.nrows == 1 $$;
  • 72. Avoiding Prepared Statements CREATE FUNCTION set_password(username text, password text) RETURNS boolean LANGUAGE plpythonu AS $$ rv = plpy.execute("UPDATE customers SET password = %s WHERE username= %s" % (plpy.quote_nullable(username), plpy.quote_literal(password))) return rv.nrows == 1 $$; (available in 9.1-to-be)
  • 73. Caching Plans CREATE FUNCTION set_password2(username text, password text) RETURNS boolean LANGUAGE plpythonu AS $$ if 'myplan' in SD: plan = SD['myplan'] else: plan = plpy.prepare("UPDATE customers SET password = $1 WHERE username= $2", ['text', 'text']) SD['myplan'] = plan rv = plpy.execute(plan, [username, password]) return rv.nrows == 1 $$;
  • 74. Processing Query Results CREATE FUNCTION get_customer_name(username text) RETURNS boolean LANGUAGE plpythonu AS $$ plan = plpy.prepare("SELECT firstname || ' ' || lastname AS ""name"" FROM customers WHERE username = $1", ['text']) rv = plpy.execute(plan, [username], 1) return rv[0]['name'] $$;
  • 75. Compare: PL/Python vs. DB-API PL/Python: plan = plpy.prepare("SELECT ...") for row in plpy.execute(plan, ...): plpy.info(row["fieldname"]) DB-API: dbconn = psycopg2.connect(...) cursor = dbconn.cursor() cursor.execute("SELECT ...") for row in cursor.fetchall() do: print row[0]
  • 76. Set-Returning and Table Functions CREATE FUNCTION get_customers(id int) RETURNS SETOF customers LANGUAGE plpythonu AS $$ plan = plpy.prepare("SELECT * FROM customers WHERE customerid = $1", ['int']) rv = plpy.execute(plan, [id]) return rv $$;
  • 77. Triggers CREATE FUNCTION delete_notifier() RETURNS trigger LANGUAGE plpythonu AS $$ if TD['event'] == 'DELETE': plpy.notice("one row deleted from table %s" % TD['table_name']) $$; CREATE TRIGGER customers_delete_notifier AFTER DELETE ON customers FOR EACH ROW EXECUTE PROCEDURE delete_notifier();
  • 78. Exceptions CREATE FUNCTION test() RETURNS text LANGUAGE plpythonu AS $$ try: rv = plpy.execute("SELECT ...") except plpy.SPIError, e: plpy.notice("something went wrong") The transaction is still aborted in < 9.1.
  • 79. New in PostgreSQL 9.1 • SPI calls wrapped in subtransactions • custom SPI exceptions: subclass per SQLSTATE, .sqlstate attribute • plpy.subtransaction() context manager • support for OUT parameters • quoting functions • validator • lots of internal improvements