Power JSON with PostgreSQL

Power JSON
with
PostgreSQL
Simon Riggs
Postgres Fellow
20 January 2021

Part 1:
SQL Development
and Indexing

© Copyright EnterpriseDB Corporation, 2020. All rights reserved.
3
PostgreSQL
● PostgreSQL is the
World’s Most Advanced Open Source Database
● PostgreSQL has very good adherence to
the SQL Standard over many years of development,
including many new proposed changes for PG14
● But what about JSON?

4
JSON datatype in PostgreSQL
• First order datatype in PostgreSQL
• Treated the same as TEXT, INTEGER, etc..
• Indexable, searchable, compressible
• Can be used in same table as other data types - truly Multi-Model
• Added to PostgreSQL 9.2 in 2012 by Robert Haas, EDB
• Significant additional work in later releases by Andrew Dunstan, EDB
• Major work on indexing and also on SQL/JSON by Oleg Bartunov and
Nikhita Glukhov at PostgresPro, supported by Andrew Dunstan
• PostgreSQL already supported XML documents, Full Text Search, Arrays,
Nested Record types, Domains so there are many options for data storage

5
JSON in the Real World
{
"Servers": {
"1": {
"Name": "Minimally Defined Server",
"Group": "Server Group 1",
"Port": 5432,
"Username": "postgres",
"Host": "localhost",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
},
"2": {
"Name: "Fully Defined Server",
"Group": "Server Group 2",
"Host": "host.domain.com",
"HostAddr": "192.168.1.2",
"Port": 5432,
"MaintenanceDB": "postgres",
"Username": "postgres",
"Role": "my_role_name",
"SSLMode": "require",
"Comment": "This server has every option configured in the JSON",
"DBRestriction": "live_db test_db",
"BGColor": "#ff9900",
"FGColor": "#000000",
"Service": "postgresql-10",
}
}
}
• This example is a config
export file from pgAdmin
• I found 305 files of type
".json" on my laptop,
used by applications...
• pgAdmin
• Trove
• Django
• Grafana
• pgjdbc
• docker
• rails
• github
• etc..

6
Good reasons to use JSON
• Javascript or similar applications
• JSON now very common format - so any messages and/or data
• XML is too verbose without compression, which is hard/unavailable
• Other databases use JSON, so you may wish to migrate to PostgreSQL

7
Less Good reasons to use JSON
• "Add new fields easily"
PostgreSQL can do all of these actions as metadata-only DDL commands
• ALTER TABLE foo ADD COLUMN
• ALTER TABLE foo DROP COLUMN
• ALTER TABLE foo ALTER COLUMN TYPE varchar(128) TO varchar(256)
• So if your JSON looks very relational, you may be better with columns
• Dynamic tagging is still important use of JSON especially in fast-moving
applications such as predictive analytics

9
Creating a Sample Table for JSON data
CREATE TABLE j (jdoc JSONB);
INSERT INTO j
SELECT jsonb_build_object(
'oid', generate_series(1,10000),
'ots', clock_timestamp(),
'price', (random()*20.0)::numeric(5,2)
);
• Create a data Table with
just one column called
“jdoc” using datatype
JSONB
• Similar to a “Collection”
• No need to have multiple
relational columns
• Insert 10000 rows of
random test data with one
JSON document per row
• field/value pairs, so each
doc has same 3 fields

10
Displaying JSON data (psql)
pset linestyle old-ascii
SELECT jsonb_pretty(jdoc) AS “JSON”
FROM j TABLESAMPLE SYSTEM (5)
LIMIT 1;
JSON
------------------------------------------------
{
"oid": 521,
"ots": "2021-01-16T15:52:14.70032+00:00",
"price": 5.62
}
{
"oid": 522,
"ots": "2021-01-16T15:52:14.700351+00:00",
"price": 13.00
}
(2 rows)
• When data in a column
wraps, we normally show
a “+” sign, so make that
invisible in the psql output
• Use jsonb_pretty()
function to neatly present
JSON data
• Note it is jsonb_pretty(),
not json_pretty()

11
Displaying JSON data (pgAdmin)
• JSON data can be
displayed in the window:
Query Editor
• At first the data is
displayed only as a single
line, so shows nothing
apart from the leading {
• Hover cursor over the cell
to see value, OR
• Double-Click on the cell to
open an multiline popup

12
Valid JSON
{
"oid": 521,
"ots": "2021-01-16T15:52:14.70032+00:00",
"price": 5.62
"descr": "Example of some text data",
"boolfield": true
"tags": ["blue", "green", "red"]
"addr": {
"city": "New York",
"state": "NY"
},
"other": null
}
• JSON data is a text format
• Whitespace has no meaning
• Field/Value pairs
• Numeric
• Timestamp
• Text
• Boolean
• Arrays
• Nested fields
• Null

13
Reporting JSON Format Errors
text
-------------------
{
"one": 1,
"two": "two",
"three": ,true
}
(1 row)
ERROR: invalid input syntax for type json
DETAIL: Expected JSON value, but found ",".
CONTEXT: JSON data, line 3: "two":,
• ERRORs show the line
number on which the error
occurred and other context
data on that line of JSON
• PG13 had a bug in it that
meant the line number was
wrong and the context
block was too large, leading
to confusion for developers
• [PATCHED]

15
Adding and removing fields from JSON docs
UPDATE j
SET jdoc = jdoc || '{ "newfield2": true}'
WHERE…
UPDATE j
SET jdoc = jdoc || '{ "newfield2": true,
"newfield3": true}'
WHERE ...
UPDATE j
SET jdoc = jdoc - 'newfield2'
WHERE …
UPDATE j
SET jdoc = jdoc - '{newfield2, newfield3}'::text[]
WHERE…
UPDATE j
SET jdoc = jdoc #- '{"addr", "city"}'
WHERE..
• Add a single new field to a
JSON doc using || operator
• Multiple new fields can be
added in one JSONB doc, so
easier to parameterize
• Remove a field from JSON
doc using - (minus) operator
• Multiple fields are removed
by supplying an array of
values, easy to parameterize
• Special operator #- for
removing an exact path

16
Updating values in JSON docs
UPDATE j SET jdoc = jdoc || '{ "price": 9.57}' WHERE
...
-- Before: { "tags": [ "blue", "green", "red"] }
UPDATE j
SET jdoc = jdoc || '{ "tags": ["yellow"]}'
WHERE …
-- Ends with { "tags": [ "yellow"] }
-- The earlier values of "blue", "green", "red" were
removed!
UPDATE j
SET jdoc = jsonb_insert(jdoc , '{tags, 999}',
'"yellow"', true)
WHERE...
-- Ends as {"tags": ["blue", "green", "red", "yellow"]}
• Also use || operator to
"update" a JSON doc
• Beware, it actually has
replace semantics, so will
remove data in arrays or
substructures accidentally
• Carefully use supplied
functions to manipulate
JSON correctly. Test!

17
Extracting JSON field values
t
SELECT jdoc->'oid' FROM j;
521
SELECT jdoc->'ots' FROM j;
"2021-01-16T15:52:14.70032+00:00"
SELECT trim(both '"' from jdoc->'ots') FROM j;
2021-01-16 15:52:14.70032
SELECT jdoc->>'ots' FROM j;
2021-01-16 15:52:14.70032
• Retrieve field value
• Returns JSONB!
• Fields of type String and
Timestamp have double
quotes around them
• No need for trim() function
to remove them
• Use ->> operator instead,
which strips double quotes
and returns TEXT

Query optimization
with JSON data

19
ANALYZE
ANALYZE j;
SELECT count(*) FROM j;
count
--------
100000
EXPLAIN ANALYZE
SELECT * FROM j WHERE (jdoc->'oid')::integer = 547;
QUERY PLAN
-------------------------------------------------------
Seq Scan on j (rows=500)(rows=1)
EXPLAIN ANALYZE
SELECT * FROM j WHERE jdoc->'noexist' = 'xxxx';
QUERY PLAN
-------------------------------------------------------
Seq Scan on j (rows=500) (actual rows=0)
• ANALYZE works on JSONB
• Not currently a typanalyze
function, so that estimates
are just heuristics
• = always shows as 0.5%
< always shows as 66%
> always shows as 66%

20
Indexing JSONB
CREATE INDEX ON j (jdoc);
CREATE INDEX ON j ((jdoc->'oid'));
EXPLAIN ANALYZE
SELECT * FROM j WHERE (jdoc->'oid')::integer = 547;
QUERY PLAN
-------------------------------------------------------
Seq Scan on j (rows=500)(actual rows=1)
EXPLAIN ANALYZE
SELECT * FROM j WHERE jdoc->>'oid' = '547';
QUERY PLAN
-------------------------------------------------------
Seq Scan on j (rows=1)(actual rows=1)
• Not useful for JSON data
• B-tree index on one field,
so index will be small-ish
• Be careful to avoid query
syntax that prevents use of
the expression index
• Use ->> operator to extract
field value as text and
compare with a text value

21
Partial Index on JSONB
CREATE UNIQUE INDEX ON j ((jdoc->'oid'))
WHERE (jdoc->'oid') IS NOT NULL;
EXPLAIN ANALYZE
SELECT * FROM j WHERE jdoc->>'oid' = '547';
QUERY PLAN
-------------------------------------------------------
Index Scan using j_expr_idx on j (rows=1)(actual rows=1)
• Use Partial Index to index
only the rows that contain
the "oid" field
• Any search on "oid" will
imply non-NULL result and
so can use partial index
• Not all operators allow
useful implications
• Now we have a Unique
Index the estimates are
now correct for query

22
General Purpose Indexing for JSONB
CREATE INDEX ON j
USING GIN (jdoc);
SELECT *
FROM j
WHERE jdoc operator value;
-- works with these search operators:
-- @>, <@, @@, @?, ?, ?|, ?&,
• Easy indexing!
• GIN index includes
all fields and all values
• Index can be very large
and may reduce
concurrency on large
tables with high
insert/update rate

23
Smaller, better indexing for JSONB
CREATE INDEX ON j
USING GIN (jdoc jsonb_path_ops);
SELECT *
FROM j
WHERE jdoc operator value;
-- works with these search operators (only)
-- @>, @@, @?
• Use non-default operator
class for JSON called
json_path_ops to reduce
size of index and speed++
• Only works with certain
operators but they are the
ones we want anyway
• json_path_ops uses
hashing - check this
reduces size without other
impacts on performance

25
Missing Fields don't cause ERRORs
SELECT count(*) FROM j
WHERE spoon IS NOT NULL;
ERROR: there is no "spoon"
SELECT count(*) FROM j
WHERE jdoc->'spoon' IS NOT NULL;
count
-------
0
• An absent column causes
an SQL ERROR
• Absence of a field does not
cause an ERROR, which
might lead to uncaught
issues in your applications

26
Arrays are different in SQL and JSON
CREATE TABLE a
(arr text[], jdoc jsonb);
INSERT INTO a VALUES (
ARRAY[1, 2, 3]
, '[1, 2, 3]'
);
SELECT arr[1],
jdoc ->> 0 as elem from a;
arr | elem
-----+------
1 | 1
• Arrays look similar in SQL and
JSON datatype but behave
differently.
• Don't "mix the streams"!
• SQL arrays start at element 1
JSON arrays start at element 0
• SQL arrays must all be of same
datatype

27
Don't rely on relative position
CREATE TABLE p (jdoc jsonb);
INSERT INTO a VALUES ('
{
"deviceid": 126216
,"metric": 17117.45
}');
SELECT jsonb_pretty(jdoc) FROM p;
jsonb_pretty
-------------------------
{
"metric": 17117.45,
"deviceid": 126216
}
• Don't rely on the relative
position of fields or arrays
• JSONB does not maintain
ordering of fields on input
• Relative position of array
elements can change on
UPDATE
• Ignore all other whitespace
except on output

28
NULL, Null and Undefined in JSON
{ "oid": 521, "other": null }
{ "oid": 522
}
{ "oid": 523, "other": "ishere"}
pset null SQLNULL
SELECT jdoc->'other' as "->"
,jdoc->'other' is NULL as "isNULL ->"
,jdoc->>'other' as "->>"
,jdoc->>'other' is NULL as "isNULL ->>"
FROM j;
-> | isNULL -> | ->> | isNULL ->>
----------+-----------+---------+------------
null | f | SQLNULL | t
SQLNULL | t | SQLNULL | t
"ishere" | f | ishere | f
• Any JSON datatype can be set
to “null”. No relationship at
all to SQL “NULL”, which is
more similar to a missing
field, for which JS returns
“undefined”.
• Recommend setting the SQL
NULL display value to
$SOMETHING_ELSE e.g.
SQLNULL to avoid confusion
• Notice that ->> retrieves value
as text and maps null to NULL

29
Datatype strictness
jdoc
--------------------------------------------
{"oid": 521, "other": null, "price": 9.57}
(1 row)
UPDATE j SET jdoc = jdoc - '{other, price}';
jdoc
--------------------------------------------
{"oid": 521, "other": null, "price": 9.57}
(1 row)
UPDATE j SET jdoc = jdoc - '{other, price}'::text[];
jdoc
--------------
{"oid": 521}
• Be careful of the
differences between
datatypes, which can cause
• Annoying errors
• Invalid SQL
• Poor optimization
• e.g. The - operator has two
modes, one for JSONB and
the other for TEXT[] array,
with different answers
• TEST, TEST, TEST!

30
Removing ObjectId() from BSON
{
"_id": ObjectId("d04a3e708305442796d7afca184490c4",
"tags": "blah"
}
CREATE FUNCTION remove_ObjectId_From_BSON(t text)
RETURNS TEXT LANGUAGE SQL
AS $$
SELECT replace(replace(t, 'ObjectId(', ''), '"),',
'",');
$$;
{
"_id": "d04a3e708305442796d7afca184490c4",
"tags": "blah"
}
• BSON can contain
ObjectId() text, which is
not valid JSON
• Other similar non-JSON
• Create a function to
remove this from data
• Leave the double quotes
because it contains hex
digits, similar to UUIDs

31
MongoDB Foreign Data Wrapper
• Open source EXTENSION, maintained and supported by EDB
• Query the BSON data directly in MongoDB
• Set up a Foreign Table that maps
• BSON to JSONB
• BSON to PostgreSQL column data
• or a mix of those two
• Send INSERTs, UPDATEs and DELETEs thru updatable views
• Caches connection data to allow fast response

32
Development with JSON
• Test!
• Use real JSON data for tests of SQL (or whatever generates SQL)
• Use automated testing to allow constant re-testing
• Use an existing test framework
• Automate performance testing for key access paths, so that you can test
for the correct plans for key SQL statements
• Plan for changes and enhancements in the JSON schema
• Check that your searches are safe against new fields or extreme values

34
Vision
● PostgreSQL will actively follow standards from SQL, IEEE, OGC, IETF (RFCs), Unicode etc..
○ (and contribute if possible)
○ More standards compliance features coming in PG15+
● “Hyperconverged Postgres” combines multiple types of data into one integrated, robust
and secure DBMS, with specialized data types and supporting data types
○ Relational data for operations and analytics
○ Document data in JSON/XML/Full Text
○ Time Series
○ Temporal/Historical
○ Graph
○ GIS

35
○ Andrew Dunstan
○ Mark Linster
○ Priti Sarode
EDB Value Add
● Support for all Production versions of PostgreSQL
● RDBA for JSON applications
● pgAdmin and PEM to manage your databases
● Maintaining and Extending PostgreSQL
● Expertise… thanks to my colleagues for blogs and feedback
○ Boriss Mejias
○ Thom Brown
○ Dave Page
○ Marco Nenciarini

End of Part 1
simon.riggs@enterprisedb.com

Power JSON with PostgreSQL

Recommended

More Related Content

What's hot (20)

Similar to Power JSON with PostgreSQL (20)

More from EDB (20)

Recently uploaded (20)

Power JSON with PostgreSQL