Querying Nested JSON Data Using N1QL and Couchbase

Editor's Notes

• #7: Scalability – Multi node, auto-sharded architecture makes it easy to scale out horizontally Availability – Multi node architecture makes high availability easy COUCH = Cluster of Unreliable Commodity Hardware Agility – JSON documents without schema enforcement makes it easy for teams to iterate quickly
• #9: Non-first normal form query language
• #10: Think millions of documents Schema is not enforced by the DB
• #11: Let’s see how to represent customer data in JSON. The primary (CustomerID) becomes the DocumentKey Column name-Column value becomes KEY-VALUE pair.
• #15: What if I wanted to filter to airports with an altitude less than 1000? Just use Javascript dot notation to access attributes at any depth You may also use Javascript square bracket array notation to access items in arrays by index
• #17: Non-first normal form query language
• #18: You can use the LOWER function to avoid case-sensitive collation String concatenation is one difference from SQL, uses double vertical bars. Since we can’t know the type in advance, we need a separate concat operator from the addition operator
• #21: Note that array elements don’t necessarily have to be of the same type, though they usually are
• #26: Non-first normal form query language
• #28: First animation: note that we use an alias on the bucket name. This prevents confusion when we’re getting multiple document types from the same bucket. Second animation: note that we’re using META().id to get the primary key of the document to join Also, note that this syntax is only available in Couchbase Server 5.5
• #29: But what if I want to join based on an attribute instead of the primary key?
• #30: The type filter on the second extent should be part of the ON clause, not the WHERE clause There must be an index to support looking up the second extent based on these clauses Not as performant as a join based on primary key, which doesn’t need an index at all
• #31: Embedding an array inside a document creates an implicit 1:N relationship between the root document and the items in the array. But how do I join across this relationship?
• #32: The type filter on the second extent should be part of the ON clause, not the WHERE clause There must be an index to support looking up the second extent based on these clauses Not as performant as a join based on primary key, which doesn’t need an index at all
• #34: Note that we want to know all the routes for a set of airports. In traditional SQL, we’d have to flatten the output, repeating the airport data for every matching route.
• #35: Nesting is analogous to GroupJoin in LINQ, where all matching documents are returned in an array We’re using an additional subquery on the array in the select projection to reduce the data we’re returning There is also LEFT OUTER NEST
• #36: Nesting is analogous to GroupJoin in LINQ, where all matching documents are returned in an array We’re using an additional subquery on the array in the select projection to reduce the data we’re returning There is also LEFT OUTER NEST
• #38: Indexes every document in the bucket by the primary key Supports any query, but with poor performance Kind of like a table scan in SQL, except it scans every document in the entire bucket Not recommend for production, except some very specific use cases
• #39: Automatically excludes any documents where “name” is MISSING The attribute must be included in the predicate for the index to be used, just like a SQL index
• #40: Automatically excludes any documents where “lastName” or “firstName” is MISSING At least the first attribute must be included in the predicate for the index to be used, just like a SQL index The second attribute will be used if possible, and so on for multiple attributes
• #41: Can use any deterministic function to adjust attributes before they are indexed Predicates must use the same function to match the index Still excludes MISSING lastName and firstName, since LOWER(MISSING) = MISSING
• #42: Can include any quantity of deterministic predicates Requires that queries must include the same predicates (all of them!) in order to match the index Because query planning occurs before parameter substitution, the type = ‘customer’ clause cannot be parameterized
• #43: Any expression in the index definition can be an ARRAY clause, though only one array is allowed Includes all values in the array, so long arrays can significantly increase index size Animation: You can also use functions as part of the array clause
• #44: DCP streams mutations (inserts, updates, and deletes) to the index nodes Streaming is async, thus indexes are eventually consistent High availability and load balancing is provided by having replicas on more than one node, each replica is a full copy of the index Any given query only accesses one copy of the index on one node, avoiding scatter/gather for low latency
• #45: When building an index, streams the entire bucket from the data nodes to the index node Only one index build can be running at a time By building more than one index with a single BUILD command, we can share the stream
• #46: Creates two complete copies of the index, on two different index nodes Provides HA and load balancing
• #47: New in 5.5, only available in enterprise edition Spreads the index across all nodes in the cluster (optionally a subset of nodes), deciding which node receives which part of the index based on a deterministic hash of the referenced attribute Good for particularly large indexes, as index can now scale horizontally Creates a scatter/gather situation which can increase latency, but that is eliminated if you include a equality predicate for the hashed attribute so it can go to just one node
• #50: Note that if the index contains all data needed by the query, it will “cover” the query, meaning steps 4 and 5 are skipped Key to optimizing this process is to reduce waste in step 4, avoid having documents returned from the index that are then thrown out by Step 7
• #52: Oversimplification, but delivers the concept Which one of these do you think is most efficient? Depends on the relative sizes of the two lists Short first list, isn’t worth the time to build the hash table
• #53: Normal attribute join uses an inner loop, which is inefficient if the left hand extent has lots of data and the right hand side is small repeating set Hash join is an optimization automatically selected by RDBMS implementations, but must be manually chosen in N1QL Builds a hash table of all possible matches on the right hand extent, and uses the hash table when processing the left hand extent Use “probe” instead of “build” to build the hash table on the left side instead of the right (should be the smaller set) Only available on 5.5 Enterprise Edition (free for dev, but costs for production, includes support)