Writing Recursive Queries
- 1. WRITING RECURSIVE QUERIES RETRIEVING HIERARCHY DATA FROM RELATIONAL TABLES BEN LIS POSTGRESCONF US 2019
- 2. IS THIS TALK RIGHT FOR YOU? What we will cover • Why we need recursive queries • How they work • How to write them What you should know • Basic knowledge of SQL through outer joins • General programming concepts • Slides and SQL: https://github.com/benjlis/talk-writing-recursive- queries 2
- 3. ABOUT ME • Data Engineer at LEI Smart • Writes recursive queries on corporate hierarchy data • Initially found recursive SQL a bit confusing • Using PostgreSQL since 2015 • Started working with Oracle databases in 1988! • Adjunct Associate Faculty in Applied Analytics at Columbia University • Held various technical, product, and management roles on Wall Street • LinkedIn and Twitter 3
- 4. AGENDA • Motivation • Understanding Recursive SQL • Writing Recursive Queries • Next Steps • Questions & Discussion 4
- 5. MOTIVATION WHY WE NEED RECURSIVE SQL 5
- 7. LET’S QUERY EMPLOYEES, THEIR BOSS & THE BOSS'S BOSS boss-queries.sql 7
- 8. LET’S QUERY EMPLOYEES’ ENTIRE REPORTING LINE HMMM… HOW DO WE DO THAT? 8
- 9. IDEA 1: EXTEND CURRENT APPROACH “Add a few more JOINs” Need to know the number of hierarchy levels in advance Not realistic or robust! 9
- 10. IDEA 2: WRITE A PROGRAM Use Python, Java, PL/pgSQL, etc. Query all employees For each employee get their boss’s record and then their boss’s boss’s record, etc. until you get a NULL Complex and resource consumptive! 10
- 11. SHOULDN’T WE BE ABLE TO WRITE THIS QUERY IN SQL? 11
- 12. WE SHOULD BE ABLE TO WRITE THIS QUERY IN SQL • In the early days (decades) of SQL & relational databases we couldn’t • Writing a program (Idea 2) was the only solution • Other names for the hierarchy problem: • Transitive Closure • Bill of Materials • Parts Explosion • Good news: We can write this query in modern SQL using recursive SQL! 12
- 13. UNDERSTANDING RECURSIVE SQL KEY IDEAS, CTE, SYNTAX, STRUCTURE & EVALUATION 13
- 14. RECURSIVE SQL SUPPORT Part of ISO Standard SQL Introduced in SQL:1999 (aka SQL 3) Supported in PostgreSQL Oracle SQL Server DB2 14
- 15. KEY IDEAS • re·cur·sive /rəˈkərsiv/: a program or routine of which a part requires the application of the whole, so that its explicit interpretation requires in general many successive executions • Self-referencing • Your Bosses = Your Manager + Your Manager’s Bosses • Recursive SQL is an iterative process • Think while loops • But SQL is declarative!!!! • Tip: Put aside knowledge of recursion in programming • Recursive SQL uses Common Table Expressions (CTEs) 15
- 16. COMMON TABLE EXPRESSION (CTE) EXAMPLE cte.sql • Primarily used to simplify queries • Think of as inline view(s) 16
- 17. RECURSIVE CTE STRUCTURE • Keyword: recursive • Always a UNION or UNION ALL of • Non-recursive select • Recursive select 17
- 18. NON-RECURSIVE SELECT • Evaluated 1st - where we start • FROM clause: • Must not refer to <cte-name> • Refers to table containing hierarchy 18
- 19. RECURSIVE SELECT • Evaluated 2..N times • FROM clause: • Must reference <cte-name> • Join of the <cte-name> and table containing hierarchy • N is a function of the data and the recursive select’s where clause 19
- 20. RECURSIVE CTE EVALUATION QR = Query Results; WT = Working Table Both initially empty 1. Execute <non-recursive select> 2. Add results to QR and WT 3. Repeat until WT is empty a. Execute <recursive select> using WT data for <cte-name> b. Add results to QR c. Replace WT with results 20
- 21. WRITING RECURSIVE QUERIES LET’S GET DOWN TO IT! 21
- 22. WHAT IS ADAMS REPORTING LINE? rl.sql 22
- 23. ALL EMPLOYEES FROM CEO TO ENTRY LEVEL oc.sql 23
- 26. NEXT STEPS RECOMMENDED READINGS & BEYOND HIERARCHY 26
- 27. RECOMMENDED READINGS • PostgreSQL documentation • Fun with SQL: Recursive CTEs in Postgres – Craig Kerstiens • Modern SQL in Open Source and Commercial Databases (slides 45-60) – Markus Winand • Exporting a Hierarchy in JSON: with recursive queries - Dmitri Fontaine • Joe Celko’s Trees and Hierarchies in SQL for Smarties (Book) 27
- 28. BEYOND HIERARCHY • Hierarchy queries are the most common application of recursive SQL, but not the only application! • Pretty wild recursive SQL queries: • Solving the Traveling Salesman Problem with Postgres Recursive CTEs - Periscope Data • Mandelbrot Set • tic-tac-toe • My views: • Can doesn’t mean should • Worth studying for learning advanced techniques 28
- 29. QUESTIONS 29