[Paper Reading] Generalized Sub-Query Fusion for Eliminating Redundant I/O from Big-Data Queries
Download as PPTX, PDF0 likes894 views
This document describes RESIN, a query optimizer that eliminates redundant I/O for big data queries. RESIN introduces two new operators - ResinMap and ResinReduce - and two optimization rules - sub-query fusion and binary-operator elimination. These optimizations were found to benefit 40% of queries in the TPC-DS benchmark, improving performance by an average of 1.4x. The optimizer works by fusing operators applied to the same table, eliminating redundant joins or unions, and combining grouped aggregations. An evaluation on a 10GB TPC-DS dataset found RESIN's optimizations significantly reduced redundant I/O for many real-world analytical queries.
![Sub-query Fusion
• Basic query fusion
– φ ：filter condition
– 𝐶 ← 𝐸：project mapping
– λ[φ, 𝐶 ← 𝐸]：filter(φ）+ project(𝐶 ← 𝐸) （λ is also called
ResinSimpleMap）](https://image.slidesharecdn.com/resin-210920113222/85/Paper-Reading-Generalized-Sub-Query-Fusion-for-Eliminating-Redundant-I-O-from-Big-Data-Queries-14-320.jpg)
![Unary Operator Fusion
• Case 1: 𝑜𝑝1, 𝑜𝑝2 are ResinSimpleMap(λ)
λ[φ1 ∧ φ𝑟1 ∧ φ2 ∧ φ𝑟2,
𝐶1 ← 𝐸1 ∪ 𝐶2 ← 𝐸2
∪ 𝐼(𝑐𝑜𝑙𝑠 φ1 ) ∪ 𝐼(𝑐𝑜𝑙𝑠 φ𝑟1 )
∪ 𝐼(𝑐𝑜𝑙𝑠 φ2 ) ∪ 𝐼(𝑐𝑜𝑙𝑠 φ𝑟2 ))]
Q
λ[φ1 ∧ φ𝑟1, 𝐼(𝐶1)] λ[φ2 ∧ φ𝑟2, 𝐼(𝐶2)]
Basic
Rule](https://image.slidesharecdn.com/resin-210920113222/85/Paper-Reading-Generalized-Sub-Query-Fusion-for-Eliminating-Redundant-I-O-from-Big-Data-Queries-16-320.jpg)
![Unary Operator Fusion
• Case 2: 𝑜𝑝1, 𝑜𝑝2 are GroupBy(γ)
– ρ[k, List(φ, agg)]: ResionReduce which groups by k](https://image.slidesharecdn.com/resin-210920113222/85/Paper-Reading-Generalized-Sub-Query-Fusion-for-Eliminating-Redundant-I-O-from-Big-Data-Queries-17-320.jpg)
![Binary operator elimination
• Case 1: Union elimination
– μ[List(φ,C ← E)]: ResinMap](https://image.slidesharecdn.com/resin-210920113222/85/Paper-Reading-Generalized-Sub-Query-Fusion-for-Eliminating-Redundant-I-O-from-Big-Data-Queries-22-320.jpg)
![Binary operator elimination
• Case 2: Join elimination
– ρ[k, List(φ, agg)]: ResionReduce which groups by k](https://image.slidesharecdn.com/resin-210920113222/85/Paper-Reading-Generalized-Sub-Query-Fusion-for-Eliminating-Redundant-I-O-from-Big-Data-Queries-23-320.jpg)
[Paper Reading] Generalized Sub-Query Fusion for Eliminating Redundant I/O from Big-Data Queries
- 1. Generalized Sub-Query Fusion for Eliminating Redundant I/O from Big-Data Queries Kaushik Rajan, Partho Sarthi, Akash Lal, Abhishek Modi, Ashit Gosalia, Prakhar Jain, Mo Liu, Saurabh Kalikar OSDI 20’ Presented by Mingcong Han (GitHub: francis0407) Credits: some pictures from the conference presentation slides
- 2. Big data query compilation
- 3. Redundant stages of processing • TPCDS, 40% of queries have redundant I/O • 16% of all queries, high-impact spend at least 50% time with redundant I/O • 9% medium impact, spend 10-50% time with redundant I/O
- 4. State-of-the-art Query Optimizer SparkSQL Catalyst Optimizer • SQL -> LogicalPlan -> PhysicalPlan -> MapReduce
- 5. RESIN: MapReduce reasoning during optimization
- 6. RESIN: MapReduce reasoning during optimization
- 7. Example 1 —— ResinMap
- 8. Example 1 —— ResinMap
- 9. Example 2 —— ResinReduce F1 = SELECT id as 𝑐1, max(𝑠𝑖𝑔𝑛𝑎𝑙1) as 𝑠1 FROM iotLogs WHERE ℎ𝑟1 ≤ 12 GROUP BY id F2 = SELECT id as 𝑐2, max(𝑠𝑖𝑔𝑛𝑎𝑙2) as 𝑠2 FROM iotLogs WHERE ℎ𝑟2 ≤ 18 GROUP BY id R = SELECT 𝑐1, 𝑠1, 𝑠2 FROM F1 JOIN F2 ON 𝑐1 = 𝑐2
- 10. Example 2 —— ResinReduce 𝑚𝑎𝑥1 = 𝑚𝑎𝑥2 = ∞ 𝑟𝑐1 = 𝑟𝑐2 = 0 foreach (<id, 𝑠𝑖𝑔𝑛𝑎𝑙1, 𝑠𝑖𝑔𝑛𝑎𝑙2, ℎ𝑟1, ℎ𝑟2> in partition) { if (ℎ𝑟1 ≤ 12) { 𝑚𝑎𝑥1=max(𝑚𝑎𝑥1, 𝑠𝑖𝑔𝑛𝑎𝑙1); 𝑟𝑐1++; } if (ℎ𝑟2 ≤ 18) { 𝑚𝑎𝑥2=max(𝑚𝑎𝑥2, 𝑠𝑖𝑔𝑛𝑎𝑙2); 𝑟𝑐2++; } } output(id, 𝑚𝑎𝑥1, 𝑚𝑎𝑥2, 𝑟𝑐1, 𝑟𝑐2)
- 11. Resin Operators
- 12. Resin Optimization Rules 1. Sub-query fusion – i.e. fuse the operators applied on the same table 2. Binary operator elimination – i.e. eliminate the redundant Union/Join after fusion
- 13. Resin Optimization Rules 1. Sub-query fusion – i.e. fuse the operators applied on the same table 2. Binary operator elimination – i.e. eliminate the redundant Union/Join after fusion
- 14. Sub-query Fusion • Basic query fusion – φ ：filter condition – 𝐶 ← 𝐸：project mapping – λ[φ, 𝐶 ← 𝐸]：filter(φ）+ project(𝐶 ← 𝐸) （λ is also called ResinSimpleMap）
- 15. Unary Operator Fusion • Fuse the operators where 𝑜𝑝1, 𝑜𝑝2 are one of GroupBy(γ), ResinReduce(ρ), and ResinSimpleMap(λ)
- 16. Unary Operator Fusion • Case 1: 𝑜𝑝1, 𝑜𝑝2 are ResinSimpleMap(λ) λ[φ1 ∧ φ𝑟1 ∧ φ2 ∧ φ𝑟2, 𝐶1 ← 𝐸1 ∪ 𝐶2 ← 𝐸2 ∪ 𝐼(𝑐𝑜𝑙𝑠 φ1 ) ∪ 𝐼(𝑐𝑜𝑙𝑠 φ𝑟1 ) ∪ 𝐼(𝑐𝑜𝑙𝑠 φ2 ) ∪ 𝐼(𝑐𝑜𝑙𝑠 φ𝑟2 ))] Q λ[φ1 ∧ φ𝑟1, 𝐼(𝐶1)] λ[φ2 ∧ φ𝑟2, 𝐼(𝐶2)] Basic Rule
- 17. Unary Operator Fusion • Case 2: 𝑜𝑝1, 𝑜𝑝2 are GroupBy(γ) – ρ[k, List(φ, agg)]: ResionReduce which groups by k
- 18. Binary Operator Fusion • Fuse the operators where 𝑜𝑝1, 𝑜𝑝2 are one of Join(ψ, jt) and Union()
- 19. Binary Operator Fusion • Case 1: 𝑜𝑝1, 𝑜𝑝2 are Join(ψ, jt) – Ψ: join condition – jt: join type (inner, outer, …)
- 20. Binary Operator Fusion • Case 2: 𝑜𝑝1, 𝑜𝑝2 are Union
- 21. Resin Optimization Rules 1. Sub-query fusion – i.e. fuse the operators applied on the same table 2. Binary operator elimination – i.e. eliminate the redundant Union/Join after fusion
- 22. Binary operator elimination • Case 1: Union elimination – μ[List(φ,C ← E)]: ResinMap
- 23. Binary operator elimination • Case 2: Join elimination – ρ[k, List(φ, agg)]: ResionReduce which groups by k
- 24. Example 3
- 25. Join id = did Project id, signal Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Filter (hr>=5 and hr<=19) Filter (hr<=7 or hr>=17) Filter (ht <= 2) Filter (ht >= 11) Project id, signal Project did, city Project did, city Project city, signal Join id = did Project city, signal GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2 Standard Plan
- 26. Step 1: Fuse Filter+Project Join id = did Project id, signal Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Filter (hr>=5 and hr<=19) Filter (hr<=7 or hr>=17) Filter (ht <= 2) Filter (ht >= 11) Project id, signal Project did, city Project did, city Project city, signal Join id = did Project city, signal GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2
- 27. Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Project city, signal Join id = did Project city, signal GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2 ResinSimpleMap hr>=5 and hr<=19 id, signal ResinSimpleMap hr<=7 or hr>=17 id, signal ResinSimpleMap ht <= 2 did, city ResinSimpleMap ht >= 11 did, city Step 1: Fuse Filter+Project
- 28. Step 2: Fuse Join Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Project city, signal Join id = did Project city, signal GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2 ResinSimpleMap hr>=5 and hr<=19 id, signal ResinSimpleMap hr<=7 or hr>=17 id, signal ResinSimpleMap ht <= 2 did, city ResinSimpleMap ht >= 11 did, city
- 29. Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2 ResinSimpleMap hr>=5 and hr<=19 and ht<=2 city, signal ResinSimpleMap (hr<=7 or hr>=17) and ht>=11 city, signal ResinSimpleMap True id, signal, hr ResinSimpleMap ht <= 2 or ht >= 11 did, city, ht, area Step 2: Fuse Join
- 30. Step 3: Fuse GroupBy Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) GroupBy c1 ← city s1 ← max(signal) GroupBy c2 ← city s2 ← max(signal) Join c1 = c2 ResinSimpleMap hr>=5 and hr<=19 and ht<=2 city, signal ResinSimpleMap (hr<=7 or hr>=17) and ht>=11 city, signal ResinSimpleMap True id, signal, hr ResinSimpleMap ht <= 2 or ht >= 11 did, city, ht, area
- 31. Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Join c1 = c2 ResinSimpleMap True id, signal, hr ResinSimpleMap ht <= 2 or ht >= 11 did, city, ht, area ResinReduce GroupBy: c ← city Filter1: hr>=5 and hr<=19 and ht<=2; Agg1: s1← max(signal), rc1←count(*) Filter2: (hr<=7 or hr>=17) and ht>=11; Agg2: s2 ← max(signal), rc2 ←cont(*) ResinSimpleMap rc1 > 0 c1 ← c, s1 ResinSimpleMap rc1 > 0 c1 ← c, s1 Step 3: Fuse GroupBy
- 32. Step 4: Eliminate Join Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) Join c1 = c2 ResinSimpleMap True id, signal, hr ResinSimpleMap ht <= 2 or ht >= 11 did, city, ht, area ResinReduce GroupBy: c ← city Filter1: hr>=5 and hr<=19 and ht<=2; Agg1: s1← max(signal), rc1←count(*) Filter2: (hr<=7 or hr>=17) and ht>=11; Agg2: s2 ← max(signal), rc2 ←cont(*) ResinSimpleMap rc1 > 0 c1 ← c, s1 ResinSimpleMap rc1 > 0 c1 ← c, s1
- 33. Join id = did Table: signals (id, hr, signal) Table: dInfo (did,city,ht,area) ResinSimpleMap True id, signal, hr ResinSimpleMap ht <= 2 or ht >= 11 did, city, ht, area ResinReduce GroupBy: c ← city Filter1: hr>=5 and hr<=19 and ht<=2; Agg1: s1← max(signal), rc1←count(*) Filter2: (hr<=7 or hr>=17) and ht>=11; Agg2: s2 ← max(signal), rc2 ←cont(*) ResinSimpleMap rc1 > 0 and rc2 > 0 c1 ← c, s1, s2 Step 4: Eliminate Join
- 34. Evaluation • TPC-DS 10G, 40 of 104 queries are affected
- 35. Evaluation
- 36. Conclusion • RESIN, a query optimizer that eliminates the redundant I/O for big-data queries – Two new operators: ResinMap, ResinReduce – Two new rules: sub-query fusion, binary-operator elimination • The optimizations are useful for 40% of queries in TPCDS with 1.4x improvement on average
- 37. Q & A
- 38. Thanks