Parallelising Dynamic Programming
1 like390 views
The document discusses the parallelization of dynamic programming algorithms with a focus on efficiency, characterizing dynamic programming recurrences, and finding efficient parallel implementations. It examines complexity theory, relative runtimes, and asymptotic scalability, emphasizing the significance of utilizing processors effectively for large inputs. The paper also highlights future work including theoretical advancements, generalizing for more dimensions, and improving implementations and compiler integration.
![Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p
But...
...what are good values?
Clear: SA
p ∈ [0, p] and EA
p ∈ [0, 1]](https://image.slidesharecdn.com/slides-160111164910/85/Parallelising-Dynamic-Programming-20-320.jpg)
![Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p
But...
...what are good values?
Clear: SA
p ∈ [0, p] and EA
p ∈ [0, 1] – but we can certainly not always
hit the optima!](https://image.slidesharecdn.com/slides-160111164910/85/Parallelising-Dynamic-Programming-21-320.jpg)
![Reducing to dependencies
e(i, j) :=
0 i = j = 0
j i = 0 ∧ j > 0
i i > 0 ∧ j = 0
min
e(i − 1, j) + 1
e(i, j − 1) + 1
e(i − 1, j − 1) + [ vi = wj ]
else](https://image.slidesharecdn.com/slides-160111164910/85/Parallelising-Dynamic-Programming-32-320.jpg)
![Reducing to dependencies
e(i, j) :=
0 i = j = 0
j i = 0 ∧ j > 0
i i > 0 ∧ j = 0
min
e(i − 1, j) + 1
e(i, j − 1) + 1
e(i − 1, j − 1) + [ vi = wj ]
else](https://image.slidesharecdn.com/slides-160111164910/85/Parallelising-Dynamic-Programming-33-320.jpg)
Parallelising Dynamic Programming
- 1. Parallelising Dynamic Programming Raphael Reitzig University of Kaiserslautern Department of Computer Science Algorithms and Complexity Group September 27th, 2012
- 2. Vision Compile dynamic programming recurrences into efficient parallel code.
- 3. Goal 1 Understand what efficiency means in parallel algorithms.
- 4. Goal 1 Understand what efficiency means in parallel algorithms. Goal 2 Characterise dynamic programming recurrences in a suitable way.
- 5. Goal 1 Understand what efficiency means in parallel algorithms. Goal 2 Characterise dynamic programming recurrences in a suitable way. Goal 3 Find and implement efficient parallel algorithms for DP.
- 8. Complexity theory Classifies problems Focuses on inherent parallelism
- 9. Complexity theory Classifies problems Focuses on inherent parallelism Answers: How many processors do you need to be really fast on inputs of a given size?
- 10. Complexity theory Classifies problems Focuses on inherent parallelism Answers: How many processors do you need to be really fast on inputs of a given size? But... ...p grows with n – no statement about constant p and growing n!
- 11. Amdahl’s law Parallel speedup ≤ 1 1−γ+γ p .
- 12. Amdahl’s law Parallel speedup ≤ 1 1−γ+γ p . Answers: How many processors can you utilise on given inputs?
- 13. Amdahl’s law Parallel speedup ≤ 1 1−γ+γ p . Answers: How many processors can you utilise on given inputs? But... ...does not capture growth of n!
- 14. Work and depth Work W = TA 1 and depth D = TA ∞
- 15. Work and depth Work W = TA 1 and depth D = TA ∞ Brent’s Law: A with W p ≤ TA p < W p + D is possible in a certain setting.
- 16. Work and depth Work W = TA 1 and depth D = TA ∞ Brent’s Law: A with W p ≤ TA p < W p + D is possible in a certain setting. But... ...has limited applicability and D can be slippery!
- 17. Relative runtimes Speedup SA p := TA 1 TA p
- 18. Relative runtimes Speedup SA p := TA 1 TA p Efficiency EA p := TB p·TA p
- 19. Relative runtimes Speedup SA p := TA 1 TA p Efficiency EA p := TB p·TA p But... ...what are good values?
- 20. Relative runtimes Speedup SA p := TA 1 TA p Efficiency EA p := TB p·TA p But... ...what are good values? Clear: SA p ∈ [0, p] and EA p ∈ [0, 1]
- 21. Relative runtimes Speedup SA p := TA 1 TA p Efficiency EA p := TB p·TA p But... ...what are good values? Clear: SA p ∈ [0, p] and EA p ∈ [0, 1] – but we can certainly not always hit the optima!
- 22. Proposal: Asymptotic relative runtimes Definition SA p(∞) := lim inf n→∞ SA p(n) ? = p EA p (∞) := lim inf n→∞ EA p (n) ? = 1
- 23. Proposal: Asymptotic relative runtimes Definition SA p(∞) := lim inf n→∞ SA p(n) ? = p EA p (∞) := lim inf n→∞ EA p (n) ? = 1 Goal Find parallel algorithms that are asymptotically as scalable and efficient as possible for all p.
- 24. Disclaimer This means: A good parallel algorithm can utilise any number of processors if the inputs are large enough.
- 25. Disclaimer This means: A good parallel algorithm can utilise any number of processors if the inputs are large enough. Not: More processors are always better.
- 26. Disclaimer This means: A good parallel algorithm can utilise any number of processors if the inputs are large enough. Not: More processors are always better. Just as in sequential algorithmics.
- 27. Afterthoughts Machine model Keep it simple: (P)RAM with p processors and spawn/join.
- 28. Afterthoughts Machine model Keep it simple: (P)RAM with p processors and spawn/join. Which quantities to analyse? Elementary operations, memory accesses, inter-thread communication, ...
- 29. Afterthoughts Machine model Keep it simple: (P)RAM with p processors and spawn/join. Which quantities to analyse? Elementary operations, memory accesses, inter-thread communication, ... Implicit interaction – blocking, communication via memory, ... – is invisible in code!
- 31. Disclaimer Only two dimensions Only finite domains Only rectangular domains Memoisation-table point-of-view
- 32. Reducing to dependencies e(i, j) := 0 i = j = 0 j i = 0 ∧ j > 0 i i > 0 ∧ j = 0 min e(i − 1, j) + 1 e(i, j − 1) + 1 e(i − 1, j − 1) + [ vi = wj ] else
- 33. Reducing to dependencies e(i, j) := 0 i = j = 0 j i = 0 ∧ j > 0 i i > 0 ∧ j = 0 min e(i − 1, j) + 1 e(i, j − 1) + 1 e(i − 1, j − 1) + [ vi = wj ] else
- 34. Gold standard
- 37. ?
- 38. ?
- 39. ?
- 43. ?
- 48. Simplification DL D DR UL U UR L R
- 51. Three cases Assuming dependencies are area-complete and uniform, there are only three cases up to symmetry:
- 52. Facing Reality
- 55. Challenges Contention Method of synchronisation Metal issues (moving threads, cache sync)
- 56. Performance Examples Edit distance on two-core shared memory machine: 0 0.2 0.4 0.6 0.8 1 1.2 1.4 ·105 0 0.5 1 1.5 2 2.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 ·105 0 0.5 1 1.5 2 2.5
- 57. Performance Examples Edit distance on four-core NUMA machine: 0 1 2 3 4 ·105 0 1 2 3 4 0 1 2 3 4 ·105 0 1 2 3 4
- 58. Performance Examples Pseudo-Bellman-Ford on two-core shared memory machine: 0 0.2 0.4 0.6 0.8 1 1.2 1.4 ·105 0 0.5 1 1.5 2 2.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 ·105 0 1 2 3 4
- 59. Performance Examples Pseudo-Bellman-Ford on four-core NUMA machine: 0 1 2 3 4 ·105 0 1 2 3 4 0 1 2 3 4 ·105 0 2 4 6 8
- 60. Future Work Fill gaps in theory (caching and communication).
- 61. Future Work Fill gaps in theory (caching and communication). Generalise theory to more dimensions and interleaved DPs.
- 62. Future Work Fill gaps in theory (caching and communication). Generalise theory to more dimensions and interleaved DPs. Improve and extend implementations.
- 63. Future Work Fill gaps in theory (caching and communication). Generalise theory to more dimensions and interleaved DPs. Improve and extend implementations. More experiments (different problems, more diverse machines).
- 64. Future Work Fill gaps in theory (caching and communication). Generalise theory to more dimensions and interleaved DPs. Improve and extend implementations. More experiments (different problems, more diverse machines). Improve compiler integration (detection, backtracing, result functions).
- 65. Future Work Fill gaps in theory (caching and communication). Generalise theory to more dimensions and interleaved DPs. Improve and extend implementations. More experiments (different problems, more diverse machines). Improve compiler integration (detection, backtracing, result functions). Integrate with other tools.