TMPA-2017: Using Functional Directives to Analyze Code Complexity and Communication
0 likes834 views
The document discusses the integration of functional directives in analyzing code complexity and communication within high-performance computing. It outlines the evolution of CPUs, the implications of Amdahl's Law on parallel computing, and presents methods to improve programmability and portability through higher-order functions and hierarchical structures. Additionally, it presents examples related to heat dissipation and the n-body problem, illustrating code transformation and performance optimization strategies.
![:: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 10
map :: (a -> b) -> [a] -> [b]
map (*2) [1,2,3,4] = [2,4,6,8]
Higher Order Functions
• Apply to all:
Daniel Rubio BonillaTMPA 2017](https://image.slidesharecdn.com/18-00bonilacomplexity-170323081019/85/TMPA-2017-Using-Functional-Directives-to-Analyze-Code-Complexity-and-Communication-10-320.jpg)
![:: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 11
foldl :: (a -> b -> a) -> a -> [b] -> a
foldl (+) 0 [1,2,3,4] = 10
map :: (a -> b) -> [a] -> [b]
map (*2) [1,2,3,4] = [2,4,6,8]
Higher Order Functions
• Apply to all:
• Reduction:
Daniel Rubio BonillaTMPA 2017](https://image.slidesharecdn.com/18-00bonilacomplexity-170323081019/85/TMPA-2017-Using-Functional-Directives-to-Analyze-Code-Complexity-and-Communication-11-320.jpg)
![:: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 16
Flat Structure
Graph of a Complex Structure of two same level Higher
Order Functions (HOFs)
• The output of one HOF is the input for another HOF
foldl (+) 0 (map (*2) [0..n-1])
foldl :: (a -> b -> a) -> a -> [b] -> a
map :: (a -> b) -> [a] -> [b]
Daniel Rubio BonillaTMPA 2017](https://image.slidesharecdn.com/18-00bonilacomplexity-170323081019/85/TMPA-2017-Using-Functional-Directives-to-Analyze-Code-Complexity-and-Communication-16-320.jpg)
![:: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 17
Hierarchical Structure
Graph of a Complex Hierarchical Structure of two different
level Higher Order Functions (HOF)
• The operator of one HOF is another HOF
map (foldl (+) 0) [[..]..[..]]
foldl :: (a -> b -> a) -> a -> [b] -> a
map :: (a -> b) -> [a] -> [b]
Daniel Rubio BonillaTMPA 2017](https://image.slidesharecdn.com/18-00bonilacomplexity-170323081019/85/TMPA-2017-Using-Functional-Directives-to-Analyze-Code-Complexity-and-Communication-17-320.jpg)
![:: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 25
Transformed Code
if (rank < size - 1)
MPI_Send(&hm[LOCAL_N_ELEM],1, MPI_FLOAT, rank + 1, 0, MPI_COMM_WORLD);
if (rank > 0)
MPI_Recv(&hm[0], 1, MPI_FLOAT, rank-1, 0, MPI_COMM_WORLD, &status);
if (rank > 0)
MPI_Send(&hm[1], 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD );
if (rank < size - 1)
MPI_Recv(&hm[LOCAL_N_ELEM+1],1,MPI_FLOAT, rank+1, 1, MPI_COMM_WORLD,
&status);
#pragma polca stencil1D 1 G hm hm_tmp
#pragma omp parallel for
for(i=1; i<LOCAL_N_ELEM+1; i++)
{
#pragma polca G
#pragma polca input (hm[i-1] hm[i] hm[i+1]) output(hm_tmp[i])
hm_tmp[i] = hm[i] + K * (hm[i-1] + hm[i+1] - 2 * hm[i]);
}
Daniel Rubio BonillaTMPA 2017](https://image.slidesharecdn.com/18-00bonilacomplexity-170323081019/85/TMPA-2017-Using-Functional-Directives-to-Analyze-Code-Complexity-and-Communication-25-320.jpg)
TMPA-2017: Using Functional Directives to Analyze Code Complexity and Communication
- 1. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: Daniel Rubio BonillaTMPA 2017 Using Functional Directives to Analyze code Complexity and Communication Daniel Rubio Bonilla HLRS – University of Stuttgart
- 2. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 2 CPU Evolution Daniel Rubio BonillaTMPA 2017
- 3. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 3 Hazel Hen CPU E5-2680 v3 12 Cores 30MiB Cache 2.5 GhZ Node 2 CPUs – 24C 128 GB Comp. Nodes 7712 Total Cores 185,088 Performance 7420 TFlops Storage ~10 PB Weight 61.5 T Power 3200 KW Daniel Rubio BonillaTMPA 2017
- 4. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 4 Amdahl Law # processing units speedup 100% parallelizable 98% parallelizable 90% parallelizable 50% parallelizable Daniel Rubio BonillaTMPA 2017
- 5. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 5 Real Amdahl Law # processing units speedup 100% parallelizable 98% parallelizable 90% parallelizable 50% parallelizable Daniel Rubio BonillaTMPA 2017
- 6. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 6 The Problem Daniel Rubio Bonilla In High Performance Computing… • Performance is increased by • Integrating more cores (millions!?) • Using heterogeneous accelerators (GPU, FPGA, ...) • Issues • Programmability • Portability TMPA 2017
- 7. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 7 New Approaches Different Programming Model • Focused on mathematical problems • Engineering • Science • To enable: • Parallelization and concurrency • Portability across different hardware and accelerators Daniel Rubio BonillaTMPA 2017
- 8. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 8 Our Approach To obtain the structural information of the application by annotating the imperative code with a functional-like directives (mathematical / algorithmic structure) • The main difficulty in this approach are: • “deriving” the structure of the application • matching the structure to the source code Daniel Rubio BonillaTMPA 2017
- 9. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 9 Higher Order Functions • Higher Order functions are mathematical functions • Takes one or more function as an argument • Can return a function as a result • Clear repetitive execution structure • These structures can be transformed to equivalent ones • But with different non-functional properties Daniel Rubio BonillaTMPA 2017
- 10. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 10 map :: (a -> b) -> [a] -> [b] map (*2) [1,2,3,4] = [2,4,6,8] Higher Order Functions • Apply to all: Daniel Rubio BonillaTMPA 2017
- 11. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 11 foldl :: (a -> b -> a) -> a -> [b] -> a foldl (+) 0 [1,2,3,4] = 10 map :: (a -> b) -> [a] -> [b] map (*2) [1,2,3,4] = [2,4,6,8] Higher Order Functions • Apply to all: • Reduction: Daniel Rubio BonillaTMPA 2017
- 12. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 12 Other Higher Order Functions Daniel Rubio BonillaTMPA 2017
- 13. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 13 Higher Order Functions Transformations total = foldl (+) 0 vs One possible transformation Only if the operation is associative and we know its neutral element Daniel Rubio BonillaTMPA 2017
- 14. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 14 Transformations • Changes in the mathematical formulation • Or the algorithm execution • Produce equivalent code • Change computing load • Change memory distribution • Modify communication • Allow adaptation to different architectures • While maintaining correctness! Daniel Rubio BonillaTMPA 2017
- 15. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 15 Hierarchical Structures • Functional annotations allow the construction of multiple structural levels: • Emerging complexity of the structural information • We distinguish between: • Output of one Higher Order Function is input of another • This can be achieved by analyzing the data dependencies between the functions • The operator of one (Higher Order) Function is composed of other functions Daniel Rubio BonillaTMPA 2017
- 16. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 16 Flat Structure Graph of a Complex Structure of two same level Higher Order Functions (HOFs) • The output of one HOF is the input for another HOF foldl (+) 0 (map (*2) [0..n-1]) foldl :: (a -> b -> a) -> a -> [b] -> a map :: (a -> b) -> [a] -> [b] Daniel Rubio BonillaTMPA 2017
- 17. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 17 Hierarchical Structure Graph of a Complex Hierarchical Structure of two different level Higher Order Functions (HOF) • The operator of one HOF is another HOF map (foldl (+) 0) [[..]..[..]] foldl :: (a -> b -> a) -> a -> [b] -> a map :: (a -> b) -> [a] -> [b] Daniel Rubio BonillaTMPA 2017
- 18. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 18 Other requirements • Strong binding between directives and code • Description of memory organization Daniel Rubio BonillaTMPA 2017
- 19. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 19 Example - Heat t Daniel Rubio Bonilla 1-D heat dissipation function Discretization TMPA 2017
- 20. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 20 Complexity Daniel Rubio BonillaTMPA 2017 O(N_ELEM) O(N_ITER) O(N_ELEM) O(1) O(1) O(1)
- 21. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 21 Complexity Daniel Rubio BonillaTMPA 2017 O(N_ELEM) O(N_ITER) O(N_ELEM) O(1) O(1) O(1) O(1) + O(1) + O(N_ITER) * (O(N_ELEM)*O(1) + O(N_ELEM)) O(N_ITER * N_ELEM)
- 22. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 22 Transformations – Partitioning 1 let heatDiffusion = itn HEATTIMESTEP hm_array N_ITER PAR1 v = stencil1D TKernel 1 v where TKernel x y z = y + K * (x - 2*y + z) HEATTIMESTEP vs = map PAR1 vs Daniel Rubio BonillaTMPA 2017
- 23. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 23 Transformations – Partitioning 2 let heatDiffusion = itn HEATTIMESTEP hm_array N_ITER PAR2 v = stencil1D TKernel 1 v where TKernel x y z = y + K * (x - 2*y + z) PAR1 vs = map PAR2 vs HEATTIMESTEP vss = map PAR1 vss Daniel Rubio BonillaTMPA 2017
- 24. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 24 Platform Specific Transformations • OpenMP: • Relatively straightforward • MPI: • Communication • Halos Daniel Rubio BonillaTMPA 2017
- 25. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 25 Transformed Code if (rank < size - 1) MPI_Send(&hm[LOCAL_N_ELEM],1, MPI_FLOAT, rank + 1, 0, MPI_COMM_WORLD); if (rank > 0) MPI_Recv(&hm[0], 1, MPI_FLOAT, rank-1, 0, MPI_COMM_WORLD, &status); if (rank > 0) MPI_Send(&hm[1], 1, MPI_FLOAT, rank-1, 1, MPI_COMM_WORLD ); if (rank < size - 1) MPI_Recv(&hm[LOCAL_N_ELEM+1],1,MPI_FLOAT, rank+1, 1, MPI_COMM_WORLD, &status); #pragma polca stencil1D 1 G hm hm_tmp #pragma omp parallel for for(i=1; i<LOCAL_N_ELEM+1; i++) { #pragma polca G #pragma polca input (hm[i-1] hm[i] hm[i+1]) output(hm_tmp[i]) hm_tmp[i] = hm[i] + K * (hm[i-1] + hm[i+1] - 2 * hm[i]); } Daniel Rubio BonillaTMPA 2017
- 26. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 26 Example - NBody t Daniel Rubio Bonilla N-Body Problem TMPA 2017 Three steps 1) Calculate Forces 2) Update Velocities 3) Calculate Position
- 27. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 27 Structure Daniel Rubio BonillaTMPA 2017 O(1) O(1) O(1) O(nIters) O(nBodies) O(nBodies) O(nBodies) O(nBodies)
- 28. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 28 Structure Daniel Rubio BonillaTMPA 2017 O(nIters) O(nBodies2 ) O(nBodies) O(nBodies)
- 29. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 29 Structure Daniel Rubio BonillaTMPA 2017 O(nIters) * (O(nBodies2 ) + 2*O(nBodies)) O(nIters * nBodies2 ) O(nBodies2 ) O(nBodies) O(nBodies)
- 30. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 30 Communication Daniel Rubio BonillaTMPA 2017 Parallel Parallel Sequential Parallel (with caution)
- 31. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 31 Conclusion • Functional semantics can enable code: • Transformation • Adaptation • But also... • Algorithmic complexity analysis • Communication patterns • This information helps to predict application’s behavior Daniel Rubio BonillaTMPA 2017
- 32. :: :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 32 Questions Thank you! Contact: [email protected] Projects: POLCA www.polca-project.eu Smart-Dash www.dash-project.org CλaSH www.clash-lang.org Daniel Rubio BonillaTMPA 2017