Lecture 4 principles of parallel algorithm design updated

Principles of Parallel
Algorithm Design

Introduction
• Algorithm development is a critical component of problem solving
using computers.
• A sequential algorithm is essentially a recipe or a sequence of basic
steps for solving a given problem using a serial computer.
• A parallel algorithm is a recipe that tells us how to solve a given
problem using multiple processors.
• Parallel algorithm involves more than just specifying the steps
• dimension of concurrency
• sets of steps that can be executed simultaneously

Introduction
In practice, specifying a nontrivial parallel algorithm may include some
or all of the following:
• Identifying portions of the work that can be performed concurrently.
• Mapping the concurrent pieces of work onto multiple processes
running in parallel.
• Distributing the input, output, and intermediate data associated with
the program.
• Managing accesses to data shared by multiple processors.
• Synchronizing the processors at various stages of the parallel program
executio

Introduction
Two key steps in the design of parallel algorithms
• Dividing a computation into smaller computations
• Assigning them to different processors for parallel execution
Explaining using two examples:
• Matrix-vector multiplication
• Database query processing

Decomposition, Tasks, and Dependency
Graphs
• The process of dividing a computation into smaller parts, some or all
of which may potentially be executed in parallel, is called
decomposition.
• Tasks are programmer-defined units of computation into which the
main computation is subdivided by means of decomposition.
• Tasks are programmer-defined units of computation into which the
main computation is subdivided by means of decomposition.
• The tasks into which a problem is decomposed may not all be of the
same size.

Graphs
Decomposition of dense matrix-vector multiplication into n tasks,
where n is the number of rows in the matrix.

Graphs
• Some tasks may use data produced by other tasks and thus may need
to wait for these tasks to finish execution.
• An abstraction used to express such dependencies among tasks and
their relative order of execution is known as a task dependency graph.
• A task-dependency graph is a Directed Acyclic Graph (DAG) in which
the nodes represent tasks and the directed edges indicate the
dependencies amongst them.
• Note that task-dependency graphs can be disconnected and the edge
set of a task-dependency graph can be empty. (Ex: matrix-vector
multiplication)

Graphs
• Ex:
MODEL="Civic" AND YEAR="2001" AND (COLOR="Green" OR COLOR="White") ?

Graphs
The different tables
and their
dependencies in a
query processing
operation.

Graphs
• An alternate
data-
dependency
graph for the
query
processing
operation.

Granularity, Concurrency, and
Task-Interaction
• The number and size of tasks into which a problem is decomposed
determines the granularity of the decomposition.
• A decomposition into a large number of small tasks is called fine-
grained and a decomposition into a small number of large tasks is
called coarse-grained.
fine-grained coarse-grained

Task-Interaction
• The maximum number of tasks that can be executed simultaneously in a
parallel program at any given time is known as its maximum degree of
concurrency.
• In most cases, the maximum degree of concurrency is less than the total
number of tasks due to dependencies among the tasks.
• In general, for tasked pendency graphs that are trees, the maximum degree
of concurrency is always equal to the number of leaves in the tree.
• A more useful indicator of a parallel program's performance is the average
degree of concurrency, which is the average number of tasks that can run
concurrently over the entire duration of execution of the program.
• Both the maximum and the average degrees of concurrency usually increase
as the granularity of tasks becomes smaller (finer). (Ex: matrix vector
multipliation)

Task-Interaction
• The degree of concurrency also depends on the shape of the task-
dependency graph and the same granularity, in general, does not
guarantee the same degree of concurrency.
The average degree of
concurrency = 2.33
The average degree of
concurrency=1.88

Task-Interaction
• A feature of a task-dependency graph that determines the average
degree of concurrency for a given granularity is its critical path.
• The longest directed path between any pair of start and finish nodes is
known as the critical path.
• The sum of the weights of nodes along this path is known as the critical
path length.
• The weight of a node is the size or the amount of work associated with
the corresponding task.
• Tℎ𝑒 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑑𝑒𝑔𝑟𝑒𝑒 𝑜𝑓 𝑐𝑜𝑛𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑦 =
𝑇ℎ𝑒 𝑡𝑜𝑡𝑎𝑙 𝑎𝑚𝑜𝑢𝑛𝑡 𝑜𝑓 𝑤𝑜𝑟𝑘
𝑇ℎ𝑒 𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙−𝑝𝑎𝑡ℎ 𝑙𝑒𝑛𝑔𝑡ℎ

Task-Interaction
Tℎ𝑒 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑑𝑒𝑔𝑟𝑒𝑒 𝑜𝑓 𝑐𝑜𝑛𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑦 =
𝑇ℎ𝑒 𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 − 𝑝𝑎𝑡ℎ 𝑙𝑒𝑛𝑔𝑡ℎ
=63/27
=2.33

Task-Interaction
Tℎ𝑒 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑑𝑒𝑔𝑟𝑒𝑒 𝑜𝑓 𝑐𝑜𝑛𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑦 =
𝑇ℎ𝑒 𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 − 𝑝𝑎𝑡ℎ 𝑙𝑒𝑛𝑔𝑡ℎ
= 64/34
=1.88

Task-Interaction
• Other than limited granularity and degree of concurrency, there is
another important practical factor that limits our ability to obtain
unbounded speedup (ratio of serial to parallel execution time) from
parallelization.
• This factor is the interaction among tasks running on different
physical processors.
• The pattern of interaction among tasks is captured by what is known
as a task-interaction graph.
• The nodes in a task-interaction graph represent tasks and the edges
connect tasks that interact with each other.

Task-Interaction
• Ex:
In addition to assigning the computation of the element y[i] of the output vector to Task i, we also make it the "owner"
of row A[i, *] of the matrix and the element b[i] of the input vector.

Processes and Mapping
• we will use the term process to refer to a processing or computing agent that
performs tasks.
• It is an abstract entity that uses the code and data corresponding to a task to
produce the output of that task within a finite amount of time after the task is
activated by the parallel program.
• During this time, in addition to performing computations, a process may
synchronize or communicate with other processes, if needed.
• The mechanism by which tasks are assigned to processes for execution is called
mapping.
• Mapping of tasks onto processes plays an important role in determining how
efficient the resulting parallel algorithm is.
• Even though the degree of concurrency is determined by the decomposition, it
is the mapping that determines how much of that concurrency is actually
utilized, and how efficiently.

Processes and Mapping
Mappings of the task graphs onto four processes.

Processes versus Processors
• In the context of parallel algorithm design, processes are logical
computing agents that perform tasks.
• Processors are the hardware units that physically perform
computations.
• Treating processes and processors separately is also useful when
designing parallel programs for hardware that supports multiple
programming paradigms.

Decomposition Techniques
• In this section, we describe some commonly used decomposition
techniques for achieving concurrency.
• A given decomposition is not always guaranteed to lead to the best
parallel algorithm for a given problem.
• The decomposition techniques described in this section often provide a
good starting point for many problems and one or a combination of
these techniques can be used to obtain effective decompositions for a
large variety of problems.
• Techniques:
• Recursive decomposition
• Data-decomposition
• Exploratory decomposition
• Speculative decomposition

Recursive Decomposition
• Recursive decomposition is a method for inducing concurrency in
problems that can be solved using the divide-and-conquer strategy.
• In this technique, a problem is solved by first dividing it into a set of
independent sub-problems.
• Each one of these sub-problems is solved by recursively applying a
similar division into smaller sub-problems followed by a combination
of their results.

• Ex: Quicksort
• The quicksort
task-
dependency
graph based on
recursive
decomposition
for sorting a
sequence of 12
numbers.
a pivot element x

• The task-
dependency
graph for finding
the minimum
number in the
sequence {4, 9,
1, 7, 8, 11, 2,
12}. Each node
in the tree
represents the
task of finding
the minimum of
a pair of
numbers.

Data Decomposition
• Data decomposition is a powerful and commonly used method for deriving
concurrency in algorithms that operate on large data structures.
• The decomposition of computations is done in two steps
1. The data on which the computations are performed is partitioned
2. Data partitioning is used to induce a partitioning of the computations into tasks
• The partitioning of data can be performed in many possible ways
• Partitioning Output Data
• Partitioning Input Data
• Partitioning both Input and Output Data
• Partitioning Intermediate Data

Data Decomposition
Partitioning Output Data
Ex: matrix multiplication
Partitioning of input and output matrices into 2 x 2 submatrices.
A decomposition of matrix multiplication into four tasks based on the partitioning of the
matrices above

Data Decomposition
Partitioning Output Data
• Data-decomposition is
distinct from the
decomposition of the
computation into tasks
• Ex: Two examples of
decomposition of matrix
multiplication into eight
tasks. (same data-
decomposition)

Data Decomposition
• Partitioning Output Data
Example 2: Computing item-set frequencies in a transaction database

Data Decomposition
Partitioning Input Data

Data Decomposition
Partitioning both Input and Output Data

Data Decomposition
Partitioning Intermediate
Data
• Partitioning intermediate
data can sometimes lead
to higher concurrency
than partitioning input or
output data.
• Ex: Multiplication of
matrices A and B with
partitioning of the three-
dimensional intermediate
matrix D.

Data Decomposition
Partitioning Intermediate Data
• A decomposition of matrix multiplication based on partitioning the
intermediate three-dimensional matrix
The task-dependency graph

Exploratory decomposition
• we partition the search space into smaller parts, and search each one
of these parts concurrently, until the desired solutions are found.
• Ex: A 15-puzzle problem

Exploratory decomposition
• Note that even though exploratory decomposition may appear similar
to data-decomposition it is fundamentally different in the following
way.
• The tasks induced by data-decomposition are performed in their entirety and
each task performs useful computations towards the solution of the problem.
• On the other hand, in exploratory decomposition, unfinished tasks can be
terminated as soon as an overall solution is found.
• The work performed by the parallel formulation can be either smaller or
greater than that performed by the serial algorithm.

Speculative Decomposition
• Another special purpose decomposition technique is called
speculative decomposition.
• In the case when only one of several functions is carried out
depending on a condition (think: a switch-statement), these
functions are turned into tasks and carried out before the condition is
even evaluated.
• As soon as the condition has been evaluated, only the results of one
task are used, all others are thrown away.
• This decomposition technique is quite wasteful on resources and
seldom used.

Hybrid Decompositions
Ex: Hybrid decomposition for finding the minimum of an array of size
16 using four tasks.

Characteristics of Tasks and
Interactions

Characteristics of Tasks and Interactions
• We shall discuss the various properties of tasks and inter-task
interactions that affect the choice of a good mapping.
• Characteristics of Tasks
• Task Generation
• Task Sizes
• Knowledge of Task Sizes
• Size of Data Associated with Tasks

Characteristics of Tasks
Task Generation :
• Static task generation : the scenario where all the tasks are known before the
algorithm starts execution.
• Data decomposition usually leads to static task generation.
(Ex: matrix multiplication)
• Recursive decomposition can also lead to a static task-dependency graph
Ex: the minimum of a list of numbers
• Dynamic task generation :
• The actual tasks and the task-dependency graph are not explicitly available a priori,
although the high level rules or guidelines governing task generation are known as a part
of the algorithm.
• Recursive decomposition can lead to dynamic task generation.
• Ex: the recursive decomposition in quicksort
Exploratory decomposition can be formulated to generate tasks either statically or dynamically.
Ex: the 15-puzzle problem

Task Sizes
• The size of a task is the relative amount of time required to complete
it.
• The complexity of mapping schemes often depends on whether the
tasks are uniform or non-uniform.
• Example:
• the decompositions for matrix multiplication  would be considered uniform
• the tasks in quicksort  are non-uniform

Knowledge of Task Sizes
• If the size of all the tasks is known, then this information can often be
used in mapping of tasks to processes.
• Ex1: The various decompositions for matrix multiplication discussed so far,
the computation time for each task is known before the parallel program
starts.
• Ex2: The size of a typical task in the 15-puzzle problem is unknown (we don’t
know how many moves lead to the solution)

Size of Data Associated with Tasks
• Another important characteristic of a task is the size of data
associated with it
• The data associated with a task must be available to the process
performing that task
• The size and the location of these data may determine the process
that can perform the task without incurring excessive data-movement
overheads

Characteristics of Inter-Task Interactions
• The types of inter-task interactions can be described along different
dimensions, each corresponding to a distinct characteristic of the
underlying computations.
• Static versus Dynamic
• Regular versus Irregular
• Read-only versus Read-Write
• One-way versus Two-way

Static versus Dynamic
• An interaction pattern is static if for each task, the interactions happen at
predetermined times, and if the set of tasks to interact with at these times
is known prior to the execution of the algorithm.
• An interaction pattern is dynamic if the timing of interactions or the set of
tasks to interact with cannot be determined prior to the execution of the
algorithm.
• Static interactions can be programmed easily in the message-passing
paradigm, but dynamic interactions are harder to program.
• Shared-address space programming can code both types of interactions
equally easily.
• static inter-task interactions  Ex: matrix multiplication
• Dynamic inter-task interactions  Ex: 15-puzzle problem

• Static versus Dynamic
• Regular versus Irregular
• Read-only versus Read-Write
• One-way versus Two-way

Regular versus Irregular
• Another way of classifying the interactions is based upon their spatial
structure.
• An interaction pattern is considered to be regular if it has some
structure that can be exploited for efficient implementation.
• An interaction pattern is called irregular if no such regular pattern
exists.
• Irregular and dynamic communications are harder to handle,
particularly in the message-passing programming paradigm.

Read-only versus Read-Write
• As the name suggests, in read-only interactions, tasks require only a
read-access to the data shared among many concurrent tasks. (Matrix
multiplication – reading shared A and B).
• In read-write interactions, multiple tasks need to read and write on
some shared data. (15-puzzle problem, managing shared queue for
heuristic values)

One-way versus Two-way
• In some interactions, the data or work needed by a task or a subset of
tasks is explicitly supplied by another task or subset of tasks.
• Such interactions are called two-way interactions and usually involve
predefined producer and consumer tasks.
• One-way interaction  only one of a pair of communicating tasks
initiates the interaction and completes it without interrupting the
other one.

Mapping Techniques for Load
Balancing

Mapping Techniques for Load Balancing
• In order to achieve a small execution time, the overheads of
executing the tasks in parallel must be minimized.
• For a given decomposition, there are two key sources of overhead.
• The time spent in inter-process interaction is one source of overhead.
• The time that some processes may spend being idle.
• Therefore, a good mapping of tasks onto processes must strive to
achieve the twin objectives of
• (1) reducing the amount of time processes spend in interacting with each
other, and
• (2) reducing the total amount of time some processes are idle while the
others are engaged in performing some tasks

• In this section, we will discuss various schemes for mapping tasks
onto processes with the primary view of balancing the task workload
of processes and minimizing their idle time.
• A good mapping must ensure that the computations and interactions
among processes at each stage of the execution of the parallel
algorithm are well balanced.

• Two mappings of a hypothetical decomposition with a
synchronization

Mapping techniques used in parallel algorithms can be broadly classified into
two categories
• Static
• Static mapping techniques distribute the tasks among processes prior to the execution
of the algorithm.
• For statically generated tasks, either static or dynamic mapping can be used.
• Algorithms that make use of static mapping are in general easier to design and
program.
• Dynamic
• Dynamic mapping techniques distribute the work among processes during the
execution of the algorithm.
• If tasks are generated dynamically, then they must be mapped dynamically too.
• Algorithms that require dynamic mapping are usually more complicated, particularly in
the message-passing programming paradigm.

More about static and dynamic mapping
• If task sizes are unknown, then a static mapping can potentially lead
to serious load-imbalances and dynamic mappings are usually more
effective.
• If the amount of data associated with tasks is large relative to the
computation, then a dynamic mapping may entail moving this data
among processes. The cost of this data movement may outweigh
some other advantages of dynamic mapping and may render a static
mapping more suitable.

Schemes for Static Mapping
• Mappings Based on Data Partitioning
• Array Distribution Schemes
• Block Distributions
• Cyclic and Block-Cyclic Distributions
• Randomized Block Distributions
• Graph Partitioning
• Mappings Based on Task Partitioning
• Hierarchical Mappings

Mappings Based on Data Partitioning
• The data-partitioning actually induces a decomposition, but the
partitioning or the decomposition is selected with the final mapping
in mind.
• Array Distribution Schemes
• We now study some commonly used techniques of distributing arrays or
matrices among processes.

Array Distribution Schemes: Block
Distributions
• Block distributions are some of the simplest ways to distribute an
array and assign uniform contiguous portions of the array to different
processes.
• Block distributions of arrays are particularly suitable when there is a
locality of interaction, i.e., computation of an element of an array
requires other nearby elements in the array.

Distributions
• Examples of one-dimensional partitioning of an array among eight
processes.

Distributions
• Examples of two-dimensional distributions of an array, (a) on a 4 x 4
process grid, and (b) on a 2 x 8 process grid.

Distributions
• Data sharing needed
for matrix
multiplication with
(a) one-dimensional
and (b) two-
dimensional
partitioning of the
output matrix.
Shaded portions of
the input matrices A
and B are required
by the process that
computes the
shaded portion of
the output matrix C.

Array Distribution Schemes: Block-cyclic
distribution
• The block-cyclic
distribution is a variation
of the block distribution
scheme that can be used
to alleviate the load-
imbalance and idling
problems.
• Examples of one- and
two-dimensional block-
cyclic distributions
among four processes.

Array Distribution Schemes: Randomized
Block Distributions
• Using the block-cyclic distribution shown in (b) to distribute the
computations performed in array (a) will lead to load imbalances.

Block Distributions
• A one-dimensional randomized block mapping of 12 blocks onto four
process

Block Distributions
• Using a two-dimensional random block distribution shown in (b) to
distribute the computations performed in array (a), as shown in (c).

Graph Partitioning
• There are many algorithms that operate on sparse data structures and for
which the pattern of interaction among data elements is data dependent
and highly irregular.
• Numerical simulations of physical phenomena provide a large source of
such type of computations.
• In these computations, the physical domain is discretized and represented
by a mesh of elements.
• The simulation of the physical phenomenon being modeled then involves
computing the values of certain physical quantities at each mesh point.
• The computation at a mesh point usually requires data corresponding to
that mesh point and to the points that are adjacent to it in the mesh

Graph Partitioning
A mesh used to model Lake Superior

Graph Partitioning
• A random distribution of the mesh elements to eight processes.
Each process will need to access a large set of points belonging to other processes
to complete computations for its assigned portion of the mesh

Graph Partitioning
• A distribution of the mesh elements to eight processes, by using a
graph-partitioning algorithm.

Mappings Based on Task Partitioning
• As a simple example of a mapping based on task partitioning,
consider a task-dependency graph that is a perfect binary tree.
• Such a task-dependency graph can occur in practical problems with
recursive decomposition, such as the decomposition for finding the
minimum of a list of numbers.
A mapping for
sparse matrix-vector
multiplication onto
three processes. The
list Ci contains the
indices of b that
Process i needs to
access from other
processes.

Mappings Based on Task Partitioning
• Reducing interaction overhead in sparse matrix-vector multiplication
by partitioning the task-interaction graph

Hierarchical Mappings
• An example of
hierarchical mapping
of a task-dependency
graph. Each node
represented by an
array is a supertask.
The partitioning of
the arrays represents
subtasks, which are
mapped onto eight
processes.

Schemes for Dynamic Mapping
The primary reason for using a dynamic mapping is balancing the
workload among processes, dynamic mapping is often referred to as
dynamic load-balancing.
Dynamic mapping techniques are usually classified as either
• Centralized Schemes or
• Distributed Schemes

Centralized Schemes
• All executable tasks are maintained in a common central data
structure or they are maintained by a special process or a subset of
processes.
• If a special process is designated to manage the pool of available
tasks, then it is often referred to as the master and the other
processes that depend on the master to obtain work are referred to
as slaves.

Distributed Schemes
• In a distributed dynamic load balancing scheme, the set of executable
tasks are distributed among processes which exchange tasks at run
time to balance work.
• Each process can send work to or receive work from any other
process.
• Some of the critical parameters of a distributed load balancing
scheme are as follows:
• How are the sending and receiving processes paired together?
• Is the work transfer initiated by the sender or the receiver?
• How much work is transferred in each exchange?
• When is the work transfer performed?

Lecture 4 principles of parallel algorithm design updated

More Related Content

What's hot (20)

Similar to Lecture 4 principles of parallel algorithm design updated (20)

More from Vajira Thambawita (20)

Recently uploaded (20)

Lecture 4 principles of parallel algorithm design updated