Parallel programming concept dependency and loop parallelization

PARALLEL PROGRAMMING CONCEPT
DEPENDENCY AND LOOP
PARALLELIZATION
•Parallel programming concept and
their examples
•Dependency and their two types
with examples
•Loop parallelism and their types
with examples
R.AISHWARYA

• Simultaneous use of multiple compute
resources to solve a computational problem
• Compute resources
• Primary reasons
• Best practices
• Goals
• Steps

Statement 2: b=a+2; “5” Statement 1: a=5;

•Find dependencies within iterations of a loop
•Goal of determining different relationships
between statements.
•To allow multiple processors to work on different
portions of the loop in parallel
•First analyze the dependencies within individual
loops.
•It help determine which statements in the loop
need to be completed before other statements
can start.
•Two general categories of dependencies: Data and
Control dependency

function Dep(a,b)
c:=a.b
d:=2.c
end function
(flow dependency)
function Nodep(a,b)
c:=a.b
d:=2.b
e:=a+b
end function
(no dependency)

DATA DEPENDENCY
TYPE NOTATION DESCRIPTION
True (Flow) Dependence
S1 ->T S2
A true dependence between S1
and S2 means that S1 writes
to a location later read from
by S2
Anti Dependence S1 ->A S2
An anti-dependence between
S1 and S2 means that S1 reads
from a location later written
to by S2.(before)
Output Dependence S1 ->I S2
An input dependence between
S1 and S2 means that S1 and
S2 read from the same
location.

EXAMPLES
True dependence
S0: int a, b;
S1: a = 2;
S2: b = a + 40;
S1 ->T S2, meaning that S1 has a true dependence on S2 because
S1writes to the variable a, which S2 reads from.
Anti-dependence
S0: int a, b = 40;
S1: a = b - 38;
S2: b = -1;
S1 ->A S2, meaning that S1 has an anti-dependence on S2
because S1reads from the variable b before S2 writes to it.
Output-dependence
S0: int a, b = 40;
S1: a = b - 38;
S2: a = 2;
S1 ->O S2, meaning that S1 has an output dependence on S2
because both write to the variable a.

CONTROL DEPENDENCY
if(a == b)
then
{
c = “controlled”;
}
d=“not
controlled”;
if(a == b)
then
{
}
d=“not
controlled”;
if(a == b)
then
{
d=“not
controlled”;
}

DEPENDENCY IN LOOP
Loops can have two types of dependence:
•Loop-carried
dependency
•Loop-independent
dependency

LOOP CARRIED DEPENDENCY
• In loop-carried dependence, statements in an
iteration of a loop depend on statements in
another iteration of the loop.
for(i=0;i<4;i++)
{
S1: b[i]=8;
S2: a[i]=b[i-1] + 10;
}

LOOP INDEPENDENT DEPENDENCY
• In loop-independent dependence, loops have
inter-iteration dependence, but do not have
dependence between iterations.
• Each iteration may be treated as a block and
performed in parallel without other
synchronization efforts.
for (i=0;i<4;i++)
{
S1: b[i] = 8;
S2: a[i] =b[i] + 10;
}

for (i=1; i<4; i++)
for (j=1; j<4; j++)
S3: a[i][j] = a[i][j-1] + 1;
Node : Point in the iteration space
Directed Edge: Dependency
Node: Point in the iteration space
Directed Edge: next point that will
be encountered
after the current point is
traversed

•Extraction parallel tasks from loops
•Data is stored in random access data structures
•A program exploiting loop-level parallelism will use
multiple threads or processes which operate on same time
•It provides speedup
•Amdhal’s law

Examples of Loop
parallelization
for (int i = 0; i < n; i++)
{
S1: L[i] = L[i] + 10;
}
for (int i = 1; i < n; i++)
{
S1: L[i] = L[i - 1] + 10;
}

Can the following Loop be made
Parallel?
for (i=1;i<=100;i=i+1)
{
A[i+1] = A[i] + C[i]; /*S1*/
B[i+1] = B[i] + A[i+1]; /*S2*/
}

METHODOLOGIES FOR
PARALLELIZING LOOPS
• DISTRIBUTED Loop
• DOALL Parallelism
• DOACROSS Parallelism
• HELIX
• DOPIPE Parallelism

DISTRIBUTED LOOP
for (int i = 1; i < n; i ++)
{
S1: a[i] = a[i -1] + b[i];
S2: c[i] = c[i] + d[i];
}
loop1: for (int i = 1; i < n; i ++)
{
S1: a[i] = a[i -1] + b[i];
}
loop2: for (int i = 1; i < n; i ++)
{
S2: c[i] = c[i] + d[i];
}

DO ALL PARALLELISM
for (int i = 0; i < n; i++)
{
S1: a[i] = b[i] + c[i];
}
begin_parallelism();
for (int i = 0; i < n; i++)
{
S1: a[i] = b[i] + c[i];
end_parallelism();
}
block();

DO ACROSS PARALLELISM
for (int i = 1; i < n; i++)
{
a[i] = a[i - 1] + b[i] + 1;
}
S1: int tmp = b[i] + 1;
S2: a[i] = a[i - 1] + tmp;
post(0);
for (int i = 1; i < n; i++)
{
S1: int tmp = b[i] + 1;
wait(i - 1);
S2: a[i] = a[i - 1] + tmp;
post(i);
}

DO PIPE PARALLELISM
for (int i = 1; i < n; i++)
{
S1: a[i] = a[i - 1] + b[i];
S2: c[i] = c[i] + a[i];
}
for (int i = 1; i < n; i++)
{
S1: a[i] = a[i - 1] + b[i];
post(i);
}
for (int i = 1; i < n; i++)
{
wait(i);
S2: c[i] = c[i] + a[i];
}

Parallel programming concept dependency and loop parallelization

Recommended

More Related Content

What's hot (20)

Similar to Parallel programming concept dependency and loop parallelization (20)

Recently uploaded (20)

Parallel programming concept dependency and loop parallelization