Introduction to Parallel and Distributed Computing

한국해양과학기술진흥원
Introduction to Parallel Computing
2013.10.6
Sayed Chhattan Shah, PhD
Senior Researcher
Electronics and Telecommunications Research Institute, Korea

2
Acknowledgements
Blaise Barney, Lawrence Livermore National Laboratory,
“Introduction to Parallel Computing”
Jim Demmel, Parallel Computing Lab at UC Berkeley, “
Parallel Computing”

Outline
 Introduction
 Parallel Computer Memory Architectures
 Shared Memory
 Distributed Memory
 Hybrid Distributed-Shared Memory
 Parallel Programming Models
 Shared Memory Model
 Threads Model
 Message Passing Model
 Data Parallel Model
 Design Parallel Programs

What is Parallel Computing?
 Traditionally, software has been written
for serial computation
 To be run on a single computer having a single Central
Processing Unit
 A problem is broken into a discrete series of instructions
 Instructions are executed one after another
 Only one instruction may execute at any moment in time

Parallel computing is the simultaneous use of multiple
compute resources to solve a computational problem
 Accomplished by breaking the problem into independent parts
so that each processing element can execute its part of the
algorithm simultaneously with the others

The computational problem should be:
 Solved in less time with multiple compute resources than with a
single compute resource
The compute resources might be
 A single computer with multiple processors
 Several networked computers
 A combination of both

LLN Lab Parallel Computer:
 Each compute node is a multi-processor parallel computer
 Multiple compute nodes are networked together with an
InfiniBand network

The Real World is Massively Parallel
 In the natural world, many complex, interrelated events are
happening at the same time, yet within a temporal sequence
 Compared to serial computing, parallel computing is much
better suited for modeling, simulating and understanding
complex, real world phenomena

Uses for Parallel Computing
 Science and Engineering
 Historically, parallel computing has been used to model difficult
problems in many areas of science and engineering
• Atmosphere, Earth, Environment, Physics, Bioscience, Chemistry, Mechanical
Engineering, Electrical Engineering, Circuit Design, Microelectronics, Defense,
Weapons

Uses for Parallel Computing
 Industrial and Commercial
 Today, commercial applications provide an equal or greater
driving force in the development of faster computers
• Data mining, Oil exploration, Web search engines, Medical imaging and
diagnosis, Pharmaceutical design, Financial and economic modeling, Advanced
graphics and virtual reality, Collaborative work environments

Why Use Parallel Computing?
 Save time and money
 In theory, throwing more resources at a task will shorten its
time to completion, with potential cost savings
 Parallel computers can be built from cheap, commodity
components

Solve larger problems
 Many problems are so large and complex that it is impractical
or impossible to solve them on a single computer, especially
given limited computer memory
• en.wikipedia.org/wiki/Grand_Challenge problems requiring
PetaFLOPS and PetaBytes of computing resources.
 Web search engines and databases processing millions of
transactions per second

Provide concurrency
 A single compute resource can only do one thing at a time.
Multiple computing resources can be doing many things
simultaneously
• For example, the Access Grid (www.accessgrid.org) provides a
global collaboration network where people from around the world
can meet and conduct work "virtually”

Use of non-local resources
 Using compute resources on a wide area network, or even the
Internet when local compute resources are insufficient
• SETI@home (setiathome.berkeley.edu) over 1.3 million users, 3.4
million computers in nearly every country in the world.
Source: www.boincsynergy.com/stats/ (June, 2013).
• Folding@home (folding.stanford.edu) uses over 320,000 computers
globally (June, 2013)

Limits to serial computing
 Transmission speeds
• The speed of a serial computer is directly dependent upon how fast
data can move through hardware.
• Absolute limits are the speed of light and the transmission limit of
copper wire
• Increasing speeds necessitate increasing proximity of processing
elements
 Limits to miniaturization
• Processor technology is allowing an increasing number of
transistors to be placed on a chip
• However, even with molecular or atomic-level components, a limit
will be reached on how small components can be

Limits to serial computing
 Economic limitations
• It is increasingly expensive to make a single processor faster
• Using a larger number of moderately fast commodity processors to
achieve the same or better performance is less expensive
 Current computer architectures are increasingly relying upon
hardware level parallelism to improve performance
• Multiple execution units
• Pipelined instructions
• Multi-core

The Future
 Trends indicated by ever faster networks, distributed
systems, and multi-processor computer architectures
clearly show that parallelism is the future of computing

The Future
 There has been a greater than 1000x increase in
supercomputer performance, with no end currently in sight
 The race is already on for Exascale Computing!
 1 exaFlops = 1018, floating point operations per second

Who is Using Parallel Computing?

von Neumann Architecture
Named after the Hungarian mathematician John von
Neumann who first authored the general requirements
for an electronic computer in his 1945 papers
Since then, virtually all computers have followed this
basic design
Comprises of four main components:
 RAM is used to store both program instructions and data
 Control unit fetches instructions or data from memory,
Decodes instructions and then sequentially coordinates operations
to accomplish the programmed task.
 Arithmetic Unit performs basic arithmetic operations
 Input-Output is the interface to the human operator

Flynn's Classical Taxonomy
Different ways to classify parallel computers
 Examples available HERE
Widely used classifications is called Flynn's Taxonomy
 Based upon the number of concurrent instruction and data
streams available in the architecture

Single Instruction, Single Data
 A sequential computer which exploits no parallelism in either the
instruction or data streams
• Single Instruction: Only one instruction stream is being acted on by the
CPU during any one clock cycle
• Single Data: Only one data stream is being used as input during any
one clock cycle
• Can have concurrent processing characteristics:
Pipelined execution

Older generation mainframes, minicomputers, workstations, modern
day PCs

Single Instruction, Multiple Data
 A computer which exploits multiple data streams against a single
instruction stream to perform operations
• Single Instruction: All processing units execute the same instruction at
any given clock cycle
• Multiple Data: Each processing unit can operate on a different data
element

Single Instruction, Multiple Data
 Vector Processor implements a instruction set containing
instructions that operates on 1D arrays of data called vectors

Multiple Instruction, Single Data
 Multiple Instruction: Each processing unit operates on the data
independently via separate instruction streams
 Single Data: A single data stream is fed into multiple processing
units
 Some conceivable uses might be:
• Multiple cryptography algorithms attempting to crack a single coded
message

Multiple Instruction, Multiple Data
 Multiple autonomous processors simultaneously executing
different instructions on different data
• Multiple Instruction: Every processor may be executing a different
instruction stream
• Multiple Data: Every processor may be working with a different data
stream

Most current supercomputers, networked parallel computer
clusters, grids, clouds, multi-core PCs
Many MIMD architectures also include SIMD execution sub-
components

Some General Parallel Terminology
Parallel computing
 Using parallel computer to solve single problems faster
Parallel computer
 Multiple-processor or multi-core system supporting parallel
programming
Parallel programming
 Programming in a language that supports concurrency
explicitly
Supercomputing or High Performance Computing
 Using the world's fastest and largest computers to solve large
problems

Task
 A logically discrete section of computational work.
 A task is typically a program or program-like set of instructions that is
executed by a processor
 A parallel program consists of multiple tasks running on multiple
processors
Shared Memory
 Hardware point of view, describes a computer architecture where all
processors have direct access to common physical memory
 In a programming sense, it describes a model where all parallel tasks
have the same "picture" of memory and can directly address and
access the same logical memory locations regardless of where the
physical memory actually exists

Symmetric Multi-Processor (SMP)
 Hardware architecture where multiple processors share a
single address space and access to all resources; shared
memory computing
Distributed Memory
 In hardware, refers to network based memory access for
physical memory that is not common
 As a programming model, tasks can only logically "see" local
machine memory and must use communications to access
memory on other machines where other tasks are executing

Communications
 Parallel tasks typically need to exchange data. There are several ways this can
be accomplished, such as through a shared memory bus or over a network,
however the actual event of data exchange is commonly referred to as
communications regardless of the method employed
Synchronization
 Coordination of parallel tasks in real time, very often associated with
communications
 Often implemented by establishing a synchronization point within an
application where a task may not proceed further until another task(s) reaches
the same or logically equivalent point
 Synchronization usually involves waiting by at least one task, and can therefore
cause a parallel application's wall clock execution time to increase

Parallel Overhead
 The amount of time required to coordinate parallel tasks, as
opposed to doing useful work.
 Parallel overhead can include factors such as:
• Task start-up time
• Synchronizations
• Data communications
• Software overhead imposed by parallel languages, libraries,
operating system, etc.
• Task termination time
Massively Parallel
 Refers to the hardware that comprises a given parallel system - having many
processors.
 The meaning of "many" keeps increasing, but currently, the largest parallel
computers can be comprised of processors numbering in the hundreds of
thousands

Embarrassingly Parallel
 Solving many similar, but independent tasks simultaneously;
little to no need for coordination between the tasks
Scalability
 A parallel system's ability to demonstrate a proportionate
increase in parallel speedup with the addition of more
resources

Multi-core processor
 A single computing component with two or more independent
actual central processing units

Potential program speedup is defined by the fraction of
code (P) that can be parallelized
 If none of the code can be parallelized, P = 0 and the speedup =
1 (no speedup).
 If 50% of the code can be parallelized,
maximum speedup = 2, meaning the code will run twice as fast.
Limits and Costs of Parallel Programming

 Introducing the number of processors N performing the parallel
fraction of work P, the relationship can be modeled by:
 It soon becomes obvious that there are limits to the scalability of
parallelism. For example

Complexity
 Parallel applications are much more complex than
corresponding serial applications
• Not only do you have multiple instruction streams executing at the
same time, but you also have data flowing between them
 The costs of complexity are measured in programmer time in
virtually every aspect of the software development cycle:
• Design
• Coding
• Debugging
• Tuning
• Maintenance

Portability
 Portability issues with parallel programs are not as serious as
in years past due to standardization in several APIs, such as
MPI, POSIX threads, and OpenMP
 All of the usual portability issues associated with serial
programs apply to parallel programs
• Hardware architectures are characteristically highly variable and
can affect portability

Resource Requirements
 Amount of memory required can be greater for parallel codes
than serial codes, due to the need to replicate data and for
overheads associated with parallel support libraries and
subsystems
 For short running parallel programs, there can actually be a
decrease in performance compared to a similar serial
implementation
• The overhead costs associated with setting up the parallel
environment, task creation, communications and task termination
can comprise a significant portion of the total execution time for
short runs

Scalability
 Ability of a parallel program's performance to scale is a result of
a number of interrelated factors. Simply adding more
processors is rarely the answer
 Algorithm may have inherent limits to scalability
 Hardware factors play a significant role in scalability.
Examples:
• Memory-CPU bus bandwidth on an SMP machine
• Communications network bandwidth
• Amount of memory available on any given machine or set of
machines
• Processor clock speed

Parallel Computer Memory Architectures

All processors access all memory as global address
space
 Multiple processors can operate independently but share the
same memory resources
 Changes in a memory location effected by one processor are
visible to all other processors
Shared Memory

Shared memory machines are classified
as UMA and NUMA, based upon memory access times
 Uniform Memory Access (UMA)
• Most commonly represented today by Symmetric Multiprocessor
(SMP) machines
• Identical processors
• Equal access and access times to memory
• Sometimes called CC-UMA - Cache Coherent UMA
 Cache coherent means if one processor updates a location in shared
memory, all the other processors know about the update. Cache
coherency is accomplished at the hardware level
Shared Memory

 Non-Uniform Memory Access (NUMA)
• Often made by physically linking two or more SMPs
• One SMP can directly access memory of another SMP
• Not all processors have equal access time to all memories
• Memory access across link is slower
• If cache coherency is maintained, then may also be called CC-
NUMA - Cache Coherent NUMA
Shared Memory

Advantages
 Global address space provides a user-friendly programming
perspective to memory
 Data sharing between tasks is both fast and uniform due to the
proximity of memory to CPUs
Disadvantages
 Primary disadvantage is the lack of scalability between memory
and CPUs
• Adding more CPUs can geometrically increases traffic on the
shared memory-CPU path, and for cache coherent systems,
geometrically increase traffic associated with cache/memory
management
 Programmer responsibility for synchronization constructs that
ensure "correct" access of global memory
Shared Memory

Processors have their own local memory
 Changes to processor’s local memory have no effect on the
memory of other processors
 When a processor needs access to data in another processor,
it is usually the task of the programmer to explicitly define how
and when data is communicated
 Synchronization between tasks is likewise the programmer's
responsibility
 The network "fabric" used for data transfer varies widely,
though it can be as simple as Ethernet
Distributed Memory

Advantages
 Memory is scalable with the number of processors.
• Increase the number of processors and the size of memory
increases proportionately
 Each processor can rapidly access its own memory without
interference and without the overhead incurred with trying to
maintain global cache coherency.
 Cost effectiveness: can use commodity, off-the-shelf
processors and networking
Disadvantages
 The programmer is responsible for many of the details
associated with data communication between processors.
 Non-uniform memory access times - data residing on a remote
node takes longer to access than node local data.
Distributed Memory

Hybrid Distributed-Shared Memory
The largest and fastest computers in the world today
employ both shared and distributed memory
architectures
 The shared memory component can be a shared memory
machine or graphics processing units
 The distributed memory component is the networking of
multiple shared memory or GPU machines

Advantages and Disadvantages
 Whatever is common to both shared and distributed memory
architectures
 Increased scalability is an important advantage
 Increased programmer complexity is an important
disadvantage
Hybrid Distributed-Shared Memory

Parallel Programming Model
Programming model provides an abstract view of
computing system
 Abstraction above hardware and memory architectures
 Value of a programming model is usually judged on its
generality
• how well a range of different problems can be expressed and
• how well they execute on a range of different architectures
 The implementation of a programming model can take several
forms such as libraries invoked from traditional sequential
languages, language extensions, or complete new execution
models

Parallel programming models in common use:
 Shared Memory (without threads)
 Threads
 Distributed Memory / Message Passing
 Data Parallel
 Hybrid
 Single Program Multiple Data (SPMD)
 Multiple Program Multiple Data (MPMD)
These models are NOT specific to a particular type of
machine or memory architecture
 Any of these models can be implemented on any underlying
hardware

SHARED memory model on a DISTRIBUTED memory
machine
 Machine memory was physically distributed across networked
machines, but appeared to the user as a single shared memory
(global address space).
 This approach is referred to as virtual shared memory
DISTRIBUTED memory model on a SHARED memory
machine
 The SGI Origin 2000 employed the CC-NUMA type of shared
memory architecture, where every task has direct access to
global address space spread across all machines.
 However, the ability to send and receive messages using MPI, as
is commonly done over a network of distributed memory
machines, was implemented and commonly used

Shared Memory Model - Without Threads
Tasks share a common address space
 Efficient means of passing data between programs
 Various mechanisms such as locks / semaphores may be used
to control access to the shared memory
 Programmer's point of view
• The notion of data "ownership" is lacking, so there is no need to
specify explicitly the communication of data between tasks
• Program development can often be simplified
Disadvantage in terms of performance
 It becomes more difficult to understand and manage data locality:
• Keeping data local to the processor that works on it conserves
memory accesses, cache refreshes and bus traffic that occurs when
multiple processors use the same data

Shared Memory Model - Without Threads
Implementations
 Native compilers or hardware translate user program variables
into actual memory addresses, which are global
• On stand-alone shared memory machines, this is straightforward.
• On distributed shared memory machines, memory is physically
distributed across a network of machines, but made global through
specialized hardware and software

Threads Model
Type of shared memory programming model
 A single "heavy weight" process can have multiple "light weight",
concurrent execution paths
 Main program a.out is scheduled by native OS
 a.out loads and acquires all of the necessary system and user
resources to run. This is the "heavy weight" process
 a.out performs some serial work, and then creates a number of
tasks (threads) that can be scheduled and run by the operating
system concurrently

Threads Model
 Each thread has local data, but also, shares the entire resources
of a.out
 This saves the overhead associated with replicating a program's
resources for each thread ("light weight"). Each thread also
benefits from a global memory view because it shares the
memory space of a.out
 Threads communicate with each other through global memory
(updating address locations). This requires synchronization
constructs to ensure that more than one thread is not updating
the same global address at any time
 Threads can come and go, but a.out remains present to provide
the necessary shared resources until the application has
completed

Threads Model
Implementations
• POSIX Threads
 Library based; requires parallel coding
 C Language only
 Commonly referred to as Pthreads.
 Most hardware vendors now offer Pthreads in addition to their proprietary
threads implementations.
 Very explicit parallelism; requires significant programmer attention to detail.
• OpenMP
 Compiler directive based; can use serial code
 Portable / multi-platform, including Unix and Windows platforms
 Available in C/C++ and Fortran implementations
 Can be very easy and simple to use

Distributed Memory / Message Passing Model
A set of tasks that use their own local memory during
computation
 Multiple tasks can reside on the same physical machine and/or
across an arbitrary number of machines.
 Tasks exchange data through communications by sending and
receiving messages.
 Data transfer usually requires cooperative operations to be
performed by each process. For example, a send operation must
have a matching receive operation.

Distributed Memory / Message Passing Model
Implementations
 From a programming perspective, message passing
implementations usually comprise a library of subroutines
 Calls to these subroutines are imbedded in source code
 MPI specifications are available on the web at http://www.mpi-
forum.org/docs/
 MPI implementations exist for virtually all popular parallel
computing platforms

Address space is treated globally
A set of tasks work collectively on the same data
structure, however, each task works on a different partition
of the same data structure
 On shared memory architectures, all tasks may have access to
the data structure through global memory
 On distributed memory architectures the data structure is split up
and resides as "chunks" in the local memory of each task
Data Parallel Model

Implementations
 Unified Parallel C (UPC): an extension to the C programming
language for SPMD parallel programming. Compiler dependent.
More information: http://upc.lbl.gov/
 Global Arrays: provides a shared memory style programming
environment in the context of distributed array data structures.
Public domain library with C and Fortran77 bindings. More
information: http://www.emsl.pnl.gov/docs/global/
 X10: a PGAS based parallel programming language being
developed by IBM at the Thomas J. Watson Research Center.
More information: http://x10-lang.org/
Data Parallel Model

Hybrid Model
A hybrid model combines more than one of the previously
described programming models

Single Program Multiple Data
High level programming model that can be built upon any
combination of the previously mentioned parallel
programming models
SINGLE PROGRAM: All tasks execute their copy of the
same program simultaneously.
 This program can be threads, message passing, data parallel or
hybrid.
MULTIPLE DATA: All tasks may use different data

Multiple Program Multiple Data
High level programming model that can be built upon any
combination of the previously mentioned parallel
programming models
MULTIPLE PROGRAM: Tasks may execute different
programs simultaneously. The programs can be threads,
message passing, data parallel or hybrid.
MULTIPLE DATA: All tasks may use different data

Determine whether the problem can be parallelized
Calculate the potential energy for each of several
thousand independent conformations of a molecule. When
done, find the minimum energy conformation
 This problem can be solved in parallel.
• Each of the molecular conformations is independently determinable
• The calculation of the minimum energy conformation is also a
parallelizable problem.
Understand the Problem and the Program

Calculation of the series (0,1,1,2,3,5,8,13,21,...) by use of
the formula: F(n) = F(n-1) + F(n-2)
 This problem can't be solved in parallel
• Calculation of the Fibonacci sequence as shown would entail
dependent calculations rather than independent ones.
• The calculation of the F(n) value uses those of both F(n-1) and F(n-2).
These three terms cannot be calculated independently and therefore,
not in parallel

Identify the program's hotspots
 Know where most of the real work is being done
• The majority of scientific and technical programs usually accomplish
most of their work in a few places
• Profilers and performance analysis tools can help here
 Focus on parallelizing the hotspots and ignore those sections of
the program that account for little CPU usage

Identify bottlenecks in the program
 Are there areas that are disproportionately slow, or cause
parallelizable work to halt or be deferred?
• For example, I/O is usually something that slows a program down
 May be possible to restructure the program or use a different
algorithm to reduce or eliminate unnecessary slow areas
Identify inhibitors to parallelism
 One common class of inhibitor is data dependence

Break the problem into discrete "chunks" of work that can
be distributed to multiple tasks
Two basic ways to partition computational work among
parallel tasks:
 Domain decomposition
• Data associated with a problem is decomposed
• Each parallel task then works on a portion of the data
Partitioning

Partitioning
 Functional decomposition
• Problem is decomposed according to the work
• Each task then performs a portion of the overall work
• Functional decomposition lends itself well to problems that can be
split into different tasks

Partitioning
Ecosystem Modeling
 Each program calculates the population of a given group
• Each group's growth depends on that of its neighbors
• As time progresses, each process calculates its current state, then
exchanges information with the neighbor populations
• All tasks then progress to calculate the state at the next time step

Partitioning
Climate Modeling
 Each model component can be thought of as a separate task
 Arrows represent exchanges of data between components during
computation:
• the atmosphere model generates wind velocity data that are used by
the ocean model, the ocean model generates sea surface
temperature data that are used by the atmosphere model, and so on

Communications
Some types of problems can be decomposed and
executed in parallel with virtually no need for tasks to
share data
 For example, imagine an image processing operation where
every pixel in a black and white image needs to have its color
reversed
 The image data can easily be distributed to multiple tasks that
then act independently of each other to do their portion of the
work
Most parallel applications require tasks to share data with
each other
 For example, a 3-D heat diffusion problem requires a task to
know the temperatures calculated by the tasks that have
neighboring data
 Changes to neighboring data has a direct effect on that task's
data

Communications
Factors to Consider when designing your program's inter-
task communications
 Cost of communications
• Inter-task communication virtually always implies overhead
• Communications frequently require some type of synchronization
between tasks
• Competing communication traffic can saturate the available network
bandwidth, further aggravating performance problems
 Latency vs. Bandwidth
• latency is the time it takes to send a message from point A to B
• bandwidth is the amount of data that can be communicated per unit of
time
• Sending many small messages can cause latency to dominate
communication overheads
• Often it is more efficient to package small messages into a larger
message, thus increasing the effective communications bandwidth

Communications
 Visibility of communications
• With the Message Passing Model, communications are explicit and
generally quite visible and under the control of the programmer
• With the Data Parallel Model, communications often occur
transparently to the programmer, particularly on distributed memory
architectures
• The programmer may not even be able to know exactly how inter-task
communications are being accomplished
 Synchronous vs. asynchronous communications
• Synchronous communications are often referred to
as blocking communications since other work must wait until the
communications have completed
• Asynchronous communications allow tasks to transfer data
independently from one another
• Interleaving computation with communication is the single greatest
benefit for using asynchronous communications

Communications
 Scope of communications
• Knowing which tasks must communicate with each other is critical
during the design stage of a parallel code
• Point-to-point - involves two tasks with one task acting as the
sender/producer of data, and the other acting as the
receiver/consumer
• Collective - involves data sharing between more than two tasks, which
are often specified as being members in a common group
• Some common variations

Communications
Efficiency of communications
 Which implementation for a given model should be used?
• Using the Message Passing Model as an example, one MPI
implementation may be faster on a given hardware platform than
another
• What type of communication operations should be used?
• asynchronous communication operations can improve overall
program performance

Communications
Overhead and Complexity

Synchronization
Types of Synchronization
 Barrier
• Each task performs its work until it reaches the barrier
• When the last task reaches the barrier, all tasks are synchronized
 Lock
• Typically used to protect access to global data or code section
• Only one task at a time may use the lock
 Synchronous communication operations
• When a task performs a communication operation, some form of
coordination is required with the other task participating in the
communication
• For example, before a task can perform a send operation, it must first
receive an acknowledgment from the receiving task that it is OK to send

 A dependence exists between program statements when the order of
statement execution affects the results of the program
 Dependencies are important to parallel programming because they
are one of the primary inhibitors to parallelism
 Loop carried data dependence: The value of A(J-1) must be computed before the value
of A(J), therefore A(J) exhibits a data dependency on A(J-1)
 How to Handle Data Dependencies
 Distributed memory architectures - communicate required data at synchronization
points
 Shared memory architectures -synchronize read-write operations between tasks
Data Dependencies

 Practice of distributing approximately equal amounts of work among
tasks
 Important to parallel programs for performance reasons
 For example, if all tasks are subject to a barrier synchronization point, the slowest
task will determine the overall performance.
Load Balancing

How to Achieve Load Balance
 Equally partition the work each task receives
• For array operations where each task performs similar work, evenly
distribute the data set among the tasks
• For loop iterations where the work done in each iteration is similar,
evenly distribute the iterations across the tasks
• If a heterogeneous mix of machines, use performance analysis tool to
detect any load imbalances and adjust work accordingly
 Use dynamic work assignment
• When the amount of work each task will perform is intentionally
variable, or is unable to be predicted, it may be helpful to use
a scheduler - task pool approach
 As each task finishes its work, it queues to get a new piece of work.
• It may become necessary to design an algorithm which detects and
handles load imbalances as they occur dynamically within the code
Load Balancing

 The number of tasks into which a problem is decomposed
determines granularity
 Fine-grain Parallelism – Decomposition into large no of tasks
• Individual tasks are relatively small in terms of code size and
execution time
• The data is transferred among processors frequently
• Facilitates load balancing
• Implies high communication overhead
 Coarse-grain Parallelism – Decomposition into small no of tasks
• Individual tasks are relatively small in terms of code size and
execution time
• Implies more opportunity for performance increase
• Harder to load balance efficiently
Granularity

 Array Processing
 Example demonstrates calculations on 2-dimensional array
elements, with the computation on each array element being
independent from other array elements
 Serial program calculates one element at a time in sequential
order
Parallel Examples : Array Processing

 Array Processing - Parallel Solution 1
 Independent calculation of array elements ensures there is no
need for communication between tasks
 Each task executes the portion of the loop corresponding to the
data it owns

 One Possible Solution
 Implement as a Single Program Multiple Data model
• Master process initializes array, sends info to worker processes and
receives results
• Worker process receives info, performs its share of computation and
sends results to master
• Static load balancing
• Each task has a fixed amount of work to do
• May be significant idle time for faster or more lightly loaded processors -
slowest tasks determines overall performance

Pool of Tasks Scheme
 Two processes are employed
• Master Process
• Holds pool of tasks for worker processes to do
• Sends worker a task when requested
• Collects results from workers
• Worker Process
• Gets task from master process
• Performs computation
• Sends results to master
• Worker processes do not know before runtime which portion of array they will
handle or how many tasks they will perform
• Dynamic load balancing occurs at run time
• the faster tasks will get more work to do

Multicore computing
 A Multicore processor is a processor that includes
multiple execution units ("cores") on the same chip
Symmetric multiprocessing
 A computer system with multiple identical processors that
share memory and connect via a bus than node local data.
Distributed System
 A software system in which components located on networked
computers communicate and coordinate their actions by
passing messages
 An important goal and challenge of distributed systems is
location transparency
Classes of Parallel Computers

Cluster computing
 A cluster is a group of loosely coupled computers that work
together closely, so that in some respects they can be
regarded as a single computer
Massive parallel processing
 A massively parallel processor (MPP) is a single computer with
many networked processors.
 MPPs have many of the same characteristics as clusters, but
MPPs have specialized interconnect networks
Grid computing
 Compared to clusters, grids tend to be more loosely coupled,
heterogeneous, and geographically dispersed

Exascale computing
 Computer system capable of at least one exaFlops
• 10^18 floating point operations per second
Volunteer Computing
 Computer owners donate their computing resources to one or
more projects
Peep-to-peer Network
 A type of decentralized and distributed network architecture in
which individual nodes in the network act as both suppliers and
consumers of resources

A collection of independent computers that appear to the
users of the system as a single computer
A collection of autonomous computers, connected
through a network and distribution middleware which
enables computers to coordinate their activities and to
share the resources of the system, so that users
perceive the system as a single, integrated computing
facility
Distributed Systems

Example - Internet

Example - ATM

Example - Mobile devices in a Distributed System

Why Distributed Systems?

Transparency
Scalability
Fault tolerance
Concurrency
Openness
These challenges can also be seen as the goals or
desired properties of a distributed system
Basic problems and challenges

 Concealment from the user and the application programmer
of the separation of the components of a distributed system
 Access Transparency - Local and remote resources are accessed in
same way
 Location Transparency - Users are unaware of the location of
resources
 Migration Transparency - Resources can migrate without name
change
 Replication Transparency - Users are unaware of the existence of
multiple copies of resources
 Failure Transparency - Users are unaware of the failure of individual
components
 Concurrency Transparency - Users are unaware of sharing resources
with others
Transparency

 Addition of users and resources without suffering a noticeable
loss of performance or increase in administrative complexity
 Adding users and resources causes a system to grow:
 Size - growth with regards to the number of users or resources
• System may become overloaded
• May increase administration cost
 Geography - growth with regards to geography or the distance
between nodes
• Greater communication delays
 Administration – increase in administrative cost
Scalability

Whether the system can be extended in various ways
without disrupting existing system and services
 Hardware extensions
• adding peripherals, memory, communication interfaces
 Software extensions
• Operating System features
• Communication protocols
Openness is supported by:
 Public interfaces
 Standardized communication protocols
Openness

In a single system several processes are interleaved
In distributed systems - there are many systems with one
or more processors
Many users simultaneously invoke commands or
applications, access and update shared data
 Mutual exclusion
 Synchronization
 No global clock
Concurrency

Hardware, software and networks fail
Distributed systems must maintain availability even at
low levels of hardware, software, network reliability
Fault tolerance is achieved by
 Recovery
 Redundancy
Issues
 Detecting failures
 Masking failures
 Recovery from failures
 Redundancy
Fault tolerance

Fault tolerance
Omission and Arbitrary Failures

Performance issues
 Responsiveness
 Throughput
 Load sharing, load balancing
 Quality of service
 Correctness
 Reliability, availability, fault tolerance
 Security
 Performance
 Adaptability
Design Requirements

Introduction to Parallel and Distributed Computing

Recommended

More Related Content

What's hot (20)

Similar to Introduction to Parallel and Distributed Computing (20)

More from Sayed Chhattan Shah (17)

Recently uploaded (20)

Introduction to Parallel and Distributed Computing