Google: Cluster computing and MapReduce: Introduction to Distributed System Design

Distributed Computing Seminar

Lecture 1: Introduction to Distributed
Computing & Systems Background

Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet
Summer 2007
Except where otherwise noted, the contents of this presentation are
© Copyright 2007 University of Washington and are licensed under
the Creative Commons Attribution 2.5 License.

Course Overview
 5 lectures
1 Introduction
 2 Technical Side: MapReduce & GFS
 2 Theoretical: Algorithms for distributed
computing
 Readings + Questions nightly
 Readings: http://code.google.com/edu/content/submissions/mapreduce-minilecture/listing.html
 Questions: http://code.google.com/edu/content/submissions/mapreduce-
minilecture/MapReduceMiniSeriesReadingQuestions.doc

Outline
 Introduction to Distributed Computing
 Parallel vs. Distributed Computing
 History of Distributed Computing
 Parallelization and Synchronization
 Networking Basics

Computer Speedup

Moore’s Law: “The density of transistors on a chip doubles every 18
months, for the same cost” (1965)
Image: Tom’s Hardware and not subject to the Creative
Commons license applicable to the rest of this work.

Scope of problems
 What can you do with 1 computer?
 What can you do with 100 computers?
 What can you do with an entire data
center?

Distributed problems
 Rendering multiple frames of high-quality
animation

Image: DreamWorks Animation and not subject to the Creative Commons license applicable to the rest of this work.

 Simulating several
hundred or thousand
characters

Happy Feet © Kingdom Feature Productions;
Lord of the Rings © New Line Cinema, neither image is subject to the Creative
Commons license applicable to the rest of the work.

 Indexing the web (Google)
 Simulating an Internet-sized network for
networking experiments (PlanetLab)
 Speeding up content delivery (Akamai)

What is the key attribute that all these examples have in common?

Parallel vs. Distributed
 Parallel computing can mean:
 Vector processing of data
 Multiple CPUs in a single computer
 Distributed computing is multiple CPUs
across many computers over the network

A Brief History… 1975-85
 Parallel computing was
favored in the early
years
 Primarily vector-based
at first
 Gradually more thread-
based parallelism was
introduced

Image: Computer Pictures Database and Cray Research Corp and is not subject to the Creative Commons license
applicable to the rest of this work.

A Brief History… 1985-95
 “Massively parallel architectures” start
rising in prominence
 Message Passing Interface (MPI) and
other libraries developed
 Bandwidth was a big problem

A Brief History… 1995-Today
 Cluster/grid architecture increasingly
dominant
 Special node machines eschewed in favor
of COTS technologies
 Web-wide cluster software
 Companies like Google take this to the
extreme

Parallelization &
Synchronization

Parallelization Idea
 Parallelization is “easy” if processing can be
cleanly split into n units:

work

Partition
problem

w1 w2 w3

Parallelization Idea (2)

w1 w2 w3

Spawn worker threads:

thread thread thread

In a parallel computation, we would like to have as
many threads as we have processors. e.g., a four-
processor computer would be able to run four threads
at the same time.


Workers process data:

w1 w2 w3


w1 w2 w3

Report
results

results

Parallelization Pitfalls
But this model is too simple!

 How do we assign work units to worker threads?
 What if we have more work units than threads?
 How do we aggregate the results at the end?
 How do we know all the workers have finished?
 What if the work cannot be divided into
completely separate tasks?

What is the common theme of all of these problems?

Parallelization Pitfalls (2)
 Each of these problems represents a point
at which multiple threads must
communicate with one another, or access
a shared resource.

 Golden rule: Any memory that can be used
by multiple threads must have an
associated synchronization system!

What is Wrong With This?
Thread 1: Thread 2:
void foo() { void bar() {
x++; y++;
y = x; x+=3;
} }

If the initial state is y = 0, x = 6, what happens
after these threads finish running?

Multithreaded = Unpredictability
 Many things that look like “one step” operations
actually take several steps under the hood:
Thread 1: Thread 2:
eax = mem[x]; eax = mem[y];
inc eax; inc eax;
mem[x] = eax; mem[y] = eax;
ebx = mem[x]; eax = mem[x];
mem[y] = ebx; add eax, 3;
} mem[x] = eax;
}

 When we run a multithreaded program, we don’t
know what order threads run in, nor do we know
when they will interrupt one another.

Multithreaded = Unpredictability
This applies to more than just integers:

 Pulling work units from a queue
 Reporting work back to master unit
 Telling another thread that it can begin the
“next phase” of processing

… All require synchronization!

Synchronization Primitives
 A synchronization primitive is a special
shared variable that guarantees that it can
only be accessed atomically.

 Hardware support guarantees that
operations on synchronization primitives
only ever take one step

Semaphores
 A semaphore is a flag Set: Reset:

that can be raised or
lowered in one step
 Semaphores were
flags that railroad
engineers would use
when entering a
shared track

Only one side of the semaphore can ever be red! (Can both be
green?)

Semaphores
 set() and reset() can be thought of as
lock() and unlock()
 Calls to lock() when the semaphore is
already locked cause the thread to block.

 Pitfalls: Must “bind” semaphores to
particular objects; must remember to
unlock correctly

The “corrected” example
Thread 1: Thread 2:

sem.lock(); sem.lock();
x++; y++;
y = x; x+=3;
sem.unlock(); sem.unlock();
} }

Global var “Semaphore sem = new Semaphore();” guards access to
x&y

Condition Variables
 A condition variable notifies threads that a
particular condition has been met

 Inform another thread that a queue now
contains elements to pull from (or that it’s
empty – request more elements!)

 Pitfall: What if nobody’s listening?

The final example
Thread 1: Thread 2:

sem.lock(); sem.lock();
x++; if(!fooDone)
y = x; fooFinishedCV.wait(sem);
fooDone = true; y++;
sem.unlock(); x+=3;
fooFinishedCV.notify(); sem.unlock();
} }

Global vars: Semaphore sem = new Semaphore(); ConditionVar
fooFinishedCV = new ConditionVar(); boolean fooDone = false;

Too Much Synchronization?
Deadlock
Synchronization becomes even
more complicated when multiple
locks can be used

Can cause entire system to “get
stuck”
Thread A: Thread B:
semaphore1.lock(); semaphore2.lock();
semaphore2.lock(); semaphore1.lock();
/* use data guarded by /* use data guarded by
semaphores */ semaphores */
semaphore1.unlock(); semaphore1.unlock();
semaphore2.unlock(); semaphore2.unlock();
(Image: RPI CSCI.4210 Operating Systems notes)

The Moral: Be Careful!
 Synchronization is hard
 Need to consider all possible shared state
 Must keep locks organized and use them
consistently and correctly
 Knowing there are bugs may be tricky;
fixing them can be even worse!
 Keeping shared state to a minimum
reduces total system complexity

Sockets: The Internet = tubes?
 A socket is the basic network interface
 Provides a two-way “pipe” abstraction
between two applications
 Client creates a socket, and connects to
the server, who receives a socket
representing the other side

Ports
 Within an IP address, a port is a sub-address
identifying a listening program
 Allows multiple clients to connect to a server at
once

What makes this work?
 Underneath the socket layer are several more
protocols
 Most important are TCP and IP (which are used
hand-in-hand so often, they’re often spoken of as
one protocol: TCP/IP)

TCP
IP header Your data
header

Even more low-level protocols handle how data is sent over
Ethernet wires, or how bits are sent through the air using 802.11
wireless…

Why is This Necessary?
 Not actually tube-like “underneath the hood”
 Unlike phone system (circuit switched), the
packet switched Internet uses many routes at
once

you www.google.com

Networking Issues
 If a party to a socket disconnects, how
much data did they receive?
 … Did they crash? Or did a machine in the
middle?
 Can someone in the middle
intercept/modify our data?
 Traffic congestion makes switch/router
topology important for efficient throughput

Conclusions
 Processing more data means using more
machines at the same time
 Cooperation between processes requires
synchronization
 Designing real distributed systems requires
consideration of networking topology

 Next time: How MapReduce works

Google: Cluster computing and MapReduce: Introduction to Distributed System Design

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Google: Cluster computing and MapReduce: Introduction to Distributed System Design (20)

Recently uploaded (20)

Google: Cluster computing and MapReduce: Introduction to Distributed System Design