Analysis of algorithms description pdf yeah

A N A L Y T I C C O M B I N A T O R I C S
P A R T O N E
http://aofa.cs.princeton.edu
1. Analysis
of
Algorithms

P A R T O N E
OF
1. Analysis of Algorithms
•History and motivation
•A scientific approach
•Example: Quicksort
•Resources
1a.AofA.History

Why Analyze an Algorithm?
2. Predict performance, compare algorithms, tune parameters.
3
1. Classify problems and algorithms by difficulty.
3. Better understand and improve implementations and algorithms.
Intellectual challenge: AofA is even more interesting than programming!

Analysis of Algorithms (Babbage, 1860s)
4
“As soon as an Analytic Engine exists, it will necessarily guide the future course of the science.
Whenever any result is sought by its aid, the question will arise—By what course of
calculation can these results be arrived at by the machine in the shortest time?”
— Charles Babbage (1864)
Analytic Engine
how many times do you
have to turn the crank?

Analysis of Algorithms (Turing (!), 1940s)
5
“It is convenient to have a measure of the amount of work involved in a computing
process, even though it be a very crude one. We may count up the number of times
that various elementary operations are applied in the whole process . . .”
— Alan Turing (1947)
ROUNDING-OFF ERRORS IN MATRIX PROCESSES
By A. M. TURING
{National Physical Laboratory, Teddington, Middlesex)
[Received 4 November 1947]
SUMMARY
A number of methods of solving sets of linear equations and inverting matrices
are discussed. The theory of the rounding-off errors involved is investigated for
some of the methods. In all cases examined, including the well-known 'Gauss
elimination process', it is found that the errors are normally quite moderate: no
exponential build-up need occur.
Included amongst the methods considered is a generalization of Choleski's method
which appears to have advantages over other known methods both as regards
accuracy and convenience. This method may also be regarded as a rearrangement
of the elimination process.
THIS paper contains descriptions of a number of methods for solving sets
of linear simultaneous equations and for inverting matrices, but its main
concern is with the theoretical limits of accuracy that may be obtained in
the application of these methods, due to rounding-off errors.
The best known method for the solution of linear equations is Gauss's
elimination method. This is the method almost universally taught in
schools. It has, unfortunately, recently come into disrepute on the ground
qjmam.oxfordjournals.or
Downloaded
from

Analysis of Algorithms (Knuth, 1960s)
6
To analyze an algorithm:
•Develop a good implementation.
•Identify unknown quantities representing the basic operations.
•Determine the cost of each basic operation.
•Develop a realistic model for the input.
•Analyze the frequency of execution of the unknown quantities.
•Calculate the total running time:
DRAWBACKS:
Model may be unrealistic.
Too much detail in analysis.
BENEFITS:
Scientific foundation for AofA.
Can predict performance and compare algorithms.
X
MYLXLUJ`(X) JVZ[(X)
D. E. Knuth

To address Knuth drawbacks:
•Analyze worst-case cost
[takes model out of the picture].
•Use O-notation for upper bound
[takes detail out of analysis].
•Classify algorithms by these costs.
Theory of Algorithms (AHU, 1970s; CLR, present day)
7
DRAWBACK: Cannot use to predict performance or compare algorithms.
(An elementary fact that is often overlooked!)
BENEFIT: Enabled a new Age of Algorithm Design.
Aho, Hopcroft
and Ullman
Cormen, Leiserson,
Rivest, and Stein

Cannot use O- upper bounds to predict performance or compare algorithms.
Example: Two sorting algorithms
8
Quicksort
Worst-case number of compares: O(N2)
Classification O(N2)
Mergesort
Worst-case number of compares: N log N
Classification O(N log N)
Quicksort is twice as fast as Mergesort in practice and uses half the space
BUT
How do we know?
By analyzing both algorithms! (stay tuned)

Analytic combinatorics context
Drawbacks of Knuth approach:
• Model may be unrealistic.
• Too much detail in analysis.
Drawbacks of AHU/CLRS approach:
• Worst-case performance may not be relevant.
• Cannot use O- upper bounds to predict or compare.
Analytic combinatorics can provide:
• A calculus for developing models.
• General theorems that avoid detail in analysis.
AC Part I (this course):
• Underlying mathematics.
• Introduction to analytic combinatorics.
• Classical applications in AofA and combinatorics.
9
ALGORITHMS
ANALYSIS
OF
S E C O N D E D I T I O N
AN INTRODUCTION
TO THE
R O B E R T S E D G E W I C K
P H I L I P P E F L A J O L E T

P A R T O N E
OF
•Resources
1b.AofA.Scientific

“Big-Oh” notation for upper bounds
“Omega” notation for lower bounds
“Theta” notation for order of growth (“within a constant factor”)
Notation for theory of algorithms
N(5) = 6(M(5)) PMM |N(5)/M(5)| PZIVUKLKMYVTHIV]LHZ 5
N(5) = £(M(5)) PMM |N(5)/M(5)| PZIVUKLKMYVTILSV^HZ 5
12
N(5) = (M(5)) PMM N(5) = 6(M(5)) HUK N(5) = £(M(5))

O-notation considered dangerous
Not the scientific method: O-notation
O-notation is useful for many reasons, BUT
Common error: Thinking that O-notation is useful for predicting performance
13
Theorem: Running time is O(Nc) ✘
not at all useful for predicting performance
Hypothesis: Running time is ~aNc
Scientific method calls for tilde-notation.
✓
an effective path to predicting performance
How to predict performance (and to compare algorithms)?

Surely, we can do better
O-notation considered dangerous.
Cannot use it to predict performance.
RS (in a talk):
?? O(N log N) surely beats O(N2)
Q:
Not by the definition. O expresses upper bound.
RS:
So, use Theta.
Q:
A typical exchange in QA
Still (typically) bounding the worst case.
Is the input a worst case?
RS:
(whispers to colleague) I’d use the Θ(N log N)
algorithm, wouldn’t you?
Q:
14

Galactic algorithms
R.J. Lipton: A galactic algorithm is one that will never be used.
Why? Any effect would never be noticed in this galaxy.
Ex. Chazelle’s linear-time triangulation algorithm
• theoretical tour-de-force
• too complicated to implement
• cost of implementing would exceed savings in this galaxy, anyway
One blogger’s conservative estimate:
75% SODA, 95% STOC/FOCS are galactic
OK for basic research to drive agenda, BUT
Common error: Thinking that a galactic algorithm is useful in practice.
15

Surely, we can do better
Algorithm A is bad.
Google should be interested in my new Algorithm B.
TCS (in a talk):
What’s the matter with Algorithm A?
RS:
It is not optimal. It has an extra O(log log N) factor.
TCS:
But Algorithm B is very complicated, lg lg N is less
than 6 in this universe, and that is just an upper
bound. Algorithm A is certainly going to run 10 to
100 times faster in any conceivable real-world
situation. Why should Google care about Algorithm B?
RS:
Well, I like Algorithm B. I don’t care about Google.
TCS:
An actual exchange with a theoretical computer scientist:
16

Analysis of Algorithms (scientific approach)
Start with complete implementation suitable for application testing.
Analyze the algorithm by
• Identifying an abstract operation in the inner loop.
• Develop a realistic model for the input to the program.
• Analyze the frequency of execution CN of the op for input size N.
Hypothesize that the cost is ~aCN where a is a constant.
Validate the hypothesis by
• Developing generator for input according to model.
• Calculate a by running the program for large input.
• Run the program for larger inputs to check the analysis.
Validate the model by testing in application contexts.
Refine and repeat as necessary
17
Sedgewick and Wayne
Algorithms, 4th edition
Section 1.4

“Big-Oh” notation for upper bounds
“Omega” notation for lower bounds
“Theta” notation for order of growth (“within a constant factor”)
Notation (revisited)
N(5) = 6(M(5)) PMM |N(5)/M(5)| PZIVUKLKMYVTHIV]LHZ 5
N(5) = £(M(5)) PMM |N(5)/M(5)| PZIVUKLKMYVTILSV^HZ 5
18
N(5) = (M(5)) PMM N(5) = 6(M(5)) HUK N(5) = £(M(5))
for theory of algorithms
for analysis to predict performance
and to compare algorithms
“Tilde” notation for asymptotic equivalence
N(5) M(5) PMM |N(5)/M(5)| HZ 5

Components of algorithm analysis
*5 = 5 + +
R 5

5
(*R + *5 R )
Mathematical
•Develop mathematical model.
•Analyze algorithm within model.
Challenge: need good model, need to do the math
% java QuickCheck 1000000
10 44.44 26.05
100 847.85 721.03
1000 12985.91 11815.51
10000 175771.70 164206.81
100000 2218053.41 2102585.09
Scientific
•Run algorithm to solve real problem.
•Check for agreement with model.
Challenge: need all of the above
% java SortTest 1000000
10 44.44
100 847.85
1000 12985.91
10000 175771.70
100000 2218053.41
Empirical
•Run algorithm to solve real problem.
•Measure running time and/or
count operations.
Challenge: need good implementation
19

Potential drawbacks to the scientific approach
1. Model may not be realistic.
• A challenge in any scientific discipline.
• Advantage in CS: we can randomize to make the model apply.
➛
20
2. Math may be too difficult.
• A challenge in any scientific discipline (cf. statistical physics).
• A “calculus” for AofA is the motivation for this course!
3. Experiments may be too difficult.
• Not compared to other scientific disciplines.
• Can’t implement? Why analyze?

P A R T O N E
OF
•Resources
1c.AofA.Quicksort

Example: Quicksort
public class Quick
{
private static int partition(Comparable[] a, int lo, int hi)
{
int i = lo, j = hi+1;
while (true)
{
while (less(a[++i], a[lo])) if (i == hi) break;
while (less(a[lo], a[--j])) if (j == lo) break;
if (i = j) break;
exch(a, i, j);
}
exch(a, lo, j);
return j;
}
private static void sort(Comparable[] a, int lo, int hi)
{
if (hi = lo) return;
int j = partition(a, lo, hi);
sort(a, lo, j-1);
sort(a, j+1, hi);
}
}
23
Section 2.3
http://algs4.cs.princeton.edu/23quicksort/Quick.java

Start: Preliminary decisions
Cost model
• running time?
• better approach: separate algorithm from implementation
• for sorting, associate compares with inner loop.
• Hypothesis: if number of compares is C, running time is ~aC
Input model
• assume input randomly ordered (easy to arrange)
• assume keys all different (not always easy to arrange)
Key question: Are models/assumptions realistic?
Stay tuned.
timing
counting
24

Setup: Relevant questions about quicksort
Assume array of size N with entries distinct and randomly ordered.
Q. How many compares to partition?
A. N+1
Q. What is the probability that the partitioning
item is the kth smallest?
A. 1/N
Q. What is the size of the subarrays
in that case?
A. k −1 and N − k
Q. Are the subarrays randomly
ordered after partitioning?
A. YES.
25

Main step: Formulate a mathematical problem
Recursive program and input model lead to a recurrence relation.
Assume array of size N with entries distinct and randomly ordered.
Let CN be the expected number of compares used by quicksort.
*5 = 5 + +
R 5

5
(*R + *5 R)
for partitioning
probability
k is the
partitioning
element
compares for
subarrays
when k is the
partitioning
element
26

Simplifying the recurrence
Collect terms.
5*5 = (5 + )*5 + 5
Subtract same formula for N−1. 5*5 (5 )*5 = 5 + *5
*5 = 5 + +
R 5

5
(*R + *5 R)
Multiply both sides by N. 5*5 = 5(5 + ) +
R 5
*R
both sums are
C0 + C1 + ... + CN-1
Apply symmetry. *5 = 5 + +

5
R 5
*R
27
* =

Aside
Simplified recurrence gives efficient algorithm for computing result
*5 = 5 + +
R 5

5
(*R + *5 R )
AofA: Finding a fast way to compute the running time of a program
QUADRATIC time
c[0] = 0;
for (int N = 1; N = maxN; N++)
{
c[N] = N+1;
for (int k = 0; k N; k++)
c[N] += (c[k] + c[N-1-k])/N;
}
⤷
5*5 = (5 + )*5 + 5
c[0] = 0;
c[N] = (N+1)*c[N-1]/N + 2;
LINEAR time
⤷
28

Solving the recurrence
5*5 = (5 + )*5 + 5
Tricky (but key) step:
divide by N(N+1)
*5
5 +
=
*5
5
+

5 +
Telescope.
*5
5 +
=
*5
5
+

5 +
=
*5
5
+

5
+

5 +
=
*

+

+ . . . +

5
+

5 +
*5 5
R 5

R
5
Simplify (ignore small terms).
Approximate with an
integral (stay tuned)
*5 5(

_
K_ + ) 5
= 5 ln 5 ( )5 Euler’s constant ≐ .57721
29
N

Finish: Validation (mathematical)
It is always worthwhile to check your math with your computer.
public class QuickCheck
{
public static void main(String[] args)
{
int maxN = Integer.parseInt(args[0]);
double[] c = new double[maxN+1];
c[0] = 0;
c[N] = (N+1)*c[N-1]/N + 2;
for (int N = 10; N = maxN; N *= 10)
{
double approx = 2*N*Math.log(N) - 2*(1-.577215665)*N;
StdOut.printf(%10d %15.2f %15.2fn, N, c[N], approx);
}
}
}
% java QuickCheck 1000000
10 44.44 37.60
100 847.85 836.48
1000 12985.91 12969.94
10000 175771.70 175751.12
100000 2218053.41 2218028.23
1000000 26785482.23 26785452.45
30
5 ln 5 ( )5
5*5 = (5 + )*5 + 5

Finish: Validation (checking the model)
It is always worthwhile to use your computer to check your model.
Example: Mean number of compares used by Quicksort for randomly ordered
distinct keys is
Experiment: Run code for randomly ordered distinct keys, count compares
Observation: May be interested in distribution of costs
1000 trials for each N
one grey dot for each trial
red dot: average for each N
31
5 ln 5 ( )5
✓
5 ln 5 ( )5

Quicksort compares: limiting distribution is not “normal”
32
Bottom line:
• A great deal is known about the performance of Quicksort.
• AofA leads to intriguing new research problems.
exact distribution
(from recurrence)
for small N
centered
on mean
empirical
validation
N = 1000
see “Approximating the Limiting Quicksort Distribution.” by Fill and Janson (RSA 2001).

Easy method to predict (approximate) performance
Hypothesis: Running time of Quicksort is ~aN ln N.
Experiment.
• Run for input size N. Observe running time.
• [Could solve for a.]
• Predict time for 10N to increase by a factor of
Note: Best to also have accurate mathematical model. Why?
Example:
•Run quicksort 100 times for N = 100,000: Elapsed time: 4 seconds.
•Predict running time of 4 x 10.2 = 40.8 seconds for N = 1M.
•Observe running time of 41 seconds for N = 1M
•Confidently predict running time of 41 x 1000.5 = 11.4 hours for N = 1B.
a(10N) ln(10N)
aN ln N
= 10 +
ln 10
ln N
= 10 +
1
log10 N
33
10 10
12 48
48
48 x (70/6) x (80/7) x (90/8)
= 20 hours

Validate-refine-analyze cycle
It is always worthwhile to validate your model in applications.
Quicksort: Validation ongoing for 50 years!
Example 3 (2010s): Sorting for networking.
• Application: sort ~1B records ~1K characters each.
• Need to beat the competition or go out of business.
• Refinement: adapt to long stretches of equal chars (avoid excessive caching)
Example 1 (late 1970s): Sorting on the CRAY-1.
• Application: cryptography.
• Need to “sort the memory” 1M pseudo-random 64-bit words.
• Bottom line: analysis could predict running time to within 10−6 seconds.
as many times
as possible!
Example 2 (1990s): UNIX system sort.
• Application: general-purpose.
• User app involving files with only a few distinct values performed poorly.
• Refinements: 3-way partitioning, 3-way string quicksort (see Algs4).
• Refined models (not simple): research ongoing. see “The number of symbol comparisons in QuickSort and QuickSelect.”
byVallee, Clement, Fill, and Flajolet (ICALP 2009).
34

Double happiness
35
“People who analyze algorithms have double happiness. First of all
they experience the sheer beauty of elegant mathematical patterns
that surround elegant computational procedures. Then they
receive a practical payoﬀ when their theories make it possible to
get other jobs done more quickly and more economically.”
— D. E. Knuth (1995)

P A R T O N E
OF
•Resources
1d.AofA.Resources

Books
are the prime resources associated with this course.
38
Main
text
(2013)
Text
for
Part II
Reference
for
Algorithms
First edition
(1995)
Reference
for
Java
Reading the books is the best way to develop understanding.

Booksites
are web resources associated with the books.
39
Surf the booksite to search for information, code, and data.

Extensive original research
is the basis for the material in this course.
40
A prime goal of this course: make this work accessible to you.
research papers
and books
by hundreds
of others
20,000+ pages of material (!)
Flajolet's
collected
works
collected works
to appear 2014
Cambridge U. Press
Knuth's
collected
works

More resources
41
Symbolic math
Web references
Math typesetting

Introduce, read, discuss
1. We introduce topics in lecture.
2. You read the book and do assignments before the next lecture.
3. We discuss reading and exercises online. [No assessments.]
The main resource in this class is YOU!
Goal: For you to learn quite a few things that you do not now know.
42

Exercises 1.14 and 1.15
How many recursive calls in Quicksort?
How many exchanges?
43

Exercises 1.17 and 1.18
Switch to insertion sort for small subarrays.
What choice of the threshold minimizes the number of compares?
44

Assignments for next lecture
1. Surf booksites
• http://aofa.cs.princeton.edu
• http://algs4.cs.princeton.edu
2. Start learning to use software.
• StdJava (from Algs4 booksite)
• TeX (optional: .html/MathJax)
3. Download Quicksort and predict performance on your computer.
4. Read pages 1-39 in text.
5. Write up solutions to Exercises 1.14, 1.15, 1.17, and 1.18.
45

Analysis of algorithms description pdf yeah

More Related Content

Similar to Analysis of algorithms description pdf yeah (20)

Recently uploaded (20)

Analysis of algorithms description pdf yeah