Karmarkar's Algorithm For Linear Programming Problem

Karmarkar’s
Algorithm

AK Dhamija

Introduction Karmarkar’s Algorithm
Complexity
LP Problem An Interior Point Method of Linear Programming Problem
Klee-Minty
Example
Comparison

Original
Algorithm AK Dhamija
Steps
Iterations
Transformation
DIPR, DRDO
Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2 November 20, 2009
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 1/44

Overview

Karmarkar’s
Algorithm 1 Introduction
Complexity
AK Dhamija LP Problem
Klee-Minty Example
Introduction Comparison
Complexity
LP Problem 2 Original Algorithm
Klee-Minty Steps
Example
Iterations
Comparison
Transformation
Original
Algorithm 3 Aﬃne Variant
Steps Three Concepts
Iterations Example
Transformation Concepts 1 & 2
Aﬃne Variant & 3: Centering
Iterations
Three Concepts
Rescaling
Example
Concepts 1 & 2
Final Solution
& 3: Centering Another Example
Iterations
Rescaling 4 Further Issues
Final Solution
Another
Example 5 References
Further Issues

References

Algorithms

Karmarkar’s
Algorithm

AK Dhamija

Introduction Algorithms Complexity
Complexity
If n is the size of the problem (n may be number of variables in LP Problem)
LP Problem
Klee-Minty • Time Complexity - Time taken by algorithm
Example
Comparison • Space Complexity - Memory needed by algorithm
√
Original
• Increasing order of complexity : lg n, n, n, n lg n, n2 , n3 , 2n , n!
Algorithm • O(n2 ) = c1 n2 , O(2n ) = c2 2n
Steps
Iterations • Compare O(n2 ) = 100n2 , O(2n ) = 2n
Transformation • 22500 vs 32768 for n = 15
Affine Variant • 1000000 vs 1267650600228229401496703205376 for n = 100
Three Concepts
Example
• O(n2 ) = c1 n2 is polynomial time (aka efficient) algorithm and O(2n ) = c2 2n is exponential
time (aka inefficient) algorithm
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

LP Problem

Karmarkar’s
Algorithm

AK Dhamija
Problem Instance

Introduction • Set of n real variables e.g. x,y etc
Complexity
LP Problem
• Set of restrictions in the form of linear inequalities e.g. x + y ≤ 5, y ≥ 0.2 etc
Klee-Minty • Goal e.g. Maximize 4x + y
Example
Comparison
• Solution: (x, y) = (5, 0)

Original
Algorithm
Feasible Region
Steps
Iterations
Transformation
• Feasible region is convex
Aﬃne Variant • Each constraint generates at most one edge in
Three Concepts the feasible region
Example
Concepts 1 & 2 • As you move the goal line, G, to increase it’s
& 3: Centering value, the point that will maximize it is one of
Iterations the ’corners’.
Rescaling
Final Solution
Another
Example

Further Issues

References

LP Algorithms

Karmarkar’s Naive Algorithm
Algorithm
• Find all intersection points, check these points to satisfy all constraints i.e feasibility, and ﬁnd out
AK Dhamija the point of optimum goal value
• O(mn ) for n variables and m constraints - exponential in n
Introduction
Complexity
LP Problem Simplex (George B. Dantzig, 1947)
Klee-Minty
Example • Start from a feasible corner, and Move to the one that gives the highest value to the goal function
Comparison
• STOPPING condition : none of the neighboring points gives a higher value than the current one
Original
Algorithm
• Average time is linear; worst case time is exponential (covers all feasible corners)

Steps
• Remarkably successful in practice
Iterations • Integer Programming is NP-complete
Transformation

Aﬃne Variant
Ellipsoidal Method (Leonid Khachiyan, 1979)
Three Concepts
Example
Concepts 1 & 2
• Runtime: polynomial
& 3: Centering • Theoretical value; not useful in practice
Iterations
Rescaling
Final Solution Interior point method (Narendra Karmarkar, 1984)
Another
Example • Runtime: polynomial
Further Issues • Practical value, Recently these have become competitive in practice with simplex.
References

An example of Exponential Time

Karmarkar’s
Algorithm
Klee-Minty Example
AK Dhamija

Introduction
Complexity
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Developments
Three Concepts
Example
• The LP has n variables, n constraints and 2n extreme
Concepts 1 & 2
& 3: Centering
points
Iterations
Rescaling • The elementary simplex method, starting at x = 0, goes
Final Solution
Another
Example
through each of the extreme points before reaching the
Further Issues optimum solution at (0, 0, ..., 5n )
References

Klee-Minty Example

Karmarkar’s
Algorithm
Here is the pivot sequence for n = 3, which goes through all 8
AK Dhamija
extreme points, starting at the origin. Let s be the slack
Introduction variables.
Complexity
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example Exponential Time Algorithm in Worst case
Further Issues

References

General Introduction

Karmarkar’s
Algorithm

AK Dhamija
Interior-point methods for linear programming

Introduction Components
Complexity
LP Problem • A breakthrough development in linear programming during
Klee-Minty
Example the 1980s
Comparison

Original • Started in 1984 by a young mathematician at AT&T Bell
Algorithm
Steps Labs, N. Karmarkar
Iterations
Transformation • Polynomial-time algorithm which can solve huge linear
Aﬃne Variant
Three Concepts
programming problems beyond the reach of the simplex
Example
Concepts 1 & 2
method.
& 3: Centering
Iterations • Karmarkar’s method stimulated development in both
Rescaling
Final Solution interior-point and simplex methods.
Another
Example

Further Issues

References

Diﬀerence between Interior point methods and the
simplex method
Karmarkar’s
Algorithm

AK Dhamija The nature of trial solutions and Complexity
Introduction
Complexity
LP Problem Simplex Method
Klee-Minty
Example • CPF (Corner Point Feasible) solutions
Comparison
• Worst Case:No of iterations can increase exponentially in the number of variables n : O(2n )
Original • Practically remarkably successful except some huge problems (eg Airlines Scheduling etc)
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant Interior Point Methods
Three Concepts
Example • Interior points (points inside the boundary of the feasible region)
Concepts 1 & 2
& 3: Centering
• Polynomial time
Iterations • Advantageous in solving huge problems, but cumbersome for not-so-large problems
Rescaling
Final Solution
Another
Example

Further Issues

References

Karmarkar’s Original Algorithm

Karmarkar’s
Algorithm Karmarkar assumes that the LP is given in Canonical form of
AK Dhamija the problem
Introduction • Min Z = CX
Complexity
LP Problem • such that AX = 0, 1X = 1, X ≥ 0
Klee-Minty
Example
Comparison

Original Assumptions
Algorithm
Steps
1 1 1
• X0 = ( n , n , ...., n ) is a feasible solution
Iterations
Transformation
• Minimum(Z) = 0
Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
To apply the algorithm to LP problem in standard form, a
Iterations
Rescaling
transformation is needed
Final Solution
Another
• Min Z = CX
Example

Further Issues • such that AX ≤ b, X ≥ 0
References


Karmarkar’s
Algorithm
Examples : Standard Vs Canonical Form
AK Dhamija Standard Form
Introduction Min Z = y1 + y2 (Z = CY)
Complexity
LP Problem
such that
Klee-Minty
Example
Comparison • y1 + 2y2 ≤ 2 (AY ≤ b)
Original
Algorithm
• y1 , y2 ≥ 0
Steps
Iterations
Transformation Canonical Form
Aﬃne Variant
Three Concepts Min Z = 5x1 + 5x2 (Z = CX)
Example
Concepts 1 & 2 such that
& 3: Centering
Iterations
Rescaling • 3x1 + 8x2 + 3x3 - 2x4 = 0 (AX = 0)
Final Solution
Another
Example
• x1 + x2 + x3 + x4 = 1 (1X = 1)
Further Issues • xj ≥ 0, j = 1, 2, 3, 4
References


Karmarkar’s
Algorithm

AK Dhamija

The Principle Idea
Introduction
Complexity
Create a sequence of points x(0) , x(1) , x(2) , ..., x(k) having decreasing values of the objective function.
LP Problem
Klee-Minty In the kth step, the point x(k) is brought into the center of the simplexa by projective transformation.
Example
Comparison a
n-dimensional unit simplex S is the set of points(x1 , x2 , ...xn )T satisfying
Original
Algorithm
x1 + x2 + ... + xn = 1 and xj ≥ 0, j = 1, 2, ...n
Steps
Iterations
Transformation Three key Concepts
Aﬃne Variant • Projection of a vector onto the set of X satisfying AX = 0
Three Concepts
Example • Karmarkar’s Centering Transformation
Concepts 1 & 2 • Karmarkar’s Potential Function
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

Three Concepts

Karmarkar’s Projection
Algorithm
• we want to move from a feasible point X0 to another feasible point X1 , that for some fixed vector
AK Dhamija v, will have a larger value of vX
• if we choose to move in direction d = (d1 , d2 , ...dn ) that solves the optimization problem
Introduction Max vd such that
Complexity Ad = 0, d1 + d2 + ...dn = 0 (so that Ad remains feasible)
LP Problem and ||d|| = 1
Klee-Minty then we will be moving in a feasible direction that maximizes the increase in vX per unit length
Example moved.
Comparison
• The direction d that solves this optimization problem is given by the projection of v onto X
Original satisfying AX = 0 and x1 + x2 + ... + xn = 0 and is given by [I − BT (BBT )−1 B]v,
Algorithm A
Steps where B =
1
Iterations
Transformation

Affine Variant
Three Concepts • Geometrically, if we can write v = p + w, where p
Example satisfies Ap = 0 and w is perpendicular to all
Concepts 1 & 2 vectors X satisfying AX = 0, then p is the
& 3: Centering projection of v onto the set of X satisfying AX = 0.
Iterations
Rescaling
Final Solution
• For Example, v = (−2, −1, 7) is projected onto a
Another set of 3-d vectors satisfying x3 = 0(ie x1 − x2
Example plane), then v = (−2, −1, 0) + (0, 0, 7). So here,
p = (−2, −1, 0) is the vector in the set of X
Further Issues satisfying AX = 0 that is closest to v.
References

Three Concepts

Karmarkar’s
Algorithm
Karmarkar’s Centering Transformation
AK Dhamija
• If xk is a point in S, then f ([x1 , x2 , ...xn ]|xk ) transforms a point [x1 , x2 , ...xn ]T in S into a
point [y1 , y2 , ...yn ] in S, where
Introduction xj
Complexity xk
j
LP Problem yj = r=n xr
r=1 xk
Klee-Minty r
Example
Comparison • Consider the LP
Min z = x1 + 3x2 - 3x3 such that
Original x2 - x3 = 0
Algorithm x1 + x2 + x3 = 1
Steps xi ≥ 0
Iterations This LP has a feasible solution [ 1 , 1 , 1 ]T and the optimal value of z is 0.
3 3 3
Transformation
The feasible point [ 1 , 3 , 8 ]T yields the following transformation
4 8
3
Aﬃne Variant 1 3 3
f ([x1 , x2 , x3 ]|[ 4 , 8 , 8 ]) =
Three Concepts 8x2 8x3
Example 4x1 3 3
[x1 , x2 , x3 ]|[ 8x2 8x3 , 8x2 8x3 , 8x2 8x3 ]
Concepts 1 & 2 4x1 + + 4x1 + + 4x1 + +
3 3 3 3 3 3
& 3: Centering 1 , 1 , 1 ]|[ 1 , 3 , 3 ]) = [ 12 , 8 , 8 ]
Iterations
For example, f ([ 3 3 3 4 8 8 28 28 28
Rescaling
Final Solution
• We now refer to the variables x1 , x2 , ..., xn as being the original space and the variables
y1 , y2 , ..., yn as being the transformed space and the unit simplex involving variables
Another
Example y1 , y2 , ..., yn will be called transformed unit simplex.

Further Issues

References

Three Concepts

Karmarkar’s
Algorithm

AK Dhamija
Properties of Centering Transformation

Introduction • f (xk |xk ) = [ n , n , ..., n ]T
1 1 1

Complexity f (.|xk ) maps xk into the center of the transformed unit simplex.
LP Problem
Klee-Minty • f (x|xk ) ∈ S and For x = x , f (x|xk ) = f (x |xk )
Example Any point in S is transformed into a point in the transformed unit simplex, and no two points in S
Comparison can yield the same point (i.e. f is one-one mapping)
Original • For any point [y1 , y2 , ...yn ]T in S, there is a unique point [x1 , x2 , ...xn ]T in S satisfying
Algorithm
f ([x1 , x2 , ...xn ]T |xk ) = [y1 , y2 , ...yn ]T
Steps
Iterations
The point [x1 , x2 , ...xn ]T is given by
Transformation xk yj
j
r=n xk y
Aﬃne Variant r=1 r r
Three Concepts we can write f −1 ([y1 , y2 , ...yn ]T |xk )
= [x1 , x2 , ...xn ]T
Example The above two equations imply that f is a one-one onto mapping from s to S.
Concepts 1 & 2
& 3: Centering
• A point x) in S will satisfy Ax = 0 if A[diag(xk )]f (x|xk ) = 0
Iterations
Feasible points in the original problem correspond to points y in the transformed unit simplex that
Rescaling satisfy A[diag(xk )]y = 0
Final Solution
Another
Example

Further Issues

References


Karmarkar’s Step k = 0
Algorithm
• Start with the solution point Y0 = X0 = ( n , n , ...., n )T
1 1 1
AK Dhamija
• Compute step length parameters r = √ 1
n(n−1)
and α =
(n−1)
3n
Introduction
Complexity
LP Problem Step k
Klee-Minty
Example Deﬁne
Comparison
• Dk = diag{Xk } = diag{xk1 , xk2 , ..., xkn }
Original
Algorithm • P=
ADk
1
Steps
Iterations Compute
Transformation
• cp = [I − PT (PPT )−1 P](CDk )T
Aﬃne Variant
Three Concepts
• Ynew = Y0 - αr ||cp ||
p
c

Example
Concepts 1 & 2 • Xk+1 = 1D Y
D Y
k new
k new
& 3: Centering
Iterations • Z = CXk+1
Rescaling
Final Solution
• k=k+1
Another
Example
• Repeat iteration k until Z becomes less than prescribed tolerance
Dk −1 X
Further Issues Centering : X is brought to center by Y = −1
1D X
k
References


Karmarkar’s
Algorithm

AK Dhamija

Introduction
Complexity Certain Remarks
LP Problem
Klee-Minty
Example
• We move from the center of the transformed unit simplex in a direction opposite to the projection of
Comparison CDk )T onto the transformation of the feasible region( the set of y satisfying Adiag{Xk }y =
0). As per our explanation of projection, this ensures that we maintain feasibility(in the transformed
Original space) and move in a direction that maximizes the rate of decrease of CDk )T
Algorithm
Steps • By moving a distance αr from the center of the transformed unit simplex, we ensure that yk+1
Iterations will remain in the interior of the transformed simplex
Transformation
• When we use the inverse of Karmarkar’s centering transformation to transform yk+1 back into
Aﬃne Variant xk+1 , the deﬁnition of projection imply that xk+1 will be feasible for the original LP
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s
Algorithm

AK Dhamija
Potential Function : Why do we project CDk )T rather than C)T onto the transformed space?
Introduction
Complexity • Because we are projecting CDk T rather than CT , we can not be sure that each iteration will
LP Problem decrease Z. In fact it is possible for CXk+1 > CXk to occur.
Klee-Minty
Example • Karmarkar’s Potential Function f (X) is defined as f (X) =
Comparison j=n T
sumj=1 ln( CX
x
), X = [x1 , x2 , ..., xn ]T
j
Original
Algorithm • Karmarkar showed that if we project CDk )T (not C)T ) onto the feasible region in the
Steps transformed space, then for some δ > 0, it will be true that for k = 0, 1, 2, ... f (Xk )
Iterations - f (Xk+1 ) ≥ δ.
Transformation
• So, each iteration of karmarkar’s algorithm decreases the potential function by an amount bounded
Affine Variant away from 0.
Three Concepts
Example • Karmarkar showed that if the potential function evaluated at xk is small enough, the Z = CXk
Concepts 1 & 2 will be near 0. Because f (xk ) is decreased by at least δ per iteration, it follows that by choosing k
& 3: Centering sufficiently large, we can ensure that the Z-value for Xk is less than
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

Karmarkar’s Original Algorithm - Two Iterations

Karmarkar’s
Algorithm

AK Dhamija Problem

Min Z = 2x2 - x3
Introduction such that
Complexity
LP Problem • x1 - 2x2 + x3 = 0
Klee-Minty
Example • x1 + x2 + x3 = 1
Comparison • xj ≥ 0, j = 1, 2, 3
Original
Solution : z = 0 at (0, 0.33334, 0.66667)
Algorithm
Steps
Iterations
Transformation Preliminary Step
Aﬃne Variant • k=0
Three Concepts
Example
• X0 = ( n , n , ...., n )T = ( 1 , 1 , 1 )T
1 1 1
3 3 3
Concepts 1 & 2
& 3: Centering
• r= √ 1
n(n−1)
= √ 1
3(3−1)
= √1
6
Iterations
Rescaling • α=
(n−1)
3n
(3−1) 2
= (3)(3) = 9
Final Solution
Another
Example

Further Issues

References


Karmarkar’s Iteration 0
Algorithm
• Y0 = X0 = ( 3 , 1 , 3 )T
1
3
1
AK Dhamija
• D0 = diag{X0 } = diag{ 1 , 1 , 3 }
3 3
1

Introduction • AD0 = (1, −2, 1)diag{X0 } = diag{ 1 , 1 , 3 } = ( 1 , −2 , 1 )
3 3
1
3 3 3
Complexity
1 −2 1
LP Problem
Klee-Minty
• P=
AD0
1
= 3
1
3
1
3
1
Example
Comparison • CD0 = (0, 2, −1)diag{ 3 , 3 , 1 } = (0, 3 , −1 )
1 1
3
2
3
Original
Algorithm • PPT =
0.667
0
0
3
; (PPT )−1 =
1.5
0
0
0.333
Steps  
Iterations 0.5 0 0.5
Transformation • PT (PPT )−1 P =  0 1 0 
0.5 0 0.5
Aﬃne Variant
Three Concepts • cp = [I − PT (PPT )−1 P](CD0 )T = (0.167, 0, −0.167)T
Example
Concepts 1 & 2
• ||cp || = (0.167)2 + 0 + (−0.167)2 = 0.2362
& 3: Centering ( 2 )( √ )
1
Iterations • c
Ynew = Y0 - αr ||cp || = ( 1 , 3 , 1 )T -
p 3
1
3
9
0.2362
6
(0.167, 0, −0.167)T =
Rescaling
Final Solution (0.2692, 0.3333, 0.3974)T
Another
Example • X1 = Ynew = (0.2692, 0.3333, 0.3974)T
Further Issues • Z = CX1 = (0, 2, −1)(0.2692, 0.3333, 0.3974)T = 0.2692

References • k=0+1=1


Algorithm
• D1 = diag{X1 } = diag{0.2692, 0.3333, 0.3974}
AK Dhamija
• AD1 = (1, −2, 1)diag{X1 } = diag{0.2692, 0.3333, 0.3974} =
(0.2692, −0.6666, 0.3974)
Introduction −0.6666
Complexity • P=
AD1
1
=
0.2692
1 1
0.3974
1
LP Problem
Klee-Minty • CD1 = (0, 2, −1)diag{0.2692, 0.3333, 0.3974} = (0, 0.6666, −0.3974)
Example
Comparison • PPT =
0.675
0
0
3
; (PPT )−1 =
1.482
0
0
0.333
Original  
Algorithm 0.441 0.067 0.492
Steps • PT (PPT )−1 P =  0.067 0.992 −0.059 
Iterations 0.492 −0.059 0.567
Transformation
• cp = [I − PT (PPT )−1 P](CD1 )T = (0.151, −0.018, −0.132)T
Aﬃne Variant
Three Concepts
• ||cp || = (0.151)2 + (−0.018)2 + (−0.132)2 = 0.2014
Example ( 2 )( √ )
1
Concepts 1 & 2 • c
Ynew = Y0 - αr ||cp || = ( 1 , 3 , 1 )T -
p 3
1
3
9
0.2014
6
(0.151, −0.018, −0.132)T =
& 3: Centering
Iterations (0.2653, 0.3414, 0.3928)T
Rescaling
Final Solution • D1 Ynew = diag{0.2692, 0.3333, 0.3974}(0.2653, 0.3414, 0.3928)T =
Another (0.0714, 0.1138, 0.1561)T ; 1D1 Ynew = 0.3413
Example

Further Issues
• D Y
1 new = (0.2092, 0.3334, 0.4574)T
X2 = 1D Y
1 new

References • Z = CX1 = (0, 2, −1)(0.2092, 0.3334, 0.4574)T = 0.2094; k = 1 + 1 = 2

Transforming a LP Problem into Karmarkar’s
Special Form
Karmarkar’s Steps for Conversion of the LP problem : Min Z = CX such that AX ≥ b, X ≥ 0
Algorithm

AK Dhamija
1 Write the dual of given primal problem
Min Z = bW
such that AT W ≤ CT , W ≥ 0
Introduction
Complexity
2 1 Introduce slack & surplus variables to primal and dual problems
LP Problem 2 Combine these two problems.
Klee-Minty
Example 3 1 Introduce a bounding constraint xi + wi ≤ K), where K should be sufficiently large
Comparison to include all feasible solutions of original problem.
Original 2 Introduce a slack variable in the bounding constraint and obtain xi + wi + s = K
Algorithm
Steps
4 1 Introduce a dummy variable d (subject to condition d = 1)to homogenize the constraints.
Iterations 2 Replace the equations
Transformation xi + wi + s = K and d = 1
with the following equations
Affine Variant
xi + wi + s − Kd = 0 and xi + wi + s + d = K + 1
Three Concepts
Example 5 Introduce the following transformation so as to obtain one on the RHS of the last equation
Concepts 1 & 2 xj = (K + 1)yj , j = 1, 2, ...m + n
& 3: Centering wj = (K + 1)ym+n+j , j = 1, 2, ...m + n
Iterations
s = (K + 1)y2m+2n+1
Rescaling
d = (K + 1)y2m+2n+2
Final Solution
Another
Example 6 Introduce an artificial variable y2m+2n+3 (to be minimized) in all the equations such that the sum
of the coefficients in each homogenous equation is zero and coefficient of the artificial variable in the
Further Issues last equation is one.
References

Example of transforming a LP Problem into
Karmarkar’s Special Form
Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3,
Algorithm x1 , x2 are nonnegative
AK Dhamija
1 Write the dual of given primal problem
Min Z = 5w1 + 3w2
Introduction such that
Complexity w1 + w2 ≥ 2
LP Problem w1 − w2 ≥ 1
Klee-Minty w1 , w2 ≥ 0
Example
Comparison 2 Introduction of slack & surplus variables and combination of primal and dual problems
x1 + x2 + x3 = 5
Original x1 − x2 + x4 = 3
Algorithm w1 + w2 − w3 = 2
Steps w1 − w2 − w4 = 1
Iterations 2x1 + x2 = 5w1 + 3w2
Transformation

Aﬃne Variant
3 Addition of boundary constraint with slack variable s
4 4
i=1 xi + i=1 wi + s = K
Three Concepts
Example 4 Homogenized equivalent system with dummy variable d
Concepts 1 & 2
x1 + x2 + x3 − 5d = 0
& 3: Centering
x1 − x2 + x4 − 3d = 0
Iterations
w1 + w2 − w3 − 2d = 0
Rescaling
Final Solution
w1 − w2 − w4 − d = 0
Another 2x1 + x2 − 5w1 − 3w2 = 0
4 4
Example i=1 xi + i=1 wi + s − Kd = 0
4 4
Further Issues i=1 xi + i=1 wi + s + d = (K + 1)
with all variables non-negative
References

Example of transforming a LP Problem into
Karmarkar’s Special Form
Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3,
Algorithm x1 , x2 are nonnegative
AK Dhamija
5 Introduction of transformations
xj = (K + 1)yj , j = 1, 2, ...4
Introduction wj = (K + 1)y4+j , j = 1, 2, ...4
Complexity s = (K + 1)y9
LP Problem d = (K + 1)y10
Klee-Minty The system of equations thus obtained is as follows
Example y1 + y2 + y3 − 5y10 = 0
Comparison y1 − y2 + y4 − 3y10 = 0
y5 + y6 − y7 − 2y10 = 0
Original
y5 − y6 − y8 − y10 = 0
Algorithm
2y1 + y2 − 5y5 − 3y6 = 0
Steps 9
Iterations i=1 yi − Ky10 = 0
10
Transformation i=1 yi = 1
Aﬃne Variant
Three Concepts 6 Introduce an artiﬁcial variable y11
Example Minimize y11
Concepts 1 & 2 subject to
& 3: Centering y1 + y2 + y3 − 5y10 + 2y11 = 0
Iterations y1 − y2 + y4 − 3y10 + 2y11 = 0
Rescaling y5 + y6 − y7 − 2y10 + y11 = 0
Final Solution y5 − y6 − y8 − y10 + 2y11 = 0
Another 2y1 + y2 − 5y5 − 3y6 + 5y11 = 0
Example 9
i=1 yi − Ky10 + (K − 9)y11 = 0
Further Issues 11
i=1 yi = 1
References

The Aﬃne Variant of Algorithm

Karmarkar’s
Algorithm

AK Dhamija

Introduction Three Concepts
Complexity
LP Problem • Concept 1: Shoot through the interior of the feasible
Klee-Minty
Example
Comparison
region towards an optimal solution
Original • Concept 2: Move in the direction that improves the
Algorithm
Steps objective function value at the fastest feasible rate.
Iterations
Transformation
• Concept 3: Transform the feasible region to place the
Aﬃne Variant
Three Concepts current trail solution near its center, thereby enabling the
Example
Concepts 1 & 2 fastest feasible rate for Concept 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

An Example

Karmarkar’s The Problem
Algorithm
Max Z = x1 + 2x2
AK Dhamija such that

• x1 + x2 ≤ 8
Introduction
Complexity
• xj ≥ 0, j = 1, 2
LP Problem
The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

An Example

Karmarkar’s Concept 1: Shoot through the interior of the feasible region toward an optimal solution
Algorithm
• The algorithm begins with an initial solution that lies in the interior of the feasible region
AK Dhamija
• Arbitrarily choose (x1 , x2 ) = (2, 2) to be the initial solution

Introduction
Complexity Concept 2: Move in a direction that improves the objective function value at the fastest feasible rate
LP Problem
Klee-Minty • The direction is perpendiculars to (and toward) the objective function line.
Example
Comparison
• (3, 4) = (2, 2) + (1, 2), where the vector (1, 2) is the gradient( aka Coeﬃcients) of the Objective
Function
Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

An Example

Karmarkar’s The algorithm actually operates on the augmented form
Algorithm
Max Z = CX such that
AK Dhamija
• AX = b
• X≥0
Introduction
Complexity
LP Problem The Transformation
Klee-Minty
Example ⇒
Max Z = x1 + 2x2 = Max Z = x1 + 2x2 such that
Comparison
• x1 + x2 ≤ 8 = x1 + x2 + x3 = 8, where x3 is the slack
⇒
Original
Algorithm • xj ≥ 0, j = 1, 2 = xj ≥ 0, j = 1, 2, 3
⇒
Steps The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16
Iterations
Transformation

Aﬃne Variant The Matrices
Three Concepts
   
x1 0
Example
C= 1 2 0 X =  x2  A = 1 1 1 b= 8 0= 0 
Concepts 1 & 2
x3 0
& 3: Centering
Iterations
Rescaling
Final Solution The Solution
Another
Example • ⇒ ⇒
Initial Solution (2, 2) = (2, 2, 4), Optimum (0, 8) = (0, 8, 0)
Further Issues • ⇒
Gradient of Objective Function (1, 2) = (1, 2, 0)

References

An Interior-point Algorithm

Gradient of Objective function : C = 1 2 0
Karmarkar’s
Algorithm
Using Projected Gradient to implement Concept 1 & 2
AK Dhamija • ⇒
Adding the gradient to the initial leads to (3, 4, 4) = (2, 2, 4) + (1, 2, 0) = Infeasible
• To remain feasible, the algorithm projects the point (3, 4, 4) down onto the feasible tetrahedron
Introduction
Complexity
• The next trial solution moves in the direction of projected gradient i.e. the gradient projected onto
the feasible region
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s Using Projected Gradient to implement Concept 1 & 2
Algorithm
 
2 −1 −3 1
3 3
AK Dhamija • Projection Matrix P = I − AT (AAT )−1 A =  − 1

3
2
3
1 
−3 
−13
−13
2
3
Introduction  
0
Complexity
LP Problem
• T
Projected Gradient cP = PC =  1 
Klee-Minty −1
Example      
Comparison 2 2 0
• Move (2, 2, 4) towards cP to X =  2  + 4αcP =  2  + 4α  1 
Original 4 4 −1
Algorithm
Steps
• α determines how far we move. large α ⇒ too close to the boundary and α ⇒ more iterations we
Iterations
have chosen α = 0.5, so the new trial solution move to: X = (2, 3, 3)
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s
Algorithm
Concept 3: Transform the feasible region to place the current trail
AK Dhamija
solution near its center, thereby enabling a large improvement when
Introduction concept 2 is implemented
Complexity
LP Problem
Klee-Minty
Centering scheme for implementing Concept 3
Example
Comparison
• Why : The centering scheme keeps turning the direction of the
Original
Algorithm projected gradient to point more nearly toward an optimal
Steps solution as the algorithm converges toward this solution
Iterations
Transformation
• How : Simply changing the scale for each of the variable so that
Aﬃne Variant
Three Concepts the trail solution becomes equidistant from the constraint
Example
Concepts 1 & 2 boundaries in the new coordinate system
& 3: Centering
Iterations ˜
• x is brought to the center X = (1, 1, 1) in the new coordinate
Rescaling
Final Solution system by X˜ = D−1 X, where D = diag{X}
Another
Example

Further Issues

References


Karmarkar’s Centering scheme for implementing Concept 3
Algorithm
   
1 0 0
 1 0 0
  
x1 2 1
AK Dhamija  2 2
X = D−1 X =  0
˜ 1 0   x2  =  0 1 0 2 = 1 
  
2 2 
0 0 1 x3 0 0 1 4 1
4 4
Introduction
Complexity
LP Problem The Problem in the new coordinate system becomes
Klee-Minty
Example Max Z = 2˜1 + 4˜2
x x
Comparison such that
Original
Algorithm
• 2˜1 + 2˜2 + 4˜3 = 8
x x x
Steps • xj ≥ 0, j = 1, 2, 3
˜
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

Complete Illustration of the Algorithm

Karmarkar’s
Algorithm Iteration 1
AK Dhamija • Initial Solution (x1 , x2 , x3 ) = (2, 2, 4)
   
1 0 0 1 0 0

x1
Introduction •  2 1  ˜ −1
2
1
D = diag{x1 , x2 , x3 } =  0 0 X=D X= 0 0 x2 
 
2 2 
Complexity 0 0 1 0 0 1 x3
LP Problem   4 4
1 0 0
  
Klee-Minty 2 1
Example  2 1
= 0 0  2 = 1
 
Comparison 2
0 0 1 4 1
4
Original  
Algorithm 2 0 0
Steps
• ˜
A = AD = 1 1 1  0 2 0 = 2 2 4
Iterations
0 0 4
Transformation 
2 0 0

1


Aﬃne Variant • ˜
CT = DCT =  0 2 0  2 = 2 4 0
Three Concepts
0 0 4 0
Example
 
5 −1 −1
Concepts 1 & 2 6 6 3
& 3: Centering • Projection Matrix P = I − AT (AAT )−1 A =  − 1
˜ ˜ ˜ ˜ 
6
5
6
−1
3


Iterations −13
−13
1
3
Rescaling  
Final Solution 1
Another • Projected Gradient cP = PCT˜ = 3 
Example −2
Further Issues

References


Algorithm
• Identify v (how far to move) as the absolute value of the negative component of cP having the
AK Dhamija largest value, so that v = | − 2| = 2
 
1
Introduction • In this coordinate system the algorithm moves from current solution to x =  1  + α cP =
˜ v
Complexity 1
 
LP Problem     5
1 1 4
Klee-Minty  1  + 0.5  3  =  7 
Example 2  4 
1 −2 1
Comparison 2
Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s
   
AK Dhamija x1 2 0 0
• ˜
In the original coordinate system the solution is X =  x2  DX =  0 2 0 
x3 0 0 4
Introduction
 
5  5 
4  2
Complexity  7
 4 = 7  2
LP Problem 1
Klee-Minty 2
2
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues
• This completes the iteration 1. The new solution will be used to start the next iteration

References


Karmarkar’s

AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = ( 5 , 7 , 2)
2 2
 5   
2
0 0 1
Introduction • D = diag{x1 , x2 , x3 } =  0 7
2
0 X=D ˜ −1
X= 1 
Complexity 0 0 2 1
LP Problem    
13 − 187 −9 2 − 11
Klee-Minty 18 12 
Example • P =  − 18
 7 41
90
− 14  cP =  133 
45
 
60
Comparison 2
−9 − 14
45
37
45
− 41
15
Original
Algorithm
• The current solution becomes (0.83, 1.4, 0.5) which corresponds (2.08, 4.92, 1.0) in the original
coordinate system
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s
AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = (2.08, 4.92, 1.0)
 
2.08 0 0
Introduction • D = diag{x1 , x2 , x3 } =  0 4.92 0 
Complexity 0 0 1.0
LP Problem  
−1.63
Klee-Minty
Example • cP =  1.05 
Comparison −1.79
Original • The current solution becomes (0.54, 1.3, 0.5) which corresponds (1.13, 6.37, 0.5) in the original
Algorithm coordinate system
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References


Karmarkar’s Eﬀect of rescaling of each iteration
Algorithm
Sliding the optimal solution toward (1, 1, 1) while the other BF solutions tend to slide away
AK Dhamija

Introduction
Complexity
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation (1) (2)
Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues
(3) (4)
References


Karmarkar’s
Algorithm
More iterations
AK Dhamija
Starting from the current trial solution x, following the steps, x is moving toward the optimum (0, 8).
When the trial solution is virtually unchanged from the proceeding one, then the algorithm has virtually
Introduction converged to an optimal solution. We stop.
Complexity
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

Another Example

Karmarkar’s
Algorithm
The Problem
AK Dhamija Max Z = 5x1 + 4x2
such that
Introduction
Complexity
• 6x1 + 4x2 ≤ 24
LP Problem • x1 + 2x2 ≤ 6
Klee-Minty
Example
• −x1 + x2 ≤ 1
Comparison • x2 ≤ 2
Original • xj ≥ 0, j = 1, 2
Algorithm
Steps
Iterations
Augmented Form
Transformation

Aﬃne Variant Max Z = 5x1 + 4x2
such that
Three Concepts
Example
Concepts 1 & 2
• 6x1 + 4x2 + x3 = 24
& 3: Centering • x1 + 2x2 + x4 = 6
Iterations
Rescaling
• −x1 + x2 + x5 = 1
Final Solution • x2 + x6 = 2
Another
Example • xj ≥ 0, j = 1, 2, 3, 4, 5, 6

Further Issues

References

Another Example

Karmarkar’s
The Matrices
Algorithm
   
6 4 1 0 0 0 24
AK Dhamija  1 2 0 1 0 0   6 
A=  −1  1 C=
b=  5 4 0 0 0 0
1 0 0 1 0 
Introduction 0 1 0 0 0 1 2
Complexity
LP Problem
Klee-Minty
Example
Comparison

Original
Algorithm
Steps
Iterations
Transformation

Aﬃne Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

Another Example

Karmarkar’s
Algorithm
Starting from an initial solution X = (1, 1, 14, 3, 1, 1)
AK Dhamija

Introduction
The Steps
Complexity
LP Problem
D = diag{X}
Klee-Minty
Example ˜
A = AD
Comparison

Original ˜
c = DC
Algorithm
Steps ˜ ˜˜ ˜
P = I − AT (AAT )−1 A
Iterations
Transformation
cP = P˜
c
Aﬃne Variant  
Three Concepts
Example
1
˜ = 1 +
X α
v cP
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
1
Final Solution
Another X = DX˜
Example

Further Issues

References

Further Issues

Karmarkar’s
Algorithm

AK Dhamija
Interior-point methods is designed for dealing with big problems. Although the claim that it’s much faster
than the simplex method is controversy, many tests on huge LP problems show its outperformance
Introduction
Complexity
LP Problem Future Research
Klee-Minty
Example • Infeasible interior points method - remove the assumption that there always exits a nonempty interior
Comparison • Methods applying to LP problems in standard form
Original • Methods dealing with finding initial solution, and estimating the optimal solution
Algorithm
Steps
• Methods working with primal-dual problems
Iterations • Studies about moving step-long/short steps
Transformation
• Studies about efficient implementation and complexity of various methods
Affine Variant
Three Concepts
Example Karmarkar’s paper not only started the development of interior point methods, but also encouraged rapid
Concepts 1 & 2
& 3: Centering improvement of simplex methods
Iterations
Rescaling
Final Solution
Another
Example

Further Issues

References

References

Karmarkar’s References
Algorithm
• N. Karmarkar, A New Polynomial - Time Algorithm for Linear Programming, Combinatorica 4 (4),
AK Dhamija 1984, p. 373-395
• Yanrong Hu & Ce Yu, Paper review of ENGG*6140 Optimization Techniques – Interior-Point
Introduction Methods for LP, 2003
Complexity • I. Lustig, R. Marsten, D. Shanno, Interior-point Methods for Linear Programming: Computational
LP Problem State of the Art, ORSA Journal on Computing, 6 (1), 1994, p. 1-14
Klee-Minty
Example • Hillier,Lieberman,Introduction to Operations Research (7th edition) 320-334
Comparison
• Taha, Operations Research: An Introduction (6th edition) 336-345
Original • D. Gay, A Variant of Karmarkar’s Linear Programming Algorithm for Problems in Standard form,
Algorithm Mathematical Programming 37 (1987) 81-90
Steps
Iterations
• E.R. Barnes, A Variation on Karmarkar’s Algorithm for Solving Linear Programming problems,
Transformation Mathematical Programming 36, 1986, p. 174-182

Aﬃne Variant
• V. Klee and G.J. Minty, How Good is the Simplex Algorithm? In O. Shisha, editor, Inequalities, III,
pages 159-175. Academic Press, New York, NY, 1972.
Three Concepts
Example • Richard Bronson and Govindasamy Naadimuthu, Schaum’s Outlines, Operations Research, Second
Concepts 1 & 2 Edition, Mcgraw Hill.
& 3: Centering
Iterations
• Wayne L Winston, Operations Research, Applications and Algorithms, Fourth Edition, Thomson
Rescaling
Brooks/Cole, 2007
Final Solution
Another
Example Presentation available at
Further Issues www.geocities.com/a_k_dhamija.
References

Karmarkar's Algorithm For Linear Programming Problem

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Karmarkar's Algorithm For Linear Programming Problem (20)

More from Ajay Dhamija (14)

Recently uploaded (20)

Karmarkar's Algorithm For Linear Programming Problem