study Streaming Multigrid For Gradient Domain Operations On Large Images

Streaming Multigrid for Gradient-Domain Operations on Large Images Michael Kazhdan , Johns Hopkins University Hugues Hoppe , Microsoft Research SIGGRAPH 08

Abstract Develop a streaming multigrid solver – 2 passes over out-of-core data Solver of Poisson eq. with unconstrained boundary conditions Construct Relaxation , Restriction and Prolongation operators in multigrid method based on the B-spline basis Build up a framework to pipeline multigrid with a window of rows of images

Abstract Develop a streaming multigrid solver – 2 passes over out-of-core data 2 nd order finite-elements in a single V-cycle Temporally blocked relation Multi-level streaming to pipeline restriction & prolongation into single streaming passes Key contribution forward-difference gradient  B-spline finite-element

Outline Introduction Related work Review of finite-difference multigrid Our finite-element multigrid approach Streaming multigrid solver Efficient convergence of 2 nd -order elements Implementation Application results Conclusions & future work

Gradient-domain problem as Poisson solution Solve the problem that Find U to minimize The Poisson equation

Image processing on gradient-domain Lighting removal by zeroing small gradient [Horn 74] HDR image tone-mapped by adaptively attenuating luminance gradients [Weiss 01] Overlapping images stitched seamlessly by merging gradients [P´erez et al.03; Agarwala et al.04;Levin et al.04] Shadow removal by zeroing large luminance gradients in regions of constant chromaticity [Finlayson et al. 02] Undesirable reflections removed in flash and ambient image pairs [Agrawal et al. 05] Photographic tone management is improved using gradient constraints [Bae et al. 06]

Painting on gradient-domain Painting with interactive gradient-domain modeling [McCann & Pollard 08] Diffusion curves [Orzan et al. 08]

Large images on gradient-domain Processing of large images [Kopf et al. 07] GB pixel images are too large to fit main memory Direct solution (ex: Cholesky factorization) is impractical Iteration techniques (ex: conjugate gradients and multigrid method) are in-efficient Requiring many iterations over out-of-core data

Contributions A general, accurate and efficient solver for Poisson eq. over out-of-core images Sufficient accuracy in 2 V-cycles on GB images 2 nd order B-spline finite elements gets more accuracy than traditional finite element in multigrid method Efficiency on temporally locality Small moving windows of data in memory Pipelined V-cycle : 3 Gauss-Seidel relaxations, restriction and prolongation

Contributions On a single CPU core Solve a 3-channel, 16-MB gradient-domain problem with an rms error on the order of 10 −5 in 15 seconds. High ratio of local computation to memory bandwidth Temporal locality in L1 cache

Solvers of Poisson equation Iterative solvers Gauss-Seidel and conjugate gradient Memory-efficient Require many iterations Reduce # of iterations using multi-resolution pre-conditioners [Gortler and Cohen 95; Szeliski 06] Reduce # of iterations using multigrid solvers [Brandt 77; Briggs et al. 00] GPU + multigrid [Bolz et al. 03; Goodnight et al. 03; G¨oddeke et al. 08]

Out-of-core Multigrid are difficult to schedule out-of-core [Toledo 99] Solve out-of-core problem on a coarser-resolution grid and then upsample the resulting approximation [Kopf et al. 07a] Can not maintain robust sharp features Adaptive partition to solve Poisson eq.For image stitching, Poisson eq. in the image seams [Agarwala 07] Poisson eq. in the context of fluid flow simulation [Losasso et al. 04] Poisson eq. in the surface reconstruction [Kazhdan et al. 06]. Our method addresses the general case where The Poisson eq. is solved accurately everywhere The problem size can not be reduced No initial solution guess is available

Gradient-domain image processing  Poisson Equation

▽ u x 0 x 1 x i x N u 0 u 1 u i u N h u i+1

Δ U x i+1 x i-1 x i x N u i-1 u N h u i+1 u i

Lu=f U u 1,1 u 1,2 u 1,3 u 2,1 u 2,2 u 2,3 u 3,1 u 3,2 u 3,3

Lu=f Gauss-Seidel relaxation on u L u = f ( L+D+R ) u = f where L = ( L+D+R ) D u = f – ( L+R ) u u (k) = D -1 ( f – ( L+R ) u (k-1) ) = D -1 f + R J u (k-1)

Error, residual Consider Lu = f Error e k = u – u (k) Residual r k = f - L u (k) = L e k u (k) = D -1 f + R J u (k-1) e k = u – u (k) = ( D -1 f + R J u ) – ( D -1 f + R J u (k-1) ) = R J ( u - u (k-1) ) = R J e k-1 = R J k e 0 e  0 depends on R J and e o

Limits of Gauss-Seidel e k = R J k e 0 Gauss-Seidel relaxation is smooth but converges slowly on low-freq. components Why use coarse grids ? [Briggs et al. 00] Coarse grids can be used to compute an improved initial guess for the fine-grid relaxation Relaxation on the coarse-grid is much cheaper

coarse-grid correction Relax k times on L h u h = f h on fine-grid , initial u (0) arbitrary u (k) = D -1 f + R J u (k-1) Compute the residual of the find-grid r = L h u h (k) - f h Restrict the residual to the coarse-grid r H = R r h where H = 2h, R is the restriction Compute the error on the coarse-grid L H e H = r H , where L H = R L h P Prolongate (interpolate) the error to the fine-grid e h = P e H , where P is the prolongation Correct the fine-grid solution u h (k+1) = u h (k) + e h 2 3 1 1 1 4 5 6 Fine grid coarse grid

Restriction & Prolongation Restriction operator Fine grid  coarse grid Typically using local weighted averaging Prolongation operator Coarse grid  fine grid Typically using bilinear interpolation Restriction (1D) prolongation (1D) R = P T

2D stencils of the multigrid 2D Restriction 2D Prolongation 2D Relaxation (Laplacian)

V-cycle v h  MV h (v h , f h ) Relax k times on L h u h = f h , initial v arbitrary If Ω h is the coarsest grid, goto 4. Else f 2h  R(f h – L h v h ) = R r h // restriction v 2h  0 v 2h  MV 2h (v 2h , f 2h ) Correct v h  v h + P v 2h // prolongation Relax k times on L h u h = f h , initial guess v h Relaxation on L h u h = f h Restriction f 2h = R r h Relaxation on L 2h u 2h = f 2h Relaxation on L 4h u 4h = f 4h Restriction f 2h = R r h

Standard multigrid V-cycle f l-1 =R l l -r l u l =P l l-1 u P l-1 +u R l L lmin u lmin =f lmin Base solution

Representing the Poisson equation with 1D B-spline basis Solving Poisson eq . reduces to solving Lu = f L as the N x N matrix with L i,j = < Δ B i (x), B j (x)> = <> f as the vector with f j = < F(x), B j (x)>  L i,j = < ▽ B i (x), －▽ B j (x)>

Fitting forward-difference gradient constrains Gradient t of unknown image v Difference of B-spline

2D stencils of the multigrid operators using B-splines Prolongation Restriction Laplacian

How to pipeline ? phases of each row A l  R  A l-1  R  A l-2  P  A l-1  P  A l Does neighborhood in Laplacian stencil exist ? Gauss-Seidel Relaxation (A), Restriction (R), Prolongation (P) Time row row Perform 3 times relaxations (A) as a single streaming operations R n-1 Al R Al-1 R Al-2 P Al-1 P Al R n Al R Al-1 R Al-2 P Al-1 P R n+1 Al R Al-1 R Al-2 P Al-1 A l R A l-1 A l-2 R P A l-1 A l P R n-1 Al Al Al R n Al Al Al R n+1 Al Al Al

Temporally blocked relaxation Maintain a window of rows [i-1, i+2k+1] to perform a skipping, counter-current relaxation sweep, updating pixels in rows {i+2k-1, i+2k-3, …, i+1} Row 4 {11} Relaxed twice: Row 3 {02,06}, row 2{03,08} Relaxed once: Row 5 {7}, row {10} Perform k=3 times relaxations (A) as a single streaming operations Row 1 Row 2 Row 3 Row 4 Row 5 Row 6 The pixel row is memory-resident i The i th relaxation globally

Full data pipeline for gradient-domain processing

Memory analysis Implement the active windows on u l R and f l as circular memory buffer of images rows Window size (w, h) w: the image width at the coarser level h: 2k+3 (restriction), 2k+5 (prolongation) Memory usage is O(N x ) for an N x x N y image

Parameters (n,k,v) n-order of the finite element (n-order B-spline) K times Gauss-Seidel relaxations v passes of V-cycles 2-order B-spline basis  (2, 3, 2)

(n, v) Plot of the rms and max errors vs. # of multigrid V-cycles 2 nd -order element give the fastest convergence !

Parameter selection Sufficiently accurate solution with a minimum number of V-cycles 8-bit channel image processing Max error < 1/256  (2,5,1): 2 nd order B-spline basis, 5 Gauss-Seidel updates with a single-V cycle in all our applications

Implementation Maximizing disk throughput Larger block (4MB) transfer to minimize disk latency 2X speed improvement Storing the intermediate u,f floating-point value on dist at half precision Relaxation optimization leverage the vertical and horizontal symmetries of the stencil Make use of CPU SSE2- 4-vector instructors Non-power-of-two image Padding the input image to 2 lmax-lmin times the coarest level Multi-channel image Stitching requires full-color gradient Interleave the per-channel solutions to reduce the total # of passes

Environment NB 2.2 GHz Core 2 Duo processor 4 GB RAM Timing I/O to read the gradient field from disk Write JPEG-compressed ouput to disk Initial solution u=0 in all experiments (2,5,1) 2 nd order B-spline basis, 5 Gauss-Seidel updates with a single-V cycle

Image stitching 19,588 x 4,457 (87 MB) panorama form 9 photos Copy the image gradients and solving the Poisson eq.

Image stitching SM: streaming multigrid solver QT: quadtree (AT) solver of Agarwala [07]

Tone-mapping (HDR  normal tone) before after

Tone-mapping (HDR  normal tone) Stream multigraid is the first one to solve Poisson eq in time that is linear on the # of pixels

GB stitching and tone-mapping May not capture the true scene contrast

Conclusion Streaming multigrid Out-of-core tech. for solving large global linear system Local access A few passes of sequential I/O 2 nd order B-spline finite element formulation is compatible with traditional multigrid Efficient accurate solution in a single V-cycle

Future work Dirichlet boundary condition Modify 2D stencils Construct f Soft constraint to match some original image u 0 Weighted minimization where w(x,y) is a spatially varying 2x2 diagonal matrix that weight difference in x and y independently Bilaplacian

Future work Reduce disk bandwidth by using compression/decompression of the streamed temporary data Parallelization Many-core CPUs or GPUs for instance by partitioning image rows

study Streaming Multigrid For Gradient Domain Operations On Large Images

More Related Content

What's hot (20)

Similar to study Streaming Multigrid For Gradient Domain Operations On Large Images (20)

More from Chiamin Hsu (12)

study Streaming Multigrid For Gradient Domain Operations On Large Images

Editor's Notes