6 098 Digital and Computational Photography 6 882 Advanced Computational Photography Image Warping and Morphing Fr do Durand Bill Freeman MIT EECS Olivier Gondry s visit Thursday Friday Contact Peter Sand sand mit edu Important scientific question How to turn Dr Jekyll into Mr Hyde How to turn a man into a werewolf Powerpoint cross fading Important scientific question How to turn Dr Jekyll into Mr Hyde How to turn a man into a werewolf Powerpoint cross fading From An American Werewolf in London or Image Warping and Morphing Digression old metamorphoses http en wikipedia org wiki The Strange Case of Dr Jekyll and Mr Hyde http www eatmybrains com showtopten php id 15 http www horror wood com next gen jekyll htm Unless I m mistaken both employ the trick of making already applied makeup turn visible via changes in the color of the lighting something that works only in black and white cinematography It s an interesting alternative to the more familiar Wolf Man time lapse dissolves This technique was used to great effect on Fredric March in Rouben Mamoulian s 1932 film of Dr Jekyll and Mr Hyde although Spencer Tracy eschewed extreme makeup for his 1941 portrayal Averaging images Cross fading Pretty much the compositing equation C F 1 B Averaging vectors V P 1 Q P V Q Warping Morphing combine both For each pixel Transform its location like a vector Then linearly interpolate like an image Morphing Input two images I0 and IN Expected output image sequence Ii with i2 1 N 1 User specifies sparse correspondences on the images Pairs of vectors P0j PNj Morphing For each intermediate frame It Interpolate feature locations Pti 1 t P0i t P1i Perform two warps one for I0 one for I1 Deduce a dense warp field from the pairs of features Warp the pixels Linearly interpolate the two warped images Warping Intelligent design image warping D Arcy Thompson http www groups dcs st and ac uk history Miscellaneous darcy html http en wikipedia org wiki D Arcy Thompson Importance of shape and structure in evolution Warping Imagine your image is made of rubber warp the rubber No prairie dogs were harmed when creating this image Careful warp vs inverse warp How do you perform a given warp Forward warp Potential gap problems Inverse lookup the most useful For each output pixel Lookup color at inversewarped location in input Image Warping non parametric Move control points to specify a spline warp Spline produces a smooth vector field Slide Alyosha Efros Warp specification dense How can we specify the warp Specify corresponding spline control points interpolate to a complete warping function But we want to specify only a few points not a grid Slide Alyosha Efros Warp specification sparse How can we specify the warp Specify corresponding points interpolate to a complete warping function How do we do it How do we go from feature points to pixels Slide Alyosha Efros Triangular Mesh 1 Input correspondences at key feature points 2 Define a triangular mesh over the points Same mesh in both images Now we have triangle to triangle correspondences 3 Warp each triangle separately from source to destination Slide Alyosha Efros Problems with triangulation morphing Not very continuous only C0 Fig L Darsa Folding problems Warp as interpolation We are looking for a warping field A function that given a 2D point returns a warped 2D point We have a sparse number of correspondences These specify values of the warping field This is an interpolation problem Given sparse data find smooth function Interpolation in 1D We are looking for a function f We have N data points xi yi Scattered spacing between xi is non uniform We want f so that For each i f xi yi f is smooth Depending on notion of smoothness different f Radial Basis Functions RBF Place a smooth kernel R centered on each data point xi f z i R z xi Radial Basis Functions RBF Place a smooth kernel R centered on each data point xi f z i R z xi Find weights i to make sure we interpolate the data for each i f xi yi Kernel Many choices In Assignment 4 we simply use inverse multiquadric where c controls falloff Lazy way set c to an arbitrary constant pset 4 Smarter way c is different for each kernel For each xi set c as the squared distance to the closest other xj Enforcing interpolation f z i R z xi N equations for each j f xj yj i R xj xi yj N unknowns i Just inverse the matrix Important note f z i R z xi for each j i R xj xi yj Note that the influence of each function is non zero everywhere at a data point the value of the other bases is not zero In contrast to e g various interpolation splines Variations of RBF Lots of possible kernels Gaussians e r2 2 Thin plate splines r2 log r Sometimes add a global polynomial term Recap 1D scattered data interpolation Sparse input output pairs xi yi non uniformly sampled RBFs Radial Basis Functions Weighted sum of kernels R centered on data points xi f z i R z xi Compute the weights i by enforcing interpolation f xj yj Simple linear system Recap 1D scattered data interpolation Sparse input output pairs xi yi non uniformly sampled RBFs Radial Basis Functions Weighted sum of kernels R centered on data points xi f z i R z xi Compute the weights i by enforcing interpolation f xj yj Simple linear system QUESTION RBF for warping 2D case Instead of f R R we now deal with f R2 R2 For each 2D point f gives us another 2D warped point We have N data points Pairs of input 2D vector output 2D vector Careful xi is now a 2D vector so is yi Don t be confused with coordinates x y Place 2D kernels at each data point The weights i are now 2D vectors Solve a linear system of 2N equations and 2N unknowns Applying a warp USE INVERSE Forward warp For each pixel in input image Paste color to warped location in output Problem gaps Inverse warp For each pixel in output image Lookup color from inverse warped location Example Example Fold problems Oh well 1D equivalent of folds There is no guarantee that our 1D RBF is monotonic Yes it means that the notion of inverse of the warp is questionable input result remember inverse warp Hardcore Photoshop for portrait Morphing Input images Feature correspondences The feature locations will be our yi Interpolate feature location Provides the xi Warp each image to intermediate location Two different warps Same target location different source location Note the yi do not change along the animation but the xi are different for each intermediate image Here we show t 0 5 the yi are in the middle i e the xi are the same intermediate
View Full Document
Unlocking...