6.098 Digital and Computational Photography 6.882 Advanced Computational Photography Image Warping and MorphingOlivier Gondry's visitImportant scientific questionSlide 4Digression: old metamorphosesAveraging imagesAveraging vectorsWarping & Morphing combine bothMorphingSlide 10Slide 11Intelligent design & image warpingWarpingCareful: warp vs. inverse warpImage Warping – non-parametricWarp specification - denseWarp specification - sparseTriangular MeshProblems with triangulation morphingWarp as interpolationInterpolation in 1DRadial Basis Functions (RBF)Slide 23KernelEnforcing interpolationImportant noteVariations of RBFRecap: 1D scattered data interpolationSlide 29RBF for warping: 2D caseApplying a warp: USE INVERSEExampleSlide 331D equivalent of foldsHardcore Photoshop for portraitSlide 36Slide 37Slide 38Input imagesFeature correspondencesInterpolate feature locationWarp each image to intermediate locationSlide 43Interpolate colors linearlyRecapMovie timeSlide 47The sampling problemIntuitionSimilar case: texture aliasingThe BibleResamplingNotationsSlide 54Slide 55Slide 56Resampling: progressionSlide 58Put it togetherSlide 60Resampling – convolution viewSlide 62Slide 63Slide 64Slide 65Slide 66Resampling filterResampling FilterEWA resamplingImage Quality ComparisonSlide 71Morphing & mattingUniform morphingNon-uniform morphingVideoSlide 76Problem with morphingView morphingMain trickSlide 80Slide 81Slide 82Slide 83Slide 84Shape VectorThe Morphable face modelThe Morphable Face ModelSubpopulation meansDeviations from the meanUsing 3D Geometry: Blanz & Vetter, 1999Manipulating Facial Appearance through Shape and ColorMorphable face modelsEigenFacesThe average faceFigure-centric averagesJason Salavon“100 Special Moments” by Jason Salavon3D morphingSlide 99Automatic morphingSlide 101Slide 102Slide 103RefsSoftwareNext time: Panoramas6.098 Digital and Computational Photography 6.882 Advanced Computational PhotographyImage Warping and MorphingFrédo DurandBill FreemanMIT - EECSOlivier Gondry's visit•Thursday Friday•Contact Peter Sand: [email protected] scientific question•How to turn Dr. Jekyll into Mr. Hyde?•How to turn a man into a werewolf?•Powerpoint cross-fading?Important scientific question•How to turn Dr. Jekyll into Mr. Hyde?•How to turn a man into a werewolf?•Powerpoint cross-fading? •or•Image Warping and Morphing?From An American Werewolf in LondonDigression: old metamorphoses •http://en.wikipedia.org/wiki/The_Strange_Case_of_Dr._Jekyll_and_Mr._Hyde•http://www.eatmybrains.com/showtopten.php?id=15•http://www.horror-wood.com/next_gen_jekyll.htm •Unless I’m mistaken, both employ the trick of making already-applied makeup turn visible via changes in the color of the lighting, something that works only in black-and-white cinematography. It’s an interesting alternative to the more familiar Wolf Man time-lapse dissolves. This technique was used to great effect on Fredric March in Rouben Mamoulian’s 1932 film of Dr. Jekyll and Mr. Hyde, although Spencer Tracy eschewed extreme makeup for his 1941 portrayal.Averaging images•Cross-fading–Pretty much the compositing equationC= F +(1-) BAveraging vectors•V=  P + (1-) QPVQWarping & Morphing combine both•For each pixel–Transform its location like a vector–Then linearly interpolate like an imageMorphing•Input: two images I0 and IN•Expected output: image sequence Ii, with i2 1..N-1•User specifies sparse correspondences on the images–Pairs of vectors {(P0j, PNj)}Morphing•For each intermediate frame It–Interpolate feature locations Pti= (1- t) P0i + t P1i–Perform two warps: one for I0, one for I1•Deduce a dense warp field from the pairs of features•Warp the pixels–Linearly interpolate the two warped imagesWarpingIntelligent design & image warping•D'Arcy Thompson http://www-groups.dcs.st-and.ac.uk/~history/Miscellaneous/darcy.htmlhttp://en.wikipedia.org/wiki/D'Arcy_Thompson•Importance of shape and structure in evolutionWarping•Imagine your image is made of rubber•warp the rubberNo prairie dogs were harmed when creating this imageCareful: warp vs. inverse warpHow do you perform a given warp:•Forward warp–Potential gap problems•Inverse lookup the most useful–For each output pixel•Lookup color at inverse-warped location in inputImage Warping – non-parametric•Move control points to specify a spline warp•Spline produces a smooth vector fieldSlide Alyosha EfrosWarp specification - dense•How can we specify the warp?Specify corresponding spline control points•interpolate to a complete warping functionBut we want to specify only a few points, not a gridSlide Alyosha EfrosWarp specification - sparse•How can we specify the warp?Specify corresponding points•interpolate to a complete warping function•How do we do it?How do we go from feature points to pixels?Slide Alyosha EfrosTriangular Mesh1. Input correspondences at key feature points2. Define a triangular mesh over the points–Same mesh in both images!–Now we have triangle-to-triangle correspondences3. Warp each triangle separately from source to destinationSlide Alyosha EfrosProblems with triangulation morphing•Not very continuous –only C0•Folding problemsFig. L. DarsaWarp as interpolation•We are looking for a warping field–A function that given a 2D point, returns a warped 2D point•We have a sparse number of correspondences–These specify values of the warping field•This is an interpolation problem–Given sparse data, find smooth functionInterpolation in 1D•We are looking for a function f•We have N data points: xi, yi–Scattered: spacing between xi is non-uniform•We want f so that–For each i, f(xi)=yi–f is smooth•Depending on notion of smoothness, different fRadial Basis Functions (RBF)•Place a smooth kernel R centered on each data point xi•f (z) =  i R(z, xi)Radial Basis Functions (RBF)•Place a smooth kernel R centered on each data point xi•f (z) =  i R(z, xi) •Find weights i to make sure we interpolate the datafor each i, f(xi)=yiKernel•Many choices•In Assignment 4, we simply use inverse multiquadric•where c controls falloff•Lazy way: set c to an arbitrary constant (pset 4)•Smarter way: c is different for each kernel. For each xi, set c as the squared distance to the closest other xjEnforcing interpolation•f (z) =  i R(z, xi) •N equationsfor each j, f(xj) = yji R(xj, xi) = yj•N unknowns