1Image Warping and Mosacing15-463: Rendering and Image ProcessingAlexei Efros…with a lot of slides stolen from Steve Seitz and Rick SzeliskiTodayMosacsImage WarpingHomographiesProgramming Assignment #2 OUTReading:Paul Heckbert, “Projective Mappings for Image Warping”, 1999Rick Szeliski, Chapter on Mosaicing from his new book (2005-2006), being written as we speak, hopefully on the web by Monday2Mosaics: stitching images togethervirtual wide-angle cameraHow to do it?Basic Procedure• Take a sequence of images from the same position– Rotate the camera about its optical center• Compute transformation between second image and first• Transform the second image to overlap with the first• Blend the two together to create a mosaic• If there are more images, repeat…but wait, why should this work at all?• What about the 3D geometry of the scene?• Why aren’t we using it?3A pencil of rays contains all viewsrealcamerasyntheticcameraCan generate any synthetic camera viewas long as it has the same center of projection!Aligning imagesTranslations are not enough to align the imagesleft on top right on top4mosaic PPImage reprojectionThe mosaic has a natural interpretation in 3D• The images are reprojected onto a common plane• The mosaic is formed on this plane• Mosaic is a synthetic wide-angle cameraImage reprojectionBasic question• How to relate two images from the same camera center?– how to map a pixel from PP1 to PP2PP2PP1Answer• Cast a ray through each pixel in PP1• Draw the pixel where that ray intersects PP2But don’t we need to know the geometryof the two planes in respect to the eye?Observation:Rather than thinking of this as a 3D reprojection, think of it as a 2D image warp from one image to another5Image Warpingimage filtering: change range of imageg(x) = h(f(x))fxhfxfxhfximage warping: change domain of imageg(x) = f(h(x))Image Warpinghhffggimage filtering: change range of imageg(x) = h(f(x))image warping: change domain of imageg(x) = f(h(x))6Parametric (global) warpingExamples of parametric warps:translationrotationaspectaffineperspectivecylindrical2D image transformationsThese transformations are a nested set of groups• Closed under composition and inverse is a member7Recovering warp parametersGiven two images and a transformation type, how do we recover the parameters? 1. Search• parameters of h() are: • Easy for translation: loop through txand ty• But harder with more DOF• Minimization, e.g. Newton’s Method2. Point Correspondences (often user-defined)• Click on the same point in two images• How many correspondences for:– translation, Euclidean, similarity, affine, projective[]∑−ihxhIxI221))(()(minarghI1I2HomographyQ: Which t-form is the right one for warping PP1 into PP2?• translation, Euclidean, affine, projectiveA: Projective – mapping between any two PPs with the same center of projection• rectangle should map to arbitrary quadrilateral • must preserve straight lines• same as: project, rotate, reprojectcalled HomographyPP2PP1=1yx*********wwy'wx'H pp’To apply a homography H• Compute p’ = Hp (regular matrix multiply)• Convert p’ from homogeneous to image coordinates8Image warping with homographiesimage plane in front image plane belowblack areawhere no pixelmaps toImage rectificationTo unwarp (rectify) an image• solve for homography H given a set of p and p’ pairs• solve equations of the form: wp’ = Hp– linear in unknowns: w and coefficients of H– H is defined up to an arbitrary scale factor– how many points are necessary to solve for H?pp’9Solving for homographiesCan set scale factor i=1. So, there are 8 unkowns.Set up a system of linear equations:Ah = bwhere vector of unknowns h = [a,b,c,d,e,f,g,h]TNeed at least 8 eqs, but the more the better…Solve for h. If overconstrained, solve using least-squares: Can be done in Matlab using “\” command• see “help lmdivide”=1yxihgfedcbawwy'wx'p’ = Hp2min bAh −Fun with homographiesSt.Petersburgphoto by A. TikhonovVirtual camera rotationsOriginal image10Panoramas1. Pick one image (red)2. Warp the other images towards it (usually, one by one)3. blendchanging camera centerDoes it still work?synthetic PPPP1PP211Planar scene (or far away)PP3 is a projection plane of both centers of projection, so we are OK!This is how big areal photographs are madePP1PP3PP2Planar mosaic12Image warpingGiven a coordinate transform (x’,y’) = h(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(h(x,y))?x x’h(x,y)f(x,y) g(x’,y’)y y’f(x,y) g(x’,y’)Forward warpingSend each pixel f(x,y) to its corresponding location (x’,y’) = h(x,y) in the second imagex x’h(x,y)Q: what if pixel lands “between” two pixels?y y’13f(x,y) g(x’,y’)Forward warpingSend each pixel f(x,y) to its corresponding location (x’,y’) = h(x,y) in the second imagex x’h(x,y)Q: what if pixel lands “between” two pixels?y y’A: distribute color among neighboring pixels (x’,y’)– Known as “splatting”f(x,y) g(x’,y’)xyInverse warpingGet each pixel g(x’,y’) from its corresponding location (x,y) = h-1(x’,y’) in the first imagex x’Q: what if pixel comes from “between” two pixels?y’h-1(x,y)14f(x,y) g(x’,y’)xyInverse warpingGet each pixel g(x’,y’) from its corresponding location (x,y) = h-1(x’,y’) in the first imagex x’h-1(x,y)Q: what if pixel comes from “between” two pixels?y’A: resample color value– We discussed resampling techniques before• nearest neighbor, bilinear, Gaussian, bicubicForward vs. inverse warpingQ: which is better?A: usually inverse—eliminates holes• however, it requires an invertible warp function—not always possible...15Programming Assignment #2Homographies and Panoramic Mosaics• Capture photographs (and possibly video)• can check out cameras and/or tripods from us (1 day loan)• Compute homographies (define correspondences)• will need to figure out how to setup system of eqs.• Warp images (show some nice warps)• Produce panoramic mosaics• will discuss blending next time• Do some of the Bells and Whistles (can be in pairs)Bells and WhistlesBlending and Compositing• use homographies to combine images or video and images together in an interesting (fun) way.
View Full Document
Unlocking...