Homographies and MosaicsWhy Mosaic?Slide 3Slide 4Mosaics: stitching images togetherA pencil of rays contains all viewsHow to do it?Aligning imagesImage reprojectionSlide 10Back to Image WarpingHomographyImage warping with homographiesImage rectificationLeast Squares ExampleSolving for homographiesFun with homographiesAnalysing patterns and shapesSlide 19Slide 20Slide 21Julian Beever: Manual HomographiesPanoramaschanging camera centerPlanar scene (or far away)Planar mosaicProgramming Project #4Bells and WhistlesSlide 29Slide 30From previous year’s classesGo Explore!Homographies and Mosaics15-463: Computational PhotographyAlexei Efros, CMU, Fall 2008© Jeffrey Martin (jeffrey-martin.com)with a lot of slides stolen from Steve Seitz and Rick SzeliskiWhy Mosaic?Are you getting the whole picture?•Compact Camera FOV = 50 x 35°Slide from Brown & LoweWhy Mosaic?Are you getting the whole picture?•Compact Camera FOV = 50 x 35°•Human FOV = 200 x 135°Slide from Brown & LoweWhy Mosaic?Are you getting the whole picture?•Compact Camera FOV = 50 x 35°•Human FOV = 200 x 135°•Panoramic Mosaic = 360 x 180°Slide from Brown & LoweMosaics: stitching images togethervirtual wide-angle cameraA pencil of rays contains all viewsrealcamerasyntheticcameraCan generate any synthetic camera viewas long as it has the same center of projection!How to do it?Basic Procedure•Take a sequence of images from the same position–Rotate the camera about its optical center•Compute transformation between second image and first•Transform the second image to overlap with the first•Blend the two together to create a mosaic•If there are more images, repeat…but wait, why should this work at all?•What about the 3D geometry of the scene?•Why aren’t we using it?Aligning imagesTranslations are not enough to align the imagesleft on top right on topmosaic PPImage reprojectionThe mosaic has a natural interpretation in 3D•The images are reprojected onto a common plane•The mosaic is formed on this plane•Mosaic is a synthetic wide-angle cameraImage reprojectionBasic question•How to relate two images from the same camera center?–how to map a pixel from PP1 to PP2PP2PP1Answer•Cast a ray through each pixel in PP1•Draw the pixel where that ray intersects PP2But don’t we need to know the geometryof the two planes in respect to the eye?Observation:Rather than thinking of this as a 3D reprojection, think of it as a 2D image warp from one image to anotherBack to Image WarpingTranslation2 unknownsAffine6 unknownsPerspective8 unknownsWhich t-form is the right one for warping PP1 into PP2? e.g. translation, Euclidean, affine, projectiveHomographyA: Projective – mapping between any two PPs with the same center of projection•rectangle should map to arbitrary quadrilateral •parallel lines aren’t•but must preserve straight lines•same as: project, rotate, reproject called HomographyPP2PP11yx*********wwy'wx'H pp’ To apply a homography H•Compute p’ = Hp (regular matrix multiply)•Convert p’ from homogeneous to image coordinatesImage warping with homographiesimage plane in front image plane belowblack areawhere no pixelmaps toImage rectificationTo unwarp (rectify) an image•Find the homography H given a set of p and p’ pairs•How many correspondences are needed?•Tricky to write H analytically, but we can solve for it!•Find such H that “best” transforms points p into p’ •Use least-squares!pp’Least Squares ExampleSay we have a set of data points (X1,X1’), (X2,X2’), (X3,X3’), etc. (e.g. person’s height vs. weight)We want a nice compact formula (a line) to predict X’s from Xs: Xa + b = X’We want to find a and bHow many (X,X’) pairs do we need?What if the data is noisy?'22'11XbaXXbaX'2'12111XXbaXXAx=B.........111'3'2'1321XXXbaXXXoverconstrained2min BAx Solving for homographiesCan set scale factor i=1. So, there are 8 unkowns.Set up a system of linear equations:Ah = bwhere vector of unknowns h = [a,b,c,d,e,f,g,h]TNeed at least 8 eqs, but the more the better…Solve for h. If overconstrained, solve using least-squares: Can be done in Matlab using “\” command•see “help lmdivide”1yxihgfedcbawwy'wx'p’ = Hp2min bAh Fun with homographiesSt.Petersburgphoto by A. TikhonovVirtual camera rotationsOriginal imageAnalysing patterns and shapesHomographyAutomatically rectified floorThe floor (enlarged)What is the shape of the b/w floor pattern?Slide from CriminisiFrom Martin Kemp The Science of Art (manual reconstruction)Automatic rectificationAnalysing patterns and shapes2 patterns have been discovered !Slide from CriminisiAutomatically rectified floorSt. Lucy Altarpiece, D. VenezianoAnalysing patterns and shapesWhat is the (complicated)shape of the floor pattern?Slide from CriminisiFrom Martin Kemp, The Science of Art (manual reconstruction)AutomaticrectificationAnalysing patterns and shapesSlide from CriminisiJulian Beever: Manual Homographieshttp://users.skynet.be/J.Beever/pave.htmPanoramas1. Pick one image (red)2. Warp the other images towards it (usually, one by one)3. blendchanging camera centerDoes it still work?synthetic PPPP1PP2Planar scene (or far away)PP3 is a projection plane of both centers of projection, so we are OK!This is how big aerial photographs are madePP1PP3PP2Planar mosaicProgramming Project #4Homographies and Panoramic Mosaics•Capture photographs (and possibly video)•Might want to use tripod•Compute homographies (define correspondences)•will need to figure out how to setup system of eqs.•(un)warp an image (undo perspective distortion)•Produce 3 panoramic mosaics (with blending)•Do some of the Bells and WhistlesBells and WhistlesBlending and Compositing•use homographies to combine images or video and images together in an interesting (fun) way. E.g.–put fake graffiti on buildings or chalk drawings on the ground–replace a road sign with your own poster–project a movie onto a building wall–etc.Bells and WhistlesCapture creative/cool/bizzare panoramas•Example from UW (by Brett Allen):•Ever wondered what is happening
View Full Document