6.098 Digital and Computational Photography 6.882 Advanced Computational PhotographyPanoramasBill FreemanFrédo DurandMIT - EECSLots of slides stolen from Alyosha Efros, who stole them from Steve Seitz and Rick SzeliskiOlivier Gondry• Director of music video and commercial• Special effect specialist (Morphing, rotoscoping)• Today at 5:40pm in 32-141Why Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°Slide from Brown & LoweWhy Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°Slide from Brown & LoweWhy Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°Slide from Brown & LoweMosaics: stitching images togethervirtual wide-angle cameraHow to do it?• Basic Procedure– Take a sequence of images from the same position• Rotate the camera about its optical center– Compute transformation between second image and first– Transform the second image to overlap with the first– Blend the two together to create a mosaic– If there are more images, repeat• …but wait, why should this work at all?– What about the 3D geometry of the scene?– Why aren’t we using it?A pencil of rays contains all viewsrealcamerasyntheticcameraCan generate any synthetic camera viewas long as it has the same center of projection!Aligning images: translationTranslations are not enough to align the imagesleft on top right on topmosaic PPImage reprojection• The mosaic has a natural interpretation in 3D– The images are reprojected onto a common plane– The mosaic is formed on this plane– Mosaic is a synthetic wide-angle cameraImage reprojection• Basic question– How to relate 2 images from same camera center?• how to map a pixel from PP1 to PP2PP2PP1• Answer– Cast a ray through each pixel in PP1– Draw the pixel where that ray intersects PP2But don’t we need to know the geometryof the two planes in respect to the eye?Observation:Rather than thinking of this as a 3D reprojection, think of it as a 2D image warp from one image to anotherBack to Image WarpingTranslation2 unknownsAffine6 unknownsPerspective8 unknownsWhich t-form is the right one for warping PP1 into PP2?e.g. translation, Euclidean, affine, projectiveHomography• Projective – mapping between any two PPs with the same center of projection– rectangle should map to arbitrary quadrilateral – parallel lines aren’t– but must preserve straight lines– same as: project, rotate, reproject• called HomographyPP2PP1⎥⎥⎦⎤⎢⎢⎣⎡⎥⎥⎦⎤⎢⎢⎣⎡=⎥⎥⎦⎤⎢⎢⎣⎡1yx*********wwy'wx'Hpp’To apply a homography H• Compute p’ = Hp (regular matrix multiply)•Convert p’ from homogeneous to image coordinates1D homogeneous coordinates• Add one dimension to make life simpler• (x, w) represent point x/wxww=11D homography• Reproject to different linexww=11D homography• Reproject to different linew=1xw1D homography• Reproject to different line• Equivalent to rotating 2D pointsÎ reprojection is linear in homogeneous coordinatesw=1xwSame in 2D• Reprojection = homography• 3x3 matrix⎥⎥⎦⎤⎢⎢⎣⎡⎥⎥⎦⎤⎢⎢⎣⎡=⎥⎥⎦⎤⎢⎢⎣⎡1yx*********wwy'wx'Hpp’PP2PP1Image warping with homographiesimage plane in front image plane belowblack areawhere no pixelmaps toDigression: perspective correctionFrom Photography, London et al.From Photography, London et al.From Photography, London et al.From Photography, London et al.Tilt-shift lens• 35mm SLR versionPhotoshop version (perspective crop)+ you control reflection and perspective independentlyBack to Image rectificationTo unwarp (rectify) an image• Find the homography H given a set of p and p’ pairs• How many correspondences are needed?• Tricky to write H analytically, but we can solvefor it!• Find such H that “best” transforms points p into p’• Use least-squares!pp’Least Squares Example• Say we have a set of data points (X1,X1’), (X2,X2’), (X3,X3’), etc. (e.g. person’s height vs. weight)• We want a nice compact formula (line) to predict X’s from Xs: Xa + b = X’• We want to find a and b• How many (X,X’) pairs do we need?•• What if the data is noisy?'22'11XbaXXbaX=+=+⎥⎦⎤⎢⎣⎡=⎥⎦⎤⎢⎣⎡⎥⎦⎤⎢⎣⎡'2'12111XXbaXXAx=B⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎣⎡=⎥⎦⎤⎢⎣⎡⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎣⎡.........111'3'2'1321XXXbaXXXoverconstrained2min BAx −Solving for homographies• Can set scale factor i=1. So, there are 8 unkowns.• Set up a system of linear equations:• Ah = b• where vector of unknowns h = [a,b,c,d,e,f,g,h]T• Note: we do not know w but we can compute it from x & yw=gx+hy+1• The equations are linear in the unknown⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡=⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡1yxihgfedcbawwy'wx'p’ = HpSolving for homographies• Can set scale factor i=1. So, there are 8 unkowns.• Set up a system of linear equations:• Ah = b• where vector of unknowns h = [a,b,c,d,e,f,g,h]T• Need at least 8 eqs, but the more the better…• Solve for h. If overconstrained, solve using least-squares: • Can be done in Matlab using “\” command– see “help lmdivide”⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡=⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡1yxihgfedcbawwy'wx'p’ = Hp2min bAh −Panoramas1. Pick one image (red)2. Warp the other images towards it (usually, one by one)3. blendRecap• Panorama = reprojection• 3D rotation Î homography– Homogeneous coordinates are kewl• Use feature correspondence• Solve least square problem– Se of linear equations• Warp all images to a reference one• Use your favorite blendingchanging camera center• Does it still work?synthetic PPPP1PP2Nodal point• http://www.reallyrightstuff.com/pano/index.htmlPlanar mosaicCool applications of homographies• Oh, Durand & DorseyLimitations of 2D Clone Brushing• Distortions due to foreshortening and surface orientationClone brush (Photoshop)• Click on a reference pixel (blue)• Then start painting somewhere else• Copy pixel color with a translationPerspective clone brushOh, Durand, Dorsey, unpublished• Correct for perspective• And other tricksRotational Mosaics• Can we say something more about rotationalmosaics?• i.e. can we further constrain our H?3D → 2D