Olivier Gondry 6 098 Digital and Computational Photography 6 882 Advanced Computational Photography Director of music video and commercial Special effect specialist Morphing rotoscoping Today at 5 40pm in 32 141 Panoramas Bill Freeman Fr do Durand MIT EECS Lots of slides stolen from Alyosha Efros who stole them from Steve Seitz and Rick Szeliski Why Mosaic Why Mosaic Are you getting the whole picture Compact Camera FOV 50 x 35 Are you getting the whole picture Compact Camera FOV 50 x 35 Human FOV 200 x 135 Slide from Brown Lowe Why Mosaic Slide from Brown Lowe Mosaics stitching images together Are you getting the whole picture Compact Camera FOV 50 x 35 Human FOV 200 x 135 Panoramic Mosaic 360 x 180 Slide from Brown Lowe virtual wide angle camera 1 How to do it A pencil of rays contains all views Basic Procedure Take a sequence of images from the same position real camera Rotate the camera about its optical center Compute transformation between second image and first Transform the second image to overlap with the first Blend the two together to create a mosaic If there are more images repeat but wait why should this work at all What about the 3D geometry of the scene Why aren t we using it Aligning images translation left on top synthetic camera Can generate any synthetic camera view as long as it has the same center of projection Image reprojection right on top Translations are not enough to align the images mosaic PP The mosaic has a natural interpretation in 3D The images are reprojected onto a common plane The mosaic is formed on this plane Mosaic is a synthetic wide angle camera Image reprojection Back to Image Warping Which t form is the right one for warping PP1 into PP2 Basic question e g translation Euclidean affine projective How to relate 2 images from same camera center how to map a pixel from PP1 to PP2 PP2 Answer Cast a ray through each pixel in PP1 Draw the pixel where that ray intersects PP2 Translation Affine Perspective 2 unknowns 6 unknowns 8 unknowns PP1 But don t we need to know the geometry of the two planes in respect to the eye Observation Rather than thinking of this as a 3D reprojection think of it as a 2D image warp from one image to another 2 Homography 1D homogeneous coordinates Projective mapping between any two PPs with the same center of projection rectangle should map to arbitrary quadrilateral parallel lines aren t but must preserve straight lines same as project rotate reproject called Homography wx x wy y w 1 To apply a homography H p H p Add one dimension to make life simpler x w represent point x w w PP2 w 1 PP1 Compute p Hp regular matrix multiply Convert p from homogeneous to image coordinates x 1D homography 1D homography Reproject to different line Reproject to different line w w w 1 w 1 x x 1D homography Same in 2D Reproject to different line Equivalent to rotating 2D points reprojection is linear in homogeneous coordinates Reprojection homography 3x3 matrix w w 1 wx x wy y w 1 p H p PP2 PP1 x 3 Image warping with homographies image plane in front Digression perspective correction image plane below black area where no pixel maps to From Photography London et al From Photography London et al From Photography London et al Tilt shift lens 35mm SLR version From Photography London et al 4 Photoshop version perspective crop Back to Image rectification you control reflection and perspective independently p p To unwarp rectify an image Find the homography H given a set of p and p pairs How many correspondences are needed Tricky to write H analytically but we can solve for it Find such H that best transforms points p into p Use least squares Least Squares Example Solving for homographies Say we have a set of data points X1 X1 X2 X2 X3 X3 etc e g person s height vs weight We want a nice compact formula line to predict X s from Xs Xa b X We want to find a and b How many X X pairs do we need X 1a b X 1 X 1 1 a X 1 Ax B X 2a b X 2 X2 1 b X2 What if the data is noisy X1 X 2 X3 1 1 a 1 b X 1 X2 X 3 min Ax B 2 p Hp a wx wy d g w b e h c x f y i 1 Can set scale factor i 1 So there are 8 unkowns Set up a system of linear equations Ah b where vector of unknowns h a b c d e f g h T Note we do not know w but we can compute it from x y w gx hy 1 The equations are linear in the unknown overconstrained Solving for homographies Panoramas p Hp a wx wy d g w b e h c x f y i 1 Can set scale factor i 1 So there are 8 unkowns Set up a system of linear equations Ah b where vector of unknowns h a b c d e f g h T Need at least 8 eqs but the more the better Solve for h If overconstrained solve using least squares 2 min Ah b Can be done in Matlab using command see help lmdivide 1 Pick one image red 2 Warp the other images towards it usually one by one 3 blend 5 Recap changing camera center Panorama reprojection 3D rotation homography Homogeneous coordinates are kewl Use feature correspondence Solve least square problem Se of linear equations Warp all images to a reference one Use your favorite blending Does it still work Nodal point Planar mosaic synthetic PP PP1 PP2 http www reallyrightstuff com pano index html Cool applications of homographies Limitations of 2D Clone Brushing Oh Durand Dorsey Distortions due to foreshortening and surface orientation 6 Clone brush Photoshop Perspective clone brush Click on a reference pixel blue Then start painting somewhere else Copy pixel color with a translation Oh Durand Dorsey unpublished Correct for perspective And other tricks Rotational Mosaics Can we say something more about rotational mosaics i e can we further constrain our H 3D 2D Perspective Projection Xc Yc Zc f uc u 3D Rotation Model 1 2 3 Projection equations Project from image to 3D ray x0 y0 z0 u0 uc v0 vc f Rotate the ray by camera motion R01 x0 y0 z0 x1 y1 z1 Project back into new source image u1 v1 fx1 z1 uc fy1 z1 vc Therefore x y z u v f R x y z f u v f H K0R01K1 1 K Our homography has only 3 4 or 5 DOF depending if focal length is known same or different This makes image registration much better behaved 7 Pairwise alignment Rotation about vertical axis Procrustes Algorithm Golub VanLoan Given two sets of matching points compute R with 3D rays pi R pi pi N xi yi zi …