Lecture 6: Image transformations and alignmentAnnouncementAnnouncementsAnnouncementsReadingsAnnouncementsProject 1 questionsLast time: projectionPerspective projectionPerspective projectionExtrinsicsExtrinsicsExtrinsicsExtrinsicsProjection matrixProjection matrixPerspective distortionPerspective distortionDistortionCorrecting radial distortionModeling distortionOther types of projection360 degree field of view…Rotating sensor (or object)PhotofinishQuestions?Today: Image transformations and alignmentWhy Mosaic?Why Mosaic?Why Mosaic?Mosaics: stitching images togetherReadingsImage WarpingImage WarpingParametric (global) warpingParametric (global) warpingCommon linear transformationsCommon linear transformations2x2 Matrices2x2 MatricesAll 2D Linear TransformationsTranslationAffine transformationsBasic affine transformationsAffine TransformationsProjective TransformationsProjective TransformationsImage warping with homographies2D image transformationsImage WarpingForward WarpingForward WarpingInverse WarpingInverse WarpingInterpolationQuestions?Back to mosaicsCreating a panoramaGeometric Interpretation of MosaicsImage reprojectionWhat is the transformation?What is the transformation?Can we use homography to create a 360 panorama?PanoramasSpherical projectionSpherical reprojectionAligning spherical imagesAligning spherical imagesQuestions?Lecture 6: Image transformations and alignmentCS6670: Computer VisionNoah SnavelyAnnouncement• New TA! Adarsh Kowdle• Office hours: M 11‐12, Ward Laboratory 112Announcements• Project 1 out, due Thursday, 9/24, by 11:59pm • Quiz on Thursday, first 10 minutes of class• Next week: guest lecturer, Prof. Pedro Felzenszwalb, U. ChicagoAnnouncements• Project 2 will be released on Tuesday• You can work in groups of two – Send me your groups by Friday eveningReadings• Szeliski Chapter 3.5 (image warping), 9.1 (motion models)Announcements• A total of 3 late days will be allowed for projectsProject 1 questionsLast time: projectionPerspective projectionProjection is a matrix multiply using homogeneous coordinates:divide by third coordinateEquivalent to:Perspective projection(intrinsics)in general, : aspect ratio (1 unless pixels are not square): skew (0 unless pixels are shaped like rhombi/parallelograms): principal point ((0,0) unless optical axis doesn’t intersect projection plane at origin)(upper triangular matrix)(converts from 3D rays in camera coordinate system to pixel coordinates)Extrinsics• How do we get the camer a to “canonical form”?– (Center of projection at the origin, x‐axis points right, y‐axis points up, z‐axis points backwards)0Step 1: Translate by ‐cExtrinsics• How do we get the camer a to “canonical form”?– (Center of projection at the origin, x‐axis points right, y‐axis points up, z‐axis points backwards)0Step 1: Translate by ‐cHow do we represent translation as a matrix multiplication?Extrinsics• How do we get the camer a to “canonical form”?– (Center of projection at the origin, x‐axis points right, y‐axis points up, z‐axis points backwards)0Step 1: Translate by ‐cStep 2: Rotate by R3x3 rotation matrixExtrinsics• How do we get the camer a to “canonical form”?– (Center of projection at the origin, x‐axis points right, y‐axis points up, z‐axis points backwards)0Step 1: Translate by ‐cStep 2: Rotate by RProjection matrix(t in book’s notation)translationrotationprojectionintrinsicsProjection matrix0=(in homogeneous image coordinates)Perspective distortion• What does a sphere project to?Image source: F. DurandPerspective distortion• What does a sphere project to?Distortion• Radial distortion of the image– Caused by imperfect lenses– Deviations are most noticeable for rays that pass through the edge of the lensNo distortion Pin cushion BarrelCorrecting radial distortionfrom Helmut DerschModeling distortion• To model lens distortion– Use above projection operation instead of standard projection matrix multiplicationApply radial distortionApply focal length translate image centerProject to “normalized” image coordinatesOther types of projection• Lots of intriguing variants…• (I’ll just mention a few fun ones)360 degree field of view…• Basic approach– Take a photo of a parabolic mirror with an orthographic lens (Nayar)– Or buy one a lens from a variety of omnicam manufacturers…• See http://www.cis.upenn.edu/~kostas/omni.htmlRollout Photographs © Justin Kerr http://research.famsi.org/kerrmaya.htmlRotating sensor (or object)Also known as “cyclographs”, “peripheral images”PhotofinishQuestions?Today: Image transformations and alignmentFull screen panoramas (cubic): http://www.panoramas.dk/Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.htmlWhy Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°Slide from Brown & LoweWhy Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°Slide from Brown & LoweWhy Mosaic?• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°Slide from Brown & LoweMosaics: stitching images togetherReadings• Szeliski:– Chapter 3.5: Image warping– Chapter 5.1: Feature‐based alignment– Chapter 8.1: Motion modelsRichard Szeliski Image Stitching 34Image Warping• image filtering: change range of image• g(x) = h(f(x))• image warping: change domain of image• g(x) = f(h(x))fxhgxfxhgxRichard Szeliski Image Stitching 35Image Warping• image filtering: change range of image• g(x) = h(f(x))• image warping: change domain of image• g(x) = f(h(x))hhffggRichard Szeliski Image Stitching 36Parametric (global) warping• Examples of parametric warps:translationrotationaspectaffineperspectivecylindricalParametric (global) warping• Transformation T is a coordinate‐changing machine:p’ = T(p)• What does it mean that T is global?– Is the same for any point p– can be described by just a few numbers (parameters)• Let’s consider linear xforms (can be represented by a 2D matrix):Tp = (x,y) p’ = (x’,y’)Common linear transformations• Uniform scaling by s:(0,0)(0,0)What is the inverse?Common linear transformations• Rotation by angle θ (about the origin)(0,0)(0,0)What is the inverse?For