More Mosaic MadnessHomographyRotational Mosaics3D → 2D Perspective Projection3D Rotation ModelPairwise alignmentRotation about vertical axisDo we have to project onto a plane?Full PanoramasCylindrical projectionCylindrical ProjectionInverse Cylindrical projectionCylindrical panoramasCylindrical image stitchingAssembling the panoramaProblem: DriftFull-view (360°) panoramasSpherical projectionSpherical ProjectionInverse Spherical projection3D rotationFull-view PanoramaOther projections are possibleCylindrical reprojectionWhat’s your focal length, buddy?DistortionRadial distortionPolar ProjectionCamera calibrationApproach 1: solve for projection matrixDirect linear calibrationApproach 2: solve for parametersMulti-plane calibrationMore Mosaic Madness15-463: Computational PhotographyAlexei Efros, CMU, Fall 2008© Jeffrey Martin (jeffrey-martin.com)with a lot of slides stolen from Steve Seitz and Rick SzeliskiHomographyA: Projective – mapping between any two PPs with the same center of projection•rectangle should map to arbitrary quadrilateral •parallel lines aren’t•but must preserve straight lines•same as: project, rotate, reproject called HomographyPP2PP11yx*********wwy'wx'H pp’ To apply a homography H•Compute p’ = Hp (regular matrix multiply)•Convert p’ from homogeneous to image coordinatesRotational MosaicsCan we say something more about rotational mosaics?i.e. can we further constrain our H?3D → 2D Perspective Projectionu(Xc,Yc,Zc)ucfK3D Rotation ModelProjection equations1. Project from image to 3D ray(x0,y0,z0) = (u0-uc,v0-vc,f)2. Rotate the ray by camera motion(x1,y1,z1) = R01 (x0,y0,z0)3. Project back into new (source) image(u1,v1) = (fx1/z1+uc,fy1/z1+vc)Therefore: Our homography has only 3,4 or 5 DOF, depending if focal length is known, same, or different.•This makes image registration much better behaved(u,v,f)(x,y,z)f(x,y,z)(u,v,f)R11010KRKHPairwise alignmentProcrustes Algorithm [Golub & VanLoan]Given two sets of matching points, compute Rpi’ = R pi with 3D rayspi = N(xi,yi,zi) = N(ui-uc,vi-vc,f)A = Σi pi pi’T = Σi pi piT RT = U S VT = (U S UT) RTVT = UT RTR = V UTRotation about vertical axisWhat if our camera rotates on a tripod?What’s the structure of H?mosaic PPDo we have to project onto a plane?Full PanoramasWhat if you want a 360 field of view?mosaic Projection Cylinder•Map 3D point (X,Y,Z) onto cylinderCylindrical projectionXYZunit cylinderunwrapped cylinder•Convert to cylindrical coordinatescylindrical image•Convert to cylindrical image coordinatesCylindrical ProjectionYXInverse Cylindrical projectionXYZ(X,Y,Z)(sin,h,cos)Cylindrical panoramasSteps•Reproject each image onto a cylinder•Blend •Output the resulting mosaicCylindrical image stitchingWhat if you don’t know the camera rotation?•Solve for the camera rotations–Note that a rotation of the camera is a translation of the cylinder!Assembling the panoramaStitch pairs together, blend, then cropProblem: DriftVertical Error accumulation•small (vertical) errors accumulate over time•apply correction so that sum = 0 (for 360° pan.)Horizontal Error accumulation•can reuse first/last image to find the right panorama radiusFull-view (360°) panoramasSpherical projectionunwrapped sphere•Convert to spherical coordinatesspherical image•Convert to spherical image coordinatesXYZ),,(1)ˆ,ˆ,ˆ(222ZYXZYXzyx•Map 3D point (X,Y,Z) onto sphere)ˆ,ˆ,ˆcoscossincos(sin zyxSpherical ProjectionYXInverse Spherical projectionXYZ(x,y,z)(sinθcosφ,cosθcosφ,sinφ)cos φφcos θ cos φsin φ3D rotationRotate image before placing on unrolled sphereXYZ(x,y,z)(sinθcosφ,cosθcosφ,sinφ)cos φφcos θ cos φsin φ_ __ _p = R pFull-view Panorama++++++++Other projections are possibleYou can stitch on the plane and then warp the resulting panorama•What’s the limitation here? Or, you can use these as stitching surfaces •But there is a catch…f = 180 (pixels)Cylindrical reprojectionf = 380f = 280Image 384x300top-down viewFocal length – the dirty secret…What’s your focal length, buddy?Focal length is (highly!) camera dependant•Can get a rough estimate by measuring FOV:•Can use the EXIF data tag (might not give the right thing)•Can use several images together and try to find f that would make them match•Can use a known 3D object and its projection to solve for f•Etc.There are other camera parameters too:•Optical center, non-square pixels, lens distortion, etc.DistortionRadial distortion of the image•Caused by imperfect lenses•Deviations are most noticeable for rays that pass through the edge of the lensNo distortion Pin cushion BarrelRadial distortionCorrect for “bending” in wide field of view lensesUse this instead of normal projectionPolar ProjectionExtreme “bending” in ultra-wide fields of viewCamera calibrationDetermine camera parameters from known 3D points or calibration object(s)1. internal or intrinsic parameters such as focal length, optical center, aspect ratio:what kind of camera?2. external or extrinsic (pose) parameters:where is the camera in the world coordinates?•World coordinates make sense for multiple cameras / multiple imagesHow can we do this?Approach 1: solve for projection matrixPlace a known object in the scene•identify correspondence between image and scene•compute mapping from scene to imageDirect linear calibrationSolve for Projection Matrix using least-squares (just like in homework)Advantages:•All specifics of the camera summarized in one matrix•Can predict where any world point will map to in the imageDisadvantages:•Doesn’t tell us about particular parameters•Mixes up internal and external parameters–pose specific: move the camera and everything breaksProjection equation•The projection matrix models the cumulative effect of all parameters•Useful to decompose into a series of operationsΠXx1************ZYXssysx11010000100001100'0'0311333311333xxxxxxcycxyfsxfs000 TIRΠprojectionintrinsics rotation translationidentity matrixApproach 2: solve for parametersA camera is described by several parameters•Translation T of the
View Full Document