13D Computer Visionand Video Computing3D Vision3D VisionCSc I6716Spring 2011Topic 3 of Part IIStereo VisionpgZhigang Zhu, City College of New York [email protected] Computer Visionand Video ComputingStereo VisionStereo Vision Problemz Infer 3D structure of a scene from two or more images taken from different viewpoints Two primary Sub-problemsz Correspondence problem (stereo match) -> disparity map “Similar” instead of “Same” Occlusion problem: some parts of the scene are visible only in one eyez Reconstruction problem -> 3D What we need to know about the cameras’ parameters Often a stereo calibration problem Lectures on Stereo Visionz Stereo Geometry – Epipolar Geometry (*) z Correspondence Problem (*) – Two classes of approachesz 3D Reconstruction Problems – Three approaches23D Computer Visionand Video ComputingA Stereo PairA Stereo Pair Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D3D?CMU CIL Stereo Dataset : Castle sequencehttp://www-2.cs.cmu.edu/afs/cs/project/cil/ftp/html/cil-ster.html?3D Computer Visionand Video ComputingMore Images…More Images… Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D33D Computer Visionand Video ComputingMore Images…More Images… Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D3D Computer Visionand Video ComputingMore Images…More Images… Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D43D Computer Visionand Video ComputingMore Images…More Images… Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D3D Computer Visionand Video ComputingMore Images…More Images… Problemsz Correspondence problem (stereo match) -> disparity mapz Reconstruction problem -> 3D53D Computer Visionand Video ComputingPart I. Stereo GeometryPart I. Stereo Geometry A Simple Stereo Vision Systemz Disparity Equation z Depth Resolutionz Fixated Stereo System Zero-disparity Horopter Epipolar Geometryz Epipolar lines – Where to search correspondences Epipolar Plane, Epipolar Lines and Epipoles http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.htmlz Essential Matrix and Fundamental Matrix Computing E & F by the Eight-Point Algorithm Computing the Epipoles Stereo Rectification3D Computer Visionand Video ComputingStereo GeometryStereo GeometryObject pointP(X,Y,Z)CentralProjectionRaysVergence Angleplpr Converging Axes – Usual setup of human eyes Depth obtained by triangulation Correspondence problem: pland prcorrespond to the left and right projections of P, respectively.63D Computer Visionand Video ComputingA Simple Stereo SystemA Simple Stereo SystemLEFT CAMERA RIGHT CAMERAbaselineLeft image:referenceRight image:targetdisparitybaselineZw=0 Elevation ZwDepth Z3D Computer Visionand Video ComputingDisparity EquationDisparity EquationP(X,Y,Z)Stereo system with parallel optical axesImage planeDepthImage planedxBfDZ ==Disparity: dx = xr-xl pl(xl,yl)Optical Center Olf = focal lengthLEFT CAMERAB = Baselinef = focal lengthOptical Center Orpr(xr,yr)RIGHT CAMERA73D Computer Visionand Video ComputingDisparity vs. BaselineDisparity vs. BaselineP(X,Y,Z)Stereo system with parallel optical axesImage planeDepthImage planedxBfDZ ==Disparity dx = xr-xl pl(xl,yl)Optical Center Olf = focal lengthLEFT CAMERAB = Baselinef = focal lengthOptical Center Orpr(xr,yr)RIGHT CAMERA3D Computer Visionand Video ComputingDepth AccuracyDepth Accuracy Given the same image localization errorz Angle of cones in the figure Depth Accuracy (Depth Resolution) vs. BaselinezDepth Error∝1/B (Baseline length)Two viewpointsZ1OlOrzDepth Error ∝1/B (Baseline length)z PROS of Longer baseline,  better depth estimationz CONS smaller common FOV Correspondence harder due to occlusion Depth Accuracy (Depth Resolution) vs. Depthz Disparity (>0) ∝ 1/ DepthzDepth Error∝Depth2Z2∂Z2>∂Z1∂Z11zDepth Error ∝Depthz Nearer the point, better the depth estimation An Examplez f = 16 x 512/8 pixels, B = 0.5 mz Depth error vs. depth)(Z 2dxfBZ∂=∂)(ZZ dxfBZ∂=∂Absolute errorRelative error83D Computer Visionand Video ComputingStereo with Converging CamerasStereo with Converging Cameras Stereo with Parallel Axes z Short baseline large common FOV large depth errorz Long baseline small depth error small common FOV More occlusion problems Two optical axes intersect at the FOVpFixation Pointz converging angle θz The common FOV IncreasesLeft right3D Computer Visionand Video ComputingStereo with Converging CamerasStereo with Converging Cameras Stereo with Parallel Axes z Short baseline large common FOVFOV large depth errorz Long baseline small depth error small common FOV More occlusion problems Two optical axes intersect at the pFixation Pointz converging angle θz The common FOV IncreasesLeft right93D Computer Visionand Video ComputingStereo with Converging CamerasStereo with Converging Cameras Two optical axes intersect at the Fixation Pointz converging angle θz The common FOV IncreasesFixation point Disparity propertiesz Disparity uses angle instead of distancez Zero disparity at fixation point and the Zero-disparity horopterz Disparity increases with the distance of objects from the fixation pointsFOVθ >0 : outside of the horopter <0 : inside the horopter Depth Accuracy vs. Depthz Depth Error ∝ Depth2z Nearer the point, better the depth estimationLeft right3D Computer Visionand Video ComputingStereo with Converging CamerasStereo with Converging Cameras Two optical axes intersect at the Fixation Pointz converging angle θz The common FOV IncreasesFixation pointθ Disparity propertiesz Disparity uses angle instead of distancez Zero disparity at fixation point and the Zero-disparity horopterz Disparity increases with the distance of objects from the fixation pointsHoropter >0 : outside of the horopter <0 : inside the horopter Depth Accuracy vs. Depthz Depth Error ∝ Depth2z Nearer the point, better the depth estimationLeft rightαlαrαr = αldα = 0103D Computer Visionand Video ComputingStereo with Converging CamerasStereo with Converging Cameras Two optical axes intersect at the Fixation Pointz converging angle θz The common FOV IncreasesFixation pointθ Disparity