lecture14_stereo2_outline_handoutlecture14_stereo2CS 378 Computer Vision Oct 22, 2009 Outline: Stereopsis and calibration I. Computing correspondences for stereo A. Epipolar geometry gives hard geometric constraint, but only reduces match for a point to be on a line. Other “soft” constraints are needed to assign corresponding points: ‐ Similarity – how well do the pixels match in a local region by the point? o Normalized cross correlation o Dense vs. sparse correspondences o Effect of window size ‐ Uniqueness—up to one match for every point ‐ Disparity gradient—smooth surfaces would lead to smooth disparities ‐ Ordering—points on same surface imaged in order o Enforcing ordering constraint with scanline stereo + dynamic programming (Aside from point‐based matching, or order‐constrained DP, graph cuts can be used to minimize energy function expressing preference for well‐matched local windows and smooth disparity labels.) Sources of error when computing correspondences for stereo B. Examples of applications leveraging stereo ‐ Segmentation with depth and spatial gradients ‐ Body tracking with fitting and depth ‐ Camera+microphone stereo system ‐ Virtual viewpoint video II. Camera calibration A. Estimating projection matrix ‐ Intrinsic and extrinsic parameters; we can relate them to image pixel coordinates and world point coordinates via perspective projection. ‐ Use a calibration object to collect correspondences. ‐ Set up equation to solve for projection matrix when we know the correspondences. B. Weak calibration ‐ When all we have are corresponding image points (and no camera parameters), can solve for the fundamental matrix. This gives epipolar constraint, but unlike essential matrix does not require knowing camera parameters. ‐ Stereo pipeline with weak calibration: must estimate both fundamental matrix and correspondences. Start from correspondences, estimate geometry, refine. 10/22/20091Stereo matchingCalibrationThursday, Oct 22Kristen GraumanUT‐AustinToday• Correspondences, matching for stereo– A few stereo applications• Camera calibration10/22/20092Last time: Estimating depth with stereo• Stereo: shape from “motion” between two views• We need to consider:• Info on camera pose (“calibration”)• Image point correspondences scene pointscene pointoptical optical centercenterimage planeimage planeLast time:Epipolar constraint•Potential matches forphave to lie on the corresponding•Potential matches for phave to lie on the corresponding epipolar line l’.• Potential matches for p’ have to lie on the corresponding epipolar line l.Slide credit: M. Pollefeys10/22/20093An audio camera & epipolar geometrySpherical microphone arrayAdam O' Donovan, Ramani Duraiswami and Jan NeumannMicrophone Arrays as Generalized Cameras for Integrated Audio Visual Processing, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, 2007Spherical microphone arrayAn audio camera & epipolar geometry10/22/20094Correspondence problemMultiple match phypotheses satisfy epipolar constraint, but which is correct? Figure from Gee & Cipolla 1999Correspondence problem• Beyond the hard constraint of epipolar geometry, there are “soft” constraints to help identify corresponding pointsidentify corresponding points– Similarity– Uniqueness– Ordering– Disparity gradientT fi d t h i th i i ill•To find matches in the image pair, we will assume– Most scene points visible from both views– Image regions for the matches are similar in appearance10/22/20095Correspondence problemParallel camera example: epipolar lines areSource: Andrew ZissermanParallel camera example: epipolar lines are corresponding image scanlinesCorrespondence problemIntensity Source: Andrew Zissermanprofiles10/22/20096Correspondence problemNeighborhoods of corresponding points are similar in intensity patterns.Source: Andrew ZissermanNormalized cross correlationSource: Andrew Zisserman10/22/20097Correlation‐based window matchingSource: Andrew ZissermanDense correspondence searchFor each epipolar lineFor each pixel / window in the left image• compare with every pixel / window on same epipolar line in right image• pick position with minimum match cost (e.g., SSD, correlation)Adapted from Li Zhang10/22/20098Textureless regionsTextureless regions are non‐distinct; high ambiguity for matches.Source: Andrew ZissermanEffect of window sizeSource: Andrew Zisserman10/22/20099Effect of window sizeW = 3 W = 20Figures from Li ZhangWant window large enough to have sufficient intensity variation, yet small enough to contain only pixels with about the same disparity.Foreshortening effectsSource: Andrew Zisserman10/22/200910OcclusionSlide credit: David KriegmanSparse correspondence search• Restrict search to sparse set of detected features• Rather than pixel values (or lists of pixel values) use feature descriptor and an associated feature distance• Still narrow search further by epipolar geometry10/22/200911Correspondence problem• Beyond the hard constraint of epipolar geometry, there are “soft” constraints to help identify corresponding pointsidentify corresponding points– Similarity– Uniqueness– Disparity gradient– OrderingUniqueness constraint• Up to one match in right image for every point in left imageFigure from Gee & Cipolla 199910/22/200912Disparity gradient constraint• Assume piecewise continuous surface, so want disparity estimates to be locally smooth Figure from Gee & Cipolla 1999Ordering constraint• Points on same surface (opaque object) will be in same order in both viewsFigure from Gee & Cipolla 199910/22/200913Ordering constraint• Won’t always hold, e.g. consider transparent object, or an occluding surfaceFigures from Fors yth & PonceScanline stereo• Try to coherently match pixels on the entire scanline• Different scanlines are still optimized independentlyLeft image Right imageintensity10/22/200914“Shortest paths” for scan-line stereoLeft imageRight imageII′leftSqLeft occlusiontRightocclusionCan be implemented with dynamic programmingOhta & Kanade ’85, Cox et al. ‘96rightSpsSlide credit: Y. BoykovCoherent stereo on 2D grid• Scanline stereo generates streaking artifacts• Can’t use dynamic programming to find spatially coherent disparities/ correspondences on a 2D grid10/22/200915• Example depth maps (pentagon)Stereo matching as energy
View Full Document
Unlocking...