C280 Computer VisionC280, Computer VisionProf. Trevor [email protected] 10: StereoRoadmap•Previous: Image formation filtering local features (Texture)•Previous: Image formation, filtering, local features, (Texture)…• Tues: Feature‐based Alignment –Stitching images together– Homographies, RANSAC, Warping, Blending– Global alignment of planar models•Today: Dense Motion Models•Today: Dense Motion Models– Local motion / feature displacement– Parametric optic flow• No classes next week: ICCV conference• Oct 6th: Stereo / ‘Multi‐view’: Estimating depth with known itinter‐camera pose• Oct 8th: ‘Structure‐from‐motion’: Estimation of pose and 3D structure– Factorization approaches– Global alignment with 3D point modelsLast time: Motion and FlowLast time: Motion and FlowMti ti ti•Motion estimation•Patch‐based motion (optic flow)• Regularization and line processes•Parametric (global)Parametric (global) motion•Layered motion models•Layered motion modelsToday: StereoToday: Stereo•Humanstereopsis&stereograms•Human stereopsis& stereograms• Epipolar geometry and the epipolar constraint– Case example with parallel optical axes– General case with calibrated cameras• Correspondence search• The Essential and the Fundamental Matrixe sse t a adte uda eta at• Multi‐view stereoFixation, con vergenceFixation, con vergenceGraumanHuman stereopsis: disparityHuman stereopsis: disparityDisparity occurs when eyes fi t bj t thfixate on one object; others appear at different visual anglesgHuman stereopsis: disparityd=0Disparity: d = r‐l = D‐F.Adapted from M. PollefeysRandom dot stereogramsRandom dot stereograms•Julesz 1960: Do we identify local bright nessJulesz 1960: Do we identify local bright ness patterns before fusion (monocular process) or after (binocular)?after (binocular)? T i f hii bi db•To test: pair of synthetic images obtained by randomly spraying black dots on white objectsRandom dot stereogramsRandom dot stereogramsForsyth & PonceRandom dot stereogramsRandom dot stereogramsRandom dot stereogramsRandom dot stereogramsFrom Palmer, “Vision Science”, MIT PressRandom dot stereogramsRandom dot stereograms•When viewed monocularly, they appear random;When viewed monocularly, they appear random; when viewed stereoscopically, see 3d structure.• Conclusion: human binocular fusion not directly associated with the physical retinas; must involve the pycentral nervous system• Imaginary “cyclopean retina” that combines the left and right image stimuli as a single unitGraumanAutostereogramsAutostereogramsExploit disparity as depthExploit disparity as depth cue using single image(Single image random dot (g gstereogram, Single image stereogram)Images from magiceye.comAutostereogramsAutostereogramsImages from magiceye.comStereo photography and stereo viewersTake two pictures of the same subject from two slightly different viewpoints and display so that each eye sees only one of the p py y yimages.Invented by Sir Charles Wheatstone, 1838Image courtes y of fisher‐price.comGraumanhttp://www.johnsonshawmuseum.orgGraumanhttp://www.johnsonshawmuseum.orgGraumanPublic Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923Graumanhttp://www.well.com/~jimg/stereo/stereo_list.htmlGraumanDepth with stereo: basic ideaDepth with stereo: basic ideaiiscene pointscene pointimage planeimage planeoptical centeroptical centerSource: Steve SeitzDepth with stereo: basic ideaDepth with stereo: basic ideaBasic Principle: Triangulation•Gives reconstruction as intersection of two rays•Gives reconstruction as intersection of two rays• Requires –camera pose (calibration)–point correspondenceSource: Steve SeitzCamera calibrationExtrinsicparameters:World frameIntrinsic parameters:pCamera frame  Ref erence frameCamera frameImage coordinates relative to camer a  Pixel coordinates• Extrinsic params: rotation matrix and translation vector• Intrinsic params: focal length, pixe l sizes (mm), image center point, radial distortion parametersWe’ll assume for now that these parameters are given f p gand fixed.GraumanTodayToday• Human stereopsisp• StereogramsEi l t d th il tit•Epipolar geometry and the epipolar constraint– Case example with parallel optical axes– General case with calibrated cameras• Stereopsis– Finding correspondences along the epipolar lineGeometry for a simple stereo system• First, assuming parallel optical ax es, known camer a parameters (i e calibrated cameras):parameters (i.e., calibrated cameras):World pointpointDepth of pimage point (left)image point (right)Focal lengthoptical center (left)optical center (right)lengthbaselinecenter (left)(right)Geometry for a simple stereo system• Assume parallel optical ax es, known camer a parameters (i.e., calibrated camer as). We can triangulate via:Similar triangles (pl, P, pr) and (Ol, P, Or):,r)TxxTrlZfZlxxTfZlrxxdisparityGraumanDisparity exampleimage I(x y)image I´(x´y´)Disparity map D(x y)Disparity exampleimage I(x,y)image I(x,y)Disparity map D(x,y)(x´y´)=(x+D(x y) y)(x,y)=(x+D(x,y), y)GraumanGeneral case, with calibrated camer as • The two cameras need not have parallel optical ax es.Vs.GraumanStereo correspondence constraints•Given p in left image, where can corresponding pointGiven p in left image, where can corresponding point p’ be?Stereo correspondence constraints•Given p in left image, where can corresponding pointGiven p in left image, where can corresponding point p’ be?Stereo correspondence constraintsStereo correspondence constraints•Geometry of two view s allows us to constrain where the corresponding pixel for some image point in the first view must occur in the second view.epipolar planeepipolar lineepipolar lineepipolar lineepipolar lineEpipolar constr aint: Why is this useful?Epipolar constraint: Why is this useful?• Reduces correspondence problem to 1D search along conjugateepipolar linesAdapted from Steve SeitzEpipolar geometry• Epipolar Plane • BaselineAdapted from M. Pollefeys, UNC• Epipoles • Epipolar LinesEpipolar geometry: terms• Baseline: line joining the camera centers• Epipole: point of intersection of baseline with the image plane• Epipolar plane: plane containing baseline and world point•Epipolar line: intersection of epipolar plane with the image•Epipolar line: intersection of epipolar plane with the image plane•