1CSE486, Penn StateRobert CollinsLecture 18:Generalized Stereo: Epipolar GeometryCSE486, Penn StateRobert CollinsGeneralized StereoKey idea: Any two images showing an overlapping view of the world can be treated as a stereo pair...... we just have to figure out how the two views are related.Some of the most “beautiful” math in vision concernsdescribing how multiple views are related, geometrically.CSE486, Penn StateRobert CollinsRecall: Epipolar ConstraintImportant Stereo Vision Concept:Given a point in the left image, we don’t have to search the whole right image for a corresponding point.The “epipolar constraint” reduces the search spaceto a one-dimensional line.CSE486, Penn StateRobert CollinsReview : Simple Stereo SystemLeft cameraRight camerazzxxyyzzxxyyTTxxSame Y Coord!Epipolar linePEquation relatingdepth and disparitybaselinedisparitydepthCSE486, Penn StateRobert CollinsReview: Epipolar ConstraintCorresponding features are constrained tolie along conjugate epipolar lines (on thesame row in the case of our simple setup).CSE486, Penn StateRobert CollinsGeneral StereoOO22OO11zz11zz22xx11yy11yy22xx22CC11CC22In general, the cameras may be related byIn general, the cameras may be related byan arbitrary transformation (R,T)an arbitrary transformation (R,T)In general, intrinsic camera parametersIn general, intrinsic camera parametersmay be different, and even unknownmay be different, and even unknownEpipolarEpipolarMatrixMatrixFundamental MatrixFundamental Matrix2CSE486, Penn StateRobert CollinsEPIPOLAR GEOMETRYCSE486, Penn StateRobert CollinsEpipolar GeometryeerreellOOrrpprrPPppllOOllEpipolesEpipoles::••eell: left image of O: left image of Orr••eerr: right image of : right image of OOllEpipolarEpipolarplane:plane:••Three points: Three points: OOll,O,Orr, and P define an , and P define an epipolarepipolarplaneplaneEpipolarEpipolarlines and lines and epipolarepipolarconstraint:constraint:••Intersections of Intersections of epipolarepipolarplane with the image planesplane with the image planes••Corresponding points are on Corresponding points are on ““conjugateconjugate””epipolarepipolarlineslinesThe following slides are from Dr.Camps, PSUCSE486, Penn StateRobert CollinsBORING!!!Let’s try again...CSE486, Penn StateRobert CollinsEPIPOLAR GEOMETRYCSE486, Penn StateRobert CollinsEpipolar GeometryA VisualizationWould would Pinhead’s eye look like close up?CSE486, Penn StateRobert CollinsRays to Points in SceneTie threads onto the pinsand connectfocal pointto scenepointsNow what would this look like to a second observer?3CSE486, Penn StateRobert CollinsRays Seen from Second ObserverQ uic kT im e™ a nd aTI FF ( LZ W ) de co m pr ess orar e ne ed ed t o se e th is pict ur e.Image 2epipoleepipolar linesCSE486, Penn StateRobert CollinsRays Seen by the First ViewerImage 1epipoleepipolar linesCSE486, Penn StateRobert CollinsEpipolar Geometryimage1image 2Epipole : location of cam2as seen by cam1.Epipole : location of cam1as seen by cam2.CSE486, Penn StateRobert CollinsEpipolar Geometryimage1image 2Corresponding pointslie on conjugate epipolar linesCSE486, Penn StateRobert CollinsEpipolar Geometryimage1image 2Conjugate epipolar lines inducea generalized 1D “scan-line” orderingon the images (analogous to traditionalscan line ordering of rows in an image) CSE486, Penn StateRobert CollinsEpipole not Necessarily in ImageQ uic kT im e™ a nd aTI FF ( LZ W ) de co m pr ess orar e ne ed ed t o se e th is pict ur e.Image 2epipoleepipolar lines4CSE486, Penn StateRobert CollinsEpipolar GeometryeerreellOOrrpprrPPppllOOllEpipolesEpipoles::••eell: left image of O: left image of Orr••eerr: right image of : right image of OOllEpipolarEpipolarplane:plane:••Three points: Three points: OOll,O,Orr, and P define an , and P define an epipolarepipolarplaneplaneEpipolarEpipolarlines and lines and epipolarepipolarconstraint:constraint:••Intersections of Intersections of epipolarepipolarplane with the image planesplane with the image planes••Corresponding points are on Corresponding points are on ““conjugateconjugate””epipolarepipolarlineslinesThe following slides are from Dr.Camps, PSUCSE486, Penn StateRobert CollinsEpipolar Constraint:Given Given EpipolesEpipoles::••eell: left image of O: left image of Orr••eerr: right image of : right image of OOllGiven pGiven pll::••consider its consider its epipolarepipolarline: pline: plleell••find find epipolarepipolarplane: plane: OOll,p,pll,e,ell••intersect the intersect the epipolarepipolarplane with the right image planeplane with the right image plane••search for psearch for prron the right on the right epipolarepipolarlinelineeerreellOOrrppllOOllPPpprrCSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTPPllPPrrDoes this look familiar? Recall world to cameratransformation by (R,T). Here, we are transformingfrom camera to camera.CSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTPPllPPrrCSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTEpipolarEpipolarconstraint: Pconstraint: Pll, T and P, T and Pll--T are coplanar:T are coplanar:PPllPPrrCSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTEpipolarEpipolarconstraint: Pconstraint: Pll, T and P, T and Pll--T are coplanar:T are coplanar:PPllPPrr5CSE486, Penn StateRobert CollinsVector Product as a Matrix MultiplicationS has rank 2 ; it depends only on TS has rank 2 ; it depends only on TCSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTEpipolarEpipolarconstraint: Pconstraint: Pll, T and P, T and Pll--T are coplanar:T are coplanar:PPllPPrrCSE486, Penn StateRobert CollinsEssential MatrixeerreellOOrrpprrPPppllOOllTTEpipolarEpipolarconstraint: Pconstraint: Pll, T and P, T and Pll--T are coplanar:T are coplanar:PPllPPrrEssential Matrix:Essential Matrix:CSE486, Penn StateRobert CollinsEssential Matrix Properties• has rank 2• depends only on the EXTRINSIC Parameters (R & T)We will discuss more of the wonderful properties of this matrix next