CSE486, Penn StateRobert CollinsLecture 20:The Eight-Point AlgorithmReadings T&V 7.3 and 7.4CSE486, Penn StateRobert CollinsReminder: -0.00310695 -0.0025646 2.96584 -0.028094 -0.00771621 56.3813 13.1905 -29.2007 -9999.79F =CSE486, Penn StateRobert CollinsEssential/Fundamental MatrixThe essential and fundamental matrices are 3x3 matricesthat “encode” the epipolar geometry of two views.Motivation: Given a point in one image, multiplyingby the essential/fundamental matrix will tell us which epipolar line to search along in the second view.CSE486, Penn StateRobert CollinsE/F Matrix SummaryLonguet-Higgins equationEpipolar lines:Epipoles:E vs F: E works in film coords (calibrated cameras) F works in pixel coords (uncalibrated cameras)CSE486, Penn StateRobert CollinsComputing F from Point Matches• Assume that you have m correspondences• Each correspondence satisfies:• F is a 3x3 matrix (9 entries)• Set up a HOMOGENEOUS linear system with 9unknownsCSE486, Penn StateRobert CollinsComputing FCSE486, Penn StateRobert CollinsComputing FCSE486, Penn StateRobert CollinsComputing FGiven m point correspondences…Think: how many points do we need?CSE486, Penn StateRobert CollinsHow Many Points?Unlike a homography, where each point correspondencecontributes two constraints (rows in the linear system ofequations), for estimating the essential/fundamental matrix,each point only contributes one constraint (row). [because the Longuet-Higgins / Epipolar constraint is a scalar eqn.]Thus need at least 8 points.Hence: The Eight Point algorithm!CSE486, Penn StateRobert CollinsSolving Homogeneous SystemsAssume that we need the non trivial solution of:with m equations and n unknowns, m >= n – 1 andrank(A) = n-1Since the norm of x is arbitrary, we will look fora solution with norm ||x|| = 1Self-studyCSE486, Penn StateRobert CollinsLeast Square solutionWe want Ax as close to 0 as possible and ||x|| =1:Self-studyCSE486, Penn StateRobert CollinsOptimization with constraintsDefine the following cost:This cost is called the LAGRANGIAN cost andλ is called the LAGRANGIAN multiplierThe Lagrangian incorporates the constraintsinto the cost function by introducing extravariables.Self-studyCSE486, Penn StateRobert CollinsOptimization with constraintsTaking derivatives wrt to x and λ:•The first equation is an eigenvector problem• The second equation is the original constraintSelf-studyCSE486, Penn StateRobert CollinsOptimization with constraints•x is an eigenvector of ATA with eigenvalue λ: eλ•We want the eigenvector with smallest eigenvalueSelf-studyCSE486, Penn StateRobert CollinsWe can find the eigenvectors andeigenvalues of ATA by finding theSingular Value Decomposition of ASelf-studyCSE486, Penn StateRobert CollinsSingular Value Decomposition(SVD)Any m x n matrix A can be written as the product of 3Any m x n matrix A can be written as the product of 3matrices:matrices:Where:Where:•• U is m x m and its columns are U is m x m and its columns are orthonormal orthonormal vectorsvectors•• V is n x n and its columns are V is n x n and its columns are orthonormal orthonormal vectorsvectors•• D is m x n diagonal and its diagonal elements are called D is m x n diagonal and its diagonal elements are calledthe singular values of A, and are such that:the singular values of A, and are such that: σσ11 ¸¸ σσ22 ¸¸ …… σσn n ¸¸ 0 0Self-studyCSE486, Penn StateRobert CollinsSVD Properties•• The columns of U are the eigenvectors of AA The columns of U are the eigenvectors of AATT•• The columns of V are the eigenvectors of A The columns of V are the eigenvectors of ATTAA•• The squares of the diagonal elements of D are the The squares of the diagonal elements of D are the eigenvalues eigenvalues of AAof AATT and A and ATTAASelf-studyCSE486, Penn StateRobert CollinsComputing F: The 8 pt AlgorithmA has rank 8•Find eigenvector of ATA with smallest eigenvalue!8 8 8 8 8 8 8 8 8 888CSE486, Penn StateRobert CollinsAlgorithm EIGHT_POINT• Construct the m x 9 matrix A• Find the SVD of A: A = UDVT• The entries of F are the components of thecolumn of V corresponding to the least s.v.The input is formed by m point correspondences, m >= 8CSE486, Penn StateRobert CollinsAlgorithm EIGHT_POINT• Find the SVD of F: F = UfDfVfT• Set smallest s.v. of F to 0 to create D’f• Recompute F: F = UfD’fVfTF must be singular (remember, it is rank 2, since it isimportant for it to have a left and right nullspace, i.e. the epipoles). To enforce rank 2 constraint:CSE486, Penn StateRobert CollinsNumerical Details• The coordinates of corresponding points can have a wide rangeleading to numerical instabilities.• It is better to first normalize them so they have average 0 andstddev 1 and denormalize F at the end:F = (T’)-1 Fn THartley preconditioning algorithm: this was an“optional” topic in Lecture 16. Go back and lookif you want to know more.Self-studyCSE486, Penn StateRobert CollinsA Practical IssueHow to “rectify” the images so that any scan-linestereo alorithm that works for simple stereo can be used to find dense matches (i.e. compute a disparity image for every pixel).general epipolar linesparallel epipolar linesrectifyCSE486, Penn StateRobert CollinsStereo Rectification• Image Reprojection– reproject image planes onto commonplane parallel to line between opticalcenters• Notice, only focal point of camera really mattersSeitz, UWCSE486, Penn StateRobert CollinsGeneral IdeaApply homography representing avirtual rotation to bring theimage planes parallel with thebaseline (epipoles go to infinity).CSE486, Penn StateRobert CollinsGeneral Idea, continuedCSE486, Penn StateRobert CollinsImage RectificationAssuming extrinsic parameters R & T areknown, compute a 3D rotation that makesconjugate epipolar lines collinear andparallel to the horizontal image axisRemember: a rotation around focal point of camerais just a 2D homography in the image!Method from T&V 7.3.7Note: this method from the book assumes calibrated cameras (we can recover R,T from the E matrix). In a moment, we will see a more general approch that uses F matrix.CSE486, Penn StateRobert CollinsImage Rectification• Rectification involves two rotations:– First rotation sends epipoles to infinity– Second rotation makes epipolar lines parallel• Rotate the left and right cameras with first R1 (constructed from translation