Visual Simulation CAP 6938TodayCamera calibrationCamera calibration – approachesImage Formation EquationsCalibration matrixCamera matrixCamera matrix calibrationSlide 9Optimal estimationSlide 11Levenberg-MarquardtSlide 13Slide 14Slide 15Separate intrinsics / extrinsicsSlide 17Intrinsic/extrinsic calibrationVanishing PointsSlide 20Slide 21Vanishing point calibrationMulti-plane calibrationRotational motionPose estimation and triangulationPose estimationSlide 27Slide 28TriangulationSlide 30Slide 31Structure from MotionStructure from motionSlide 34Bundle AdjustmentLots of parameters: sparsityConditioning and gauge freedomRobust error modelsStructure from motion: limitationsBibliographySlide 41Slide 42Slide 43Slide 44Slide 45Visual SimulationCAP 6938Dr. Hassan Foroosh Dept. of Computer ScienceUCF© Copyright Hassan Foroosh 2002TodayLast LectureFeature TrackingStructure from MotionTomasi and KanadeExtensionsTodayCamera CalibrationBundle adjustmentCamera calibrationDetermine camera parameters from known 3D points or calibration object(s)1. internal or intrinsic parameters such as focal length, optical center, aspect ratio:what kind of camera?2. external or extrinsic (pose)parameters:where is the camera?3. How can we do this?Camera calibration – approachesPossible approaches:1. linear regression (least squares)2. non-linear optimization3. vanishing points4. multiple planar patterns5. panoramas (rotational motion)Image Formation Equationsu(Xc,Yc,Zc)ucfCalibration matrixIs this form of K good enough?non-square pixels (digital video)skewradial distortionCamera matrixFold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrixM = K [R | t ](put 1 in lower r.h. corner for 11 d.o.f.)Camera matrix calibrationDirectly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)Camera matrix calibrationLinear regression:Bring denominator over, solve set of (over-determined) linear equations. How?Least squares (pseudo-inverse)Is this good enough?Optimal estimationFeature measurement equationsLikelihood of M given {(ui,vi)}Optimal estimationLog likelihood of M given {(ui,vi)}How do we minimize C?Non-linear regression (least squares), because ûi and vi are non-linear functions of MLevenberg-MarquardtIterative non-linear least squares [Press’92]Linearize measurement equationsSubstitute into log-likelihood equation: quadratic cost function in mIterative non-linear least squares [Press’92]Solve for minimumHessian:error:Does this look familiar…?Levenberg-MarquardtWhat if it doesn’t converge?Multiply diagonal by (1 + ), increase until it doesHalve the step size m (my favorite)Use line searchOther ideas?Uncertainty analysis: covariance  = A-1Is maximum likelihood the best idea?How to start in vicinity of global minimum?Levenberg-MarquardtCamera matrix calibrationAdvantages:very simple to formulate and solvecan recover K [R | t] from M using RQ decomposition [Golub & VanLoan 96]Disadvantages:doesn't compute internal parametersmore unknowns than true degrees of freedomneed a separate camera matrix for each new viewSeparate intrinsics / extrinsicsNew feature measurement equationsUse non-linear minimizationStandard technique in photogrammetry, computer vision, computer graphics[Tsai 87] – also estimates 1 (freeware @ CMU) [Bogart 91] – View CorrelationSeparate intrinsics / extrinsicsHow do we parameterize R and R?Euler angles: bad ideaquaternions: 4-vectors on unit sphereuse incremental rotation R(I + R)update with Rodriguez formulaIntrinsic/extrinsic calibrationAdvantages:can solve for more than one camera pose at a timepotentially fewer degrees of freedomDisadvantages:more complex update rulesneed a good initialization (recover K [R | t] from M)Vanishing PointsDetermine focal length f and optical center (uc,vc) from image of cube’s(or building’s) vanishing points[Caprile ’90][Antone & Teller ’00]uu00uu11uu22Vanishing PointsX, Y, and Z directions, Xi = (1,0,0,0) … (0,0,1,0) correspond to vanishing points that are scaled version of the rotation matrix:uu00uu11uu22u(Xc,Yc,Zc)ucfVanishing PointsOrthogonality conditions on rotation matrix R,ri ¢rj = ijDetermine (uc,vc) from orthocenter of vanishing point triangleThen, determine f2 from twoequations(only need 2 v.p.s if (uc,vc) known)uu00uu11uu22Vanishing point calibrationAdvantages:only need to see vanishing points(e.g., architecture, table, …)Disadvantages:not that accurateneed rectahedral object(s) in sceneMulti-plane calibrationUse several images of planar target held at unknown orientations [Zhang 99]Compute plane homographiesSolve for K-TK-1 from Hk’s1plane if only f unknown2 planes if (f,uc,vc) unknown3+ planes for full KCode available from Zhang and OpenCVRotational motionUse pure rotation (large scene) to estimate f1. estimate f from pairwise homographies2. re-estimate f from 360º “gap”3. optimize over all {K ,Rj} parameters[Stein 95; Hartley ’97; Shum & Szeliski ’00; Kang & Weiss ’99]Most accurate way to get f, short of surveying distant pointsf=510 f=468Pose estimation and triangulationPose estimationOnce the internal camera parameters are known, can compute camera pose[Tsai87] [Bogart91]Application: superimpose 3D graphics onto videoHow do we initialize (R,t)?Pose estimationPrevious initialization techniques:vanishing points [Caprile 90]planar pattern [Zhang 99]Other possibilitiesThrough-the-Lens Camera Control [Gleicher92]: differential update3+ point “linear methods”:[DeMenthon 95][Quan 99][Ameller 00]Pose estimationSolve orthographic problem, iterate[DeMenthon 95]Use inter-point distance constraints[Quan 99][Ameller 00]Solve set of polynomial equations in xi2pu(Xc,Yc,Zc)ucfTriangulationProblem: Given some points in correspondence across two or more images (taken from calibrated cameras), {(uj,vj)}, compute the 3D location XTriangulationMethod I: intersect viewing rays in 3D, minimize:X is the unknown 3D pointCj is the optical center of camera jVj is the viewing ray for pixel (uj,vj)sj is unknown distance along VjAdvantage: geometrically intuitiveCjVjXTriangulationMethod II: