CSCE 641 Computer Graphics Image based Modeling Cont Jinxiang Chai Image based Modeling and Rendering Model Panoroma Image based modeling Images user input range scans Images Depth Geometry Images Imagebased rendering Camera geometry Geometry Materials Light field Kinematics Dynamics Etc Images Stereo reconstruction Given two or more images of the same scene or object compute a representation of its shape known camera viewpoints Stereo reconstruction Given two or more images of the same scene or object compute a representation of its shape known camera viewpoints How to estimate camera parameters where is the camera where is it pointing what are internal parameters e g focal length Spectrum of IBMR Model Panoroma Image based modeling Images user input range scans Images Depth Geometry Images Imagebased rendering Camera geometry Geometry Materials Light field Kinematics Dynamics Etc Images Calibration from 2D motion Structure from motion SFM track points over a sequence of images estimate for 3D positions and camera positions calibrate intrinsic camera parameters before hand Self calibration solve for both intrinsic and extrinsic camera parameters SFM Holy Grail of 3D Reconstruction Take movie of object Reconstruct 3D model Would be commercially highly viable How to Get Feature Correspondences Feature based approach good for images feature detection corners or sift features feature matching using RANSAC epipolar line Pixel based approach good for video sequences patch based registration with lucas kanade algorithm register features across the entire sequence Structure from Motion Two Principal Solutions Bundle adjustment nonlinear optimization Factorization SVD through orthographic approximation affine geometry Projection Matrix Perspective projection ui f x v 0 i 1 0 K R fy 0 u0 r1T v0 r2T 1 r3T T P x t1 i yi t2 zi t3 1 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters f x r1T r2T u0 r3T P f x t1 t 2 u0t3 ui r3T P t3 vi f y r2T v0 r3T P f y t 2 t3 T 3 r P t3 f K R T Pi g K R T Pi Nonlinear Approach for SFM What s the difference between camera calibration and SFM Nonlinear Approach for SFM What s the difference between camera calibration and SFM camera calibration known 3D and 2D M N arg min u K R j T j j 1 i 1 j i f K R j T j Pi 2 vij g K R j T j Pi 2 Nonlinear Approach for SFM What s the difference between camera calibration and SFM camera calibration known 3D and 2D M N arg min u K R j T j j f K R j T j Pi 2 vij g K R j T j Pi 2 i j 1 i 1 SFM unknown 3D and known 2D M N arg min u Pi K R j T j j 1 i 1 j i f K R j T j Pi 2 vij g K R j T j Pi 2 Nonlinear Approach for SFM What s the difference between camera calibration and SFM camera calibration known 3D and 2D M N arg min u K R j T j j f K R j T j Pi 2 vij g K R j T j Pi 2 i j 1 i 1 SFM unknown 3D and known 2D M N arg min u j i f K R j T j Pi 2 vij g K R j T j Pi 2 Pi K R j T j j 1 i 1 what s 3D to 2D registration problem Nonlinear Approach for SFM What s the difference between camera calibration and SFM camera calibration known 3D and 2D M N arg min u K R j T j j f K R j T j Pi 2 vij g K R j T j Pi 2 i j 1 i 1 SFM unknown 3D and known 2D M N arg min u j i f K R j T j Pi 2 vij g K R j T j Pi 2 Pi K R j T j j 1 i 1 what s 3D to 2D registration problem SFM Bundle Adjustment M N arg min u j i f K R j T j Pi 2 vij g K R j T j Pi 2 Pi K R j T j j 1 i 1 SFM Nonlinear Least Squares problem Minimize through Gradient Descent Conjugate Gradient Gauss Newton Levenberg Marquardt common method Prone to local minima Count Constraints vs Unknowns M N arg min u j i f K R j T j Pi 2 vij g K R j T j Pi 2 Pi K R j T j j 1 i 1 M camera poses N points 2MN point constraints 6M 3N unknowns Suggests need 2mn 6m 3n But Can we really recover all parameters Count Constraints vs Unknowns M N arg min u j i f K R j T j Pi 2 vij g K R j T j Pi 2 Pi K R j T j j 1 i 1 M camera poses N points 2MN point constraints 6M 3N unknowns known intrinsic camera parameters Suggests need 2mn 6m 3n But Can we really recover all parameters Can t recover origin orientation 6 params Can t recover scale 1 param Thus we need 2mn 6m 3n 7 Are We Done No bundle adjustment has many local minima SFM Using Factorization Assume an othorgraphic camera Image xi ui r1T t1 yi v T t z i r2 2 i 1 World SFM Using Factorization Assume othorgraphic camera Image xi ui r1T t1 yi v T t z i r2 2 i 1 Subtract the mean World ui vi u i i 1 r T xi N 1 y N r2T i vi zi i 1 N N SFM Using Factorization Stack all the features from the same frame x1 x2 x N T u1 u2 u N r1 y y y N v v v T 1 2 N 1 2 r2 z z z N 1 2 SFM Using Factorization Stack all the features from the same frame x1 x2 x N T u1 u2 u N r1 y y y N v v v T 1 2 N 1 2 r2 z z z N 1 2 Stack all the features from all the images T u F 1 u F 2 u F N r1 1 v v r T x1 x2 x N v F N 1 2 F 1 F 2 y y y N T 1 2 u F 1 u F 2 u F N rF 1 z1 z 2 z N v v v r T F N F 1 F 2 F 2 W SFM Using Factorization Stack all the features from the same frame x1 x2 x N T u1 u2 u N r1 y y y N v v v T 1 2 N 1 2 r2 z z z N 1 2 Stack all the features from all the images T u F 1 u F 2 u F N r1 1 v v r T …