C280 Computer VisionC280, Computer VisionProf. Trevor [email protected]@eecs.berkeley.eduLt 2 I FtiLecture 2: Image FormationAdministriviaAdministrivia•We’re now in 405 SodaWe re now in 405 Soda…• New office hours: Thurs. 5‐6pm, 413 Soda.’ll did ili dii• I’ll decide on waitlist decisions tomorr ow.• Any Matlab issues yet?• Roster…Physical parameters of image formation•Geometric•Geometric– Type of projection–Camera posep• Optical– Sensor’s lens type– focal length, field of view, aperture• PhotometricT di ti it it f li ht hi–Type, direction, intensity of light reaching sensor– Surfaces’ reflectance properties•SensorSensor– sampling , etc.Physical parameters of image formation•Geometric•Geometric– Type of projection–Camera posep• Optical– Sensor’s lens type– focal length, field of view, aperture• PhotometricT di ti it it f li ht hi–Type, direction, intensity of light reaching sensor– Surfaces’ reflectance properties•SensorSensor– sampling , etc.Perspective and art• Use of correct perspective projection indicated in 1stcentury BC frescoescentury B.C. frescoes• Skill resurfaces in Renaissance: artists develop systematic methods to determine perspectivesystematic methods to determine perspective projection (around 1480‐1515)Durer, 1525RaphaelK. GraumanPerspective projection equations• 3d world mapped to 2d projection in image planeImage planeFocal lthlengthCamera frameOptical axisScene / world points‘’‘’Forsyth and PonceScene pointImage coordinatesHomogeneous coordinatesIs this a linear transformation?• no—division by z is nonlinearTrick: add one more coordinate:homogeneous image homogeneous scene coordinatescoordinatesConverting from homogeneous coordinatesgfgSlide by Steve SeitzPerspective Projection Matrix•Projection is a matrix multiplication using homogeneous coordinates:coordinates:⎤⎡⎥⎤⎢⎡⎤⎡0001xx⎥⎥⎥⎤⎢⎢⎢⎡=⎥⎥⎥⎢⎢⎢⎥⎥⎥⎤⎢⎢⎢⎡00100001yxzy)','(zyfzxf⇒divide by the third coordinate to convert back to non‐h di t⎥⎦⎢⎣⎥⎥⎦⎢⎢⎣⎥⎦⎢⎣'/10'/100fzzfzzhomogeneous coordinatesComplete mapping from world points to image pixelSlide by Steve SeitzComplete mapping from world points to image pixel positions?Perspective projection & calibration• Perspective equations so far in terms of camera’sref erence frame….• Camera’s intrinsic and extrinsic parameters needed to calibrate geometry.Camera frameK. GraumanPerspective projection & calibrationExtrinsic:Camera frameÅÆWorld frameWorld frameIntrinsic:Image coordinates relative to cameraCamera frame ÅÆWorld frameCamera frameImage coordinates relative to camera ÅÆ Pixel coordinatesWorld to camera coord. trans. matrixPerspectiveprojection matrix(3x4)Camera to pixel coord. trans. matrix =2Dpoint(3x1)3Dpoint(4x1)(4x4)(3x4)(3x3)(3x1)()K. GraumanIntrinsic parameters: from idealized world coordinates to pixel valuespForsyth&Poncexfu=Perspective projectionyfvzf=zfW. FreemanIntrinsic parametersxuα=But “pixels” are in some bi il iyvzα=arbitrary spatial unitszW. FreemanIntrinsic parametersxuα=Maybe pixels are not squareyvzβ=zβW. FreemanIntrinsic parameters0uxu+=αWe don’t know the origin of il di00vyvz+=βour camera pixel coordinates0zβW. FreemanIntrinsic parametersv v′ ′θu u′ vv)()()sin(θθθ′′=′)(yxθMay be skew between camera pixel axesvuvuu)cot()cos(θθ−=′−=′0)cot( yuzyzxu+−=βθαα0)sin(vzyv+=θβW. FreemanIntrinsic parameters, homogeneous coordinates0)cot( yuzyzxu +−=βθαα0 )sin(vzyv+=θβ⎛⎞t(θ)⎛ ⎞ x⎛ ⎞ Using homogenous coordinates,we can write this as:uv⎛ ⎜ ⎜ ⎜⎞ ⎟ ⎟ ⎟=α−αcot(θ)u00βi(θ)v000⎛ ⎜ ⎜ ⎜⎞ ⎟ ⎟ ⎟xy⎛ ⎜ ⎜ ⎜ ⎞ ⎟ ⎟ ⎟ we can write this as:1⎝ ⎜ ⎜ ⎠ ⎟ ⎟ sin(θ)0010⎝ ⎜ ⎠ ⎟ z1⎝ ⎜ ⎜ ⎠ ⎟ ⎟ or:ppCrr K =In camera‐based coordsIn pixelsW. FreemanExtrinsic parameters: translation d tti f fand rotation of camer a frametpRpCWWCWCrrr+= Non‐homogeneous coordinatesHomogeneous ⎟⎞⎜⎛⎟⎞⎜⎛−−−⎟⎞⎜⎛|coordinates⎟⎟⎟⎟⎟⎜⎜⎜⎜⎜⎟⎟⎟⎟⎜⎜⎜⎜−−−−−=⎟⎟⎟⎟⎟⎜⎜⎜⎜⎜ptRpWCWCWCrrr|⎟⎟⎠⎜⎜⎝⎟⎠⎜⎝⎟⎟⎠⎜⎜⎝1000W. FreemanCombining extrinsic and intrinsic calibration parameters in homogeneous coordinatesparameters, in homogeneous coordinatesCIntrinsicpixelsppCrrK =⎟⎞⎜⎛⎟⎞⎜⎛−−−⎟⎞⎜⎛|World coordinatesCamera coordinatespExtrinsic⎟⎟⎟⎟⎟⎞⎜⎜⎜⎜⎜⎛⎟⎟⎟⎟⎞⎜⎜⎜⎜⎛−−−−−=⎟⎟⎟⎟⎟⎞⎜⎜⎜⎜⎜⎛ptRpWCWCWCrrr||⎟⎟⎠⎜⎜⎝⎟⎠⎜⎝⎟⎟⎠⎜⎜⎝1000|()ptRKpWCWCWrrr =0 0 0 1Forsyth&PoncepMpWrr =W. FreemanOther wa ys to write the same equationpixel coordinatesworld coordinatespMpWrr =⎟⎞⎜⎛⎞⎛⎞⎛xWTpmuPmurr⋅=1⎟⎟⎟⎟⎜⎜⎜⎜⎟⎟⎟⎞⎜⎜⎜⎛=⎟⎟⎟⎞⎜⎜⎜⎛......21WyWxTTppmmvuPmPmurr⋅⋅23⎟⎟⎠⎜⎜⎝⎟⎠⎜⎝⎟⎠⎜⎝1...13zTpmPmmvr⋅=32Conversion back from homogeneous coordinatesConversion back from homogeneous coordinates leads to:W. FreemanCalibration targetFind the position, uiand vi, in pixels, of eachhttp://www.kinetic.bc.ca/CompVision/opti‐CAL.htmlFind the position, uiand vi, in pixels, of each calibration object feature point.Camera calibrationPmPmurr⋅⋅=31From before, we had these equations relating image positions,u,v, to points at 3‐d positions P (in homogeneous PmPmvrr⋅⋅=32coordinates):0)(PrSo for each feature point, i, we have:0)(0)(3231=⋅−=⋅−iiiiPmvmPmumrW. FreemanCamera calibration0)(31=⋅−iiPmumrrStack all these measurements of i=1…n points 0)(32=⋅−iiPmvmrinto a big matrix:⎞⎛⎟⎞⎜⎛−00PuPTTT⎟⎟⎟⎞⎜⎜⎜⎛⎟⎟⎞⎜⎜⎛⎟⎟⎟⎞⎜⎜⎜⎛−−00001111111MmPvPPuPTTT⎟⎟⎟⎟⎜⎜⎜⎜=⎟⎟⎠⎜⎜⎝⎟⎟⎟⎟⎜⎜⎜⎜−00032MLLLmmPuPTnnTTn⎟⎟⎠⎜⎜⎝⎟⎠⎜⎝−00 PvPTnnTnTW. Freeman⎟⎟⎟⎞⎜⎜⎜⎛=⎟⎟⎞⎜⎜⎛⎟⎟⎟⎟⎞⎜⎜⎜⎜⎛−−000021111111MLLLmmPvPPuPTTTTTTIn vector form:Camera calibration⎟⎟⎟⎟⎠⎜⎜⎜⎜⎝⎟⎟⎠⎜⎜⎝⎟⎟⎟⎠⎜⎜⎜⎝−−000032MmmPvPPuPTnnTnTTnnTTn⎞⎛⎟⎟⎟⎟⎞⎜⎜⎜⎜⎛131211mmmShowing all the