Computational VisionU. Minn. Psy 5036Daniel KerstenLecture 26: Spatial Layout, Structure from MotionInitializeOutlineLast timeObject recognition continuedTodayComputational theory for estimating relative depth, camera motionChallenges to computational theories of depth and spatial layoutSpatial layout: Where are objects? Where is the viewer?Recall distinctions: Between vs. within object geometry.Where are objects?‡AbsoluteDistance of objects or scene feature points from the observer. "Physiological cues": Binocular convergence--information about the distance between the eyes and the angle converged by the eyes. Crude, but constraining. Errors might be expected to be proportional to reciprocal distance. Closely related to accommodative requirements."Pictorial cue"--familiar size‡RelativeDistance between objects or object feature points. Important for scene layout. Processes include: Stereopsis (binocular parallax) and motion parallax. Also information having to do with the "pictorial" cues: occlusion, transparency, perspective, proximity luminance, focus blur, also familiar size & "assumed common physical size", "height in picture plane", cast shadows, texture & texture gradients for large-scale depth & depth gradients‡Examples of pictorial information for depth‡Cooperative computation & cue integration...over a dozen cues to depth. Theories of integration (e.g. stereo + cast shadows). Theories of cooperativity (e.g. motion parallax <=> transparency).Vision for spatial layout of objects, navigation, heading and for reachWhere is the viewer? And where is the viewer headed?Computing scene structure from motion information provides information for vision. Can't say where the viewer is in absolute terms, but can say something about the relative depth relationships between objects, and can say something about heading direction, and time to contact.2 26.SpatialLayoutScenes.nbCalculating structure from motion and heading from the motion fieldEstimation of relative depth and eye (or camera) motionIntroductionEarlier we saw: 1) how local motion measurements constrain estimates of optic flow, and thus the motion field.2) how a priori slowness and smoothness contraints constrain dense and sparse estimates of the flow field (e.g. Weiss et al.).How can we use an estimate of the motion field to estimate useful information for navigation--such as relative depth, observer motion, and time to collision??GoalsEstimate relative depth, and eye's motion from motion field, estimates of time-to-contactUltimately we would like to gain some understanding of the environment from the moving images on our retinas. There are approaches to structure from motion that are not based directly on the motion field, but rather based on a sequence of images in which a discrete set of corresponding points have been identified (Ullman, S., 1979; Dickmanns). Alternatively, suppose we have estimated the optic flow, and assume it is a good estimate of the motion field--what can we do with it? Imagine the observer is flying through the environment. The flow field should be rich with information regarding direction of heading, time-to-contact, and relative depth (Gibson, 1957). In this section we study the computational theory for the estimation of relative depth, and camera or eye-point heading from the optic flow pattern induced by general eye motion in a rigid environment. We follow a development described by Longuet-Higgins, H. C., & Prazdny, K. (1980). (See also Koenderink and van Doorn, 1976, Horn, Chapter 17, Perrone, 1992 for a biologically motivated model, and Heeger and Jepson, 1990).Rather than following the derivation of Longuet-Higgins et al., we derive the relationship between the motion field and relative depth, and camera motion parameters using homogeneous coordinates. Setting up the frame of reference and basic variablesImagine a rigid coordinate system attached to the eye, with the origin at the nodal point. General motion of the eye can be described by the instantaneous translational (U,V,W) and rotational (A,B,C) components of the frame. Let P be a fixed point in the world at (X,Y,Z) that projects to point (x,y) in the conjugate image plane which is unit distance in the z direction from the origin:26.SpatialLayoutScenes.nb 3Goal 1: Derive generative model of the motion field, where we express the motion field (u,v) in terms of Z, U,V,W,A,B,C.‡Express velocity V of world point P, (X,Y,Z) in terms of motion parameters of the camera frame of referenceLet r(t) represent the position of P in homogeneous coordinates:An instant later, the new coordinates are given by:where infinitesimal rotations and translations are represented by their respective 4x4 matrices. (Note that matrix opera-tions do not in general commute). Then, and4 26.SpatialLayoutScenes.nbUsing similar approximations for the other rotation matrices, and the relationwe will use Mathematica to show that8DX, DY, DZ, 0< = 8X, Y, Z, 1<.0 -DqzDqy0Dqz0 -Dqx0-DqyDqx0 0-Dx -Dy -Dz 0+ higher order termsand derive expressions that describe the 3D velocty of P in camera coordinates.‡Let's use Mathematica to derive this formula and expressions for the velocity of P, i.e. V= (dX/dt, dY/dt, and dZ/dt)XRotationMatrix@theta_D :=881, 0, 0, 0<, 80, Cos@thetaD, -Sin@thetaD, 0<,80, Sin@thetaD, Cos@thetaD, 0<, 80, 0, 0, 1<<;YRotationMatrix@theta_D :=88Cos@thetaD, 0, Sin@thetaD, 0<, 80, 1, 0, 0<,8-Sin@thetaD, 0, Cos@thetaD, 0<, 80, 0, 0, 1<<;ZRotationMatrix@theta_D :=88Cos@thetaD, -Sin@thetaD, 0, 0<, 8Sin@thetaD, Cos@thetaD, 0, 0<,80, 0, 1, 0<, 80, 0, 0, 1<<;TranslateMatrix@x_, y_, z_D :=881, 0, 0, 0<, 80, 1, 0, 0<, 80, 0, 1, 0<, 8x, y, z, 1<<;XRotationMatrix[q]//MatrixForm1 0 0 00 cosHqL -sinHqL 00 sinHqL cosHqL 00 0 0 126.SpatialLayoutScenes.nb 5YRotationMatrix@qD êê MatrixFormcosHqL 0 -sinHqL 00 1 0 0sinHqL 0 cosHqL 00 0 0 1ZRotationMatrix@qD êê MatrixFormcosHqL sinHqL 0 0-sinHqL cosHqL 0 00 0 1 00 0 0 1Recall the Series[] function:??SeriesSeries@f , 8x, x0, n<D generates a power seriesexpansion for f about the point x = x0to order Hx - x0Ln.SeriesAf , 8x, x0, nx<, 9y, y0, ny=E successively finds seriesexpansions with respect to x, then y. àAttributes@SeriesD = 8Protected<Options@SeriesD = 8Analytic Ø True, Assumptions ß $Assumptions<Expand the rotation matrix into a Taylor series:In[69]:=Series[XRotationMatrix[Subscript[Dq, x]],{Subscript[Dq, x],0,1}]//MatrixFormOut[69]//MatrixForm=1 0 0 00 1 + OIDqx2M -Dqx+ OIDqx2M 00 Dqx+ OIDqx2M 1 +