Computational VisionU. Minn. Psy 5036Daniel KerstenLecture 26: Spatial Layout, Structure from MotionInitializeIn[1]:=Off@General::"spell1"D;SetOptions@ArrayPlot, ColorFunction Ø "GrayTones", DataReversed Ø True,Frame Ø False, AspectRatio Ø Automatic, Mesh Ø False,PixelConstrained Ø True, ImageSize Ø SmallD;OutlineLast timePerceptual integrationTodayComputational theory for estimating relative depth, camera motionChallenges to computational theories of depth and spatial layoutSpatial layout: Where are objects? Where is the viewer?Recall distinctions: Between vs. within object geometry.Where are objects?‡AbsoluteDistance of objects or scene feature points from the observer. "Physiological cues": Binocular convergence--information about the distance between the eyes and the angle converged by the eyes. Crude, but constraining. Errors might be expected to be proportional to reciprocal distance. Closely related to accommodative requirements."Pictorial cue"--familiar size‡RelativeDistance between objects or object feature points. Important for scene layout. Processes include: Stereopsis (binocular parallax) and motion parallax. Also information having to do with the "pictorial" cues: occlusion, transparency, perspective, proximity luminance, focus blur, also familiar size & "assumed common physical size", "height in picture plane", cast shadows, texture & texture gradients for large-scale depth & depth gradients‡Examples of pictorial information for depth‡Cooperative computation & cue integration...over a dozen cues to depth. Theories of integration (e.g. stereo + cast shadows). Theories of cooperativity (e.g. motion parallax <=> transparency).Vision for spatial layout of objects, navigation, heading and for reach2 26.SpatialLayoutScenes.nbWhere is the viewer? And where is the viewer headed?Computing scene structure from motion information provides information for vision. Can't say where the viewer is in absolute terms, but can say something about the relative depth relationships between objects, and can say something about heading direction, and time to contact.Calculating structure from motion and heading from the motion fieldEstimation of relative depth and viewer/camera motionIntroductionEarlier we saw: 1) how local motion measurements constrain estimates of optic flow, and thus the motion field.2) how a priori slowness and smoothness contraints constrain dense and sparse estimates of the flow field (e.g. Weiss et al.).How can we use an estimate of the motion field to estimate useful information for navigation--such as relative depth, observer motion, and time to collision? And all in an instant!GoalsEstimate relative depth of points in a scene, the viewer’s motion from the motion field, and estimates of the viewer’s time-to-contact with a surface.Ultimately we would like to gain some understanding of the environment from the moving images on our retinas. There are approaches to structure from motion that are not based directly on the motion field, but rather based on a sequence of images in which a discrete set of corresponding points have been identified (Ullman, S., 1979; Dickmanns). A major challenge is to track the corresponding points. Alternatively, suppose we have estimated the optic flow at time t, and assume it is a good estimate of the motion field--what can we do with it? Imagine the observer is flying through the environment. The flow field should be rich with information regarding direction of heading, time-to-contact, and relative depth (Gibson, 1957). In this section we study the computational theory for the estimation of relative depth, and viewer/camera heading from the optic flow pattern induced by general eye motion in a rigid environment. We follow a development described by Longuet-Higgins, H. C., & Prazdny, K. (1980). (See also Koenderink and van Doorn, 1976, Horn, Chapter 17, Perrone, 1992 for a biologically motivated model, and Heeger and Jepson, 1990).Rather than following the particular derivation of Longuet-Higgins et al., we derive the relationship between the motion field and relative depth, and camera motion parameters using homogeneous coordinates. 26.SpatialLayoutScenes.nb 3Setting up the frame of reference and basic variablesImagine a rigid coordinate system attached to the eye, with the origin at the nodal point. General motion of the eye can be described by the instantaneous translational (U,V,W) and rotational (A,B,C) components of the frame. Let P be a fixed point in the world at (X,Y,Z) that projects to point (x,y) in the conjugate image plane which is unit distance in the z direction from the origin:Goal 1: Derive generative model of the motion field, where we express the motion field (u,v) in terms of Z, U,V,W,A,B,C.‡Homogeneous coordinates (review from Lecture 23)First we’ll review homogeneous coordinates.Rotation and scaling can be done by linear matrix operations in three-space. Translation and perspective transformations do not have a three dimensional matrix representation. By going from three dimensions to four dimensional homogeneous coordinates, all four of the basic operations, rotation, scaling, translation and perspective projection, can be represented using the formalism of matrix multiplication. Homogeneous coordinates are defined by: {xw, yw, zw, w }, (w not equal to 0). To get from homogeneous coordinates to three-space coordinates, {x,y,z}, divide the first three homogeneous coordinates by the fourth, {w}. The rotation and translation matrices can be used to describe object or eye-point changes of position. The scaling matrix allows you to squash or expand objects in any of the three directions. Any combination of the matrices can be multiplied together or concatenated. But remember, matrices do not in general commute when multiplying, so the order is important. The translation, rotation, and perspective transformation matrices can be concatenated to describe general 3-D to 2-D perspective mappings.4 26.SpatialLayoutScenes.nbIn[8]:=XRotationMatrix@theta_D :=881, 0, 0, 0<, 80, Cos@thetaD, Sin@thetaD, 0<,80, - Sin@thetaD, Cos@thetaD, 0<, 80, 0, 0, 1<<;YRotationMatrix@theta_D :=88Cos@thetaD, 0, - Sin@thetaD, 0<, 80, 1, 0, 0<,8Sin@thetaD, 0, Cos@thetaD, 0<, 80, 0, 0, 1<<;ZRotationMatrix@theta_D :=88Cos@thetaD, Sin@thetaD, 0, 0<, 8- Sin@thetaD, Cos@thetaD, 0, 0<,80, 0, 1, 0<, 80, 0, 0, 1<<;ScaleMatrix@sx_, sy_, sz_D :=88sx, 0, 0, 0<, 80, sy, 0, 0<, 80, 0, sz, 0<, 80, 0, 0,