13D Computer Visionand Video Computing3D Vision3D VisionCSc I6716Fall 2011Topic 4 of Part II Visual MotionCover Image/video credits: Rick Szeliski, MSRZhigang Zhu, City College of New York [email protected] Computer Visionand Video ComputingOutline of Motion Outline of Motion  Problems and Applications The importance of visual motion Problem Statement The Motion Field of Rigid MotionBasics–Notations and EquationsBasics Notations and Equations Three Important Special Cases: Translation, Rotation and Moving Plane Motion Parallax Optical Flow Optical flow equation and the aperture problem Estimating optical flow 3D motion & structure from optical flow Feature-based ApproachTwo-frame algorithmTwoframe algorithm Multi-frame algorithm Structure from motion – Factorization method Advanced Topics  Spatio-Temporal Image and Epipolar Plane Image Video Mosaicing and Panorama Generation Motion-based Segmentation and Layered Representation23D Computer Visionand Video ComputingThe Importance of Visual MotionThe Importance of Visual Motion Structure from Motion Apparent motion is a strong visual clue for 3D reconstruction More than a multi-camera stereo system Recognition by motion (only)  Biological visual systems use visual motion to infer properties of 3D world with little a priori knowledge of it Blurred image sequence Visual Motion = Video ! [Go to CVPR 2004-2010 Sites for Workshops]Video Coding and Compression: MPEG 1 2 4 7Video Coding and Compression: MPEG 1, 2, 4, 7… Video Mosaicing and Layered Representation for IBR Surveillance (Human Tracking and Traffic Monitoring) HCI using Human Gesture (video camera) Image-based Rendering …3D Computer Visionand Video ComputingBlurred SequenceBlurred SequenceRecognition by Actions: Recognize object from motion even if we cannot distinguish it in any images …An up-sampling from images of resolution 15x20 pixels From: James W. Davis. MIT Media Lab33D Computer Visionand Video ComputingProblem StatementProblem Statement Two Subproblems Correspondence: Which elements of a frame correspond to which elements in the next frame?RttiGi b f d d iblReconstruction :Given a number of correspondences, and possibly the knowledge of the camera’s intrinsic parameters, how to recovery the 3-D motion and structure of the observed world Main Difference between Motion and Stereo Correspondence: the disparities between consecutive frames are much smaller due to dense temporal sampling Reconstruction: the visual motion could be caused by multiple motions ( instead of a single 3D rigid transformation) The Third Subproblem, and Fourth…. Motion Segmentation: what are the regions the the image plane corresponding to different moving objects? Motion Understanding: lip reading, gesture, expression, event…3D Computer Visionand Video ComputingApproachesApproaches Two Subproblems Correspondence:Differential Methods->dense measure (optical flow)Differential Methods >dense measure (optical flow) Matching Methods -> sparse measure Reconstruction : More difficult than stereo since  Motion (3D transformation betw. Frames) as well as structure needs to be recovered Small baseline causes large errors The Third Subprobleme d Subp ob e Motion Segmentation: Chicken and Egg problem Which should be solved first? Matching or Segmentation Segmentation for matching elements Matching for Segmentation43D Computer Visionand Video ComputingThe Motion Field of Rigid ObjectsThe Motion Field of Rigid Objects Motion:  3D Motion ( R, T): camera motion (static scene) or single object motionor single object motion  Only one rigid, relative motion between the camera and the scene (object) Image motion field: 2D vector field of velocities of the image points induced by the relative motion. Data: Image sequence Many framesy captured at time t=0, 1, 2, … Basics: only consider two consecutive frames We consider a reference frame and its consecutive frame Image motion field can be viewed disparity map of the two frames captured at two consecutive camera locations ( assuming we have a moving camera)3D Computer Visionand Video ComputingThe Motion Field of Rigid ObjectsThe Motion Field of Rigid Objects Notations P = (X,Y,Z)T: 3-D point in the camera reference frame p = (x,y,f)T: the projection of the scene point in the pinhole cameraPpZfpp Relative motion between P and the camera T= (Tx,Ty,Tz)T: translation component of the motion xyz: the angular velocity Note:How to connect this with stereo geometryPωTV PVHow to connect this with stereo geometry (with R, T)? Image velocity v= ?pOXfZYv53D Computer Visionand Video ComputingThe Motion Field of Rigid ObjectsThe Motion Field of Rigid Objects Notations P = (X,Y,Z)T: 3-D point in the camera reference frame p = (x,y,f)T: the projection of the scene point in the pinhole cameraPpZfpoint in the pinhole camera Relative motion between P and the camera T= (Tx,Ty,Tz)T: translation component of the motion xyz: the angular velocityNote:PωTV PTVPP000xyxzyzNote: How to connect this with stereo geometry (with R, T)?TPP 111xyxzyzcoscoscossinsinsincossinsincossincoscossincoscossinsinsinsincoscossinsinsinsincoscoscosR3D Computer Visionand Video ComputingBasic Equations of Motion FieldBasic Equations of Motion Field Notes: Take the time derivative of both sides of the projection equation)(2PVvzVZZf The motion field is the sum of two components Translational part  Rotational part Assume known intrinsic parametersPpZfPωTVzyxzyxyxTTTyfxfZfxxyfyfyfxxyfvv001)(12222Rotation part: no depth informationTranslation part: depth Z63D Computer Visionand Video ComputingMotion Field vs. DisparityMotion