6.891Computer Vision and ApplicationsProf. Trevor. Darrell• Class overview• Administrivia & Policies• Lecture 1– Perspective projection (review)– Rigid motions (review)– Camera CalibrationReadings: Forsythe & Ponce, 1.1, 2.1, 2.2, 2.3, 3.1, 3.2Vision• What does it mean, to see? “to know what is where by looking”.• How to discover from images what is present in the world, where things are, what actions are taking place.from Marr, 1982Why study Computer Vision?• One can “see the future” (and avoid bad things…)!• Images and movies are everywhere; fast-growing collection of useful applications– building representations of the 3D world from pictures– automated surveillance (who’s doing what)– movie post-processing– face finding• Greater understanding of human vision• Various deep and attractive scientific mysteries– how does object recognition work?Why study Computer Vision?• People draw distinctions between what is seen– “Object recognition”– This could mean “is this a fish or a bicycle?”– It could mean “is this George Washington?”– It could mean “is this poisonous or not?”– It could mean “is this slippery or not?”– It could mean “will this support my weight?”– Great mystery• How to build programs that can draw useful distinctions based on image properties.Computer vision class, fast-forwardCameras, lenses, and sensorsFrom Computer Vision, Forsyth and Ponce, Prentice-Hall, 2002.•Pinhole cameras•Lenses•Projection models•Geometric camera parametersImage filtering• Review of linear systems, convolution• Bandpass filter-based image representations• Probabilistic models for imagesImageOriented, multi-scale representationFrom Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995ColorModels of textureA Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients J. Portilla and E. Simoncelli, International Journal of Computer Vision 40(1): 49-71, October 2000.© Kluwer Academic Publishers. Parametric modelNon-parametric modelA. Efros and W. T Freeman, Image quilting for texture synthesis and transfer, SIGGRAPH 2001Statistical classifiers– MIT Media Lab face localization results.– Applications: database search, human machine interaction, video conferencing.Multi-view GeometryWhat are the relationships between images of point features in more than one view?Given a point feature in one camera view, predict it’s location in a second (or third) camera?Ego-Motion / “Match-move”Where are the cameras?Track points, estimate consistent poses…Render synthetic objects in real world!Ego-Motion / “Match-move”VideoSee “Harts War” and other examples in Gallery of examples for Matchmove program at www.realviz.comStructure from MotionWhat is the shape of the scene?SegmentationHow many ways can you segment six points?(or curves)Segmentation• Which image components “belong together”?• Belong together=lie on the same object• Cues– similar colour– similar texture– not separated by contour– form a suggestive shape when assembledTrackingFollow objects and estimate location..– radar / planes– pedestrians– cars– face features / expressionsMany ad-hoc approaches…General probabilistic formulation: model density over time.Tracking• Use a model to predict next position and refine using next image• Model:– simple dynamic models (second order dynamics)– kinematic models– etc.• Face tracking and eye tracking now work rather wellArticulated ModelsFind most likely model consistent with observations….(and previous configuration)Articulated tracking• Constrained optimization• Coarse-to-fine part iteration• Propagate joint constraints through each limb• Real-time on Ghz pentium…VideoAnd…• Visual Category Learning• Image Databases• Image-based Rendering• Visual Speechreading• Medical ImagingAdministrivia• Syllabus• Grading• Collaboration Policy• ProjectGrading• Two take-home exams • Five problem sets with lab exercises in Matlab • No final exam • Final projectCollaboration PolicyProblem sets may be discussed, but all written work and coding must be done individually. Take-home exams may not be discussed. Individuals found submitting duplicate or substantially similar materials due to inappropriate collaboration may get an F in this class and other sanctions.ProjectThe final project may be – An original implementation of a new or published idea – A detailed empirical evaluation of an existing implementation of one or more methods – A paper comparing three or more papers not covered in class, or surveying recent literature in a particular area A project proposal not longer than two pages must be submitted and approved by April 1st.Problem Set 0• Out today, due 2/12• Matlab image exercises– load, display images– pixel manipulation– RGB color interpolation– image warping / morphing with interp2– simple background subtraction• All psets graded loosely: check, check-, 0.• (Outstanding solutions get extra credit.)Cameras, lenses, and calibrationToday:• Camera models (review)• Projection equations (review)You should have been exposed to this material in previous courses; this lecture is just a (quick) review.• Calibration methods (new)7-year old’s questionWhy is there no image on a white piece of paper?Virtual image, perspective projection• Abstract camera model - box with a small hole in itForsyth&PonceImages are two-dimensional patterns of brightness values.They are formed by the projection of 3D objects.Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.Animal eye: a looonnng time ago.Pinhole perspective projection: Brunelleschi, XVthCentury.Camera obscura: XVIthCentury.Photographic camera:Niepce, 1816.Reproduced by permission, the American Society of Photogrammetry andRemote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry,Thompson, Radlinski, and Speert (eds.), third edition, 1966.Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.The equation of projection°°¯°°® zyfyzxfx''''Distant objects are smallerForsyth&Ponce• Points go to points• Lines go to lines• Planes go to whole image or half-planes.• Polygons go to polygons• Degenerate