1CSE152, Spr 04 Intro Computer VisionRecognition IIntroduction to Computer VisionCSE 152Lecture 18CSE152, Spr 04 Intro Computer VisionAnnouncements• Assignment 4: Due Today• Assignment 5: Posted today• Read: Trucco & Verri, Chapter 10 on recognition• Final Exam: Wed, 6/9/04, 11:30-2:30, WLH 2207 (here)CSE152, Spr 04 Intro Computer VisionVirtual Cinematography: Making 'The Matrix' SequelsGeorge BorshukovVFX Technology Supervisor, ESC EntertainmentFriday, June 4, 20041:00 p.m. to 2:30 p.m.[Pizza lunch will precede the event from noon to 1 p.m.]Main Auditorium, San Diego Supercomputer CenterThe presentation will cover the key technologies that had to be developed and deployed to create the synthetic human sequences in the Matrix sequels including Universal Capture - image-based facial animation, realistic human face rendering, and use of measured BRDF in film production. It will also feature a breakdown of The Superpunch shot (pictured above) from "The Matrix Revolutions" (the bullet time punch that Neo delivers to Agent Smith during the film's last face-off). This difficult, important, expensive, and challenging shot was entirely computer generated and showcased the technological developments of 3.5+ years at their best by showing a full-frame close-up of a known human actor. CSE152, Spr 04 Intro Computer VisionOptical Flow:Where do pixels move to?CSE152, Spr 04 Intro Computer VisionMathematical formulation[Note change of notation: image coordinates now (x,y), not (u,v)]I (x,y,t) = brightness at image point (x,y) at time tOptical flow constraint equation :0 =∂∂+∂∂+∂∂=tIdtdyyIdtdxxIdtdIConsider scene (or camera) to be moving, so x(t), y(t)),,(),,( tyxItttdtdyytdtdxxI =+++δδδBrightness constancy assumption:0=dtdICSE152, Spr 04 Intro Computer Vision, ∂∂=yIIx, ∂∂=yIIytIIt∂∂=Measurements, =dtdxudtdyv =Flow vector2CSE152, Spr 04 Intro Computer VisionTwo ways to get flow1. Think globally, and regularize over image2. Look over window and assume constant motion in the windowCSE152, Spr 04 Intro Computer Vision()()∑∑=++==++=02),(02),(tyxytyxxIvIuIIdvvudEIvIuIIduvudEΩxyCSE152, Spr 04 Intro Computer VisionRecognitionCSE152, Spr 04 Intro Computer VisionRecognitionGiven a database of objects and an image determine what, if any of the objects are present in the image.CSE152, Spr 04 Intro Computer VisionRecognition Challenges• Within-class variability– Different objects within the class have different shapes or different material characteristics– Deformable– Articulated– Compositional• Pose variability: – 2-D Image transformation (translation, rotation, scale)– 3-D Pose Variability (perspective, orthographic projection)• Lighting– Direction (multiple sources & type)–Color–Shadows• Occlusion – partial• Clutter in background -> false positivesCSE152, Spr 04 Intro Computer VisionObject Recognition Issues:• How general is the problem?–2D vs. 3D– range of viewing conditions– available context– segmentation cues• What sort of data is best suited to the problem?– Whole images– Local 2D features (color, texture, – 3D (range) data• What information do we have in the database?– Collection of images?– 3-D models?– Learned representation?– Learned classifiers?• How many objects are involved?– small: brute force search–large: ??3CSE152, Spr 04 Intro Computer VisionA Rough Recognition SpectrumAppearance-BasedRecognition(Eigenface, Fisherface)Local Features +Spatial Relations3-DModel-BasedRecognitionGeometricInvariantsImageAbstractions/ Volumetric PrimitivesShape ContextsFunctionAspect GraphsIncreasing GeneralityCSE152, Spr 04 Intro Computer VisionAppearance-Based Vision:A Pattern Classification Viewpoint1. Feature Space + Nearest Neighbor2. Dimensionality Reduction3. Bayesian Classification4. Appearance ManifoldsCSE152, Spr 04 Intro Computer VisionSketch of a Pattern Recognition ArchitectureFeatureExtractionClassificationImage(window)ObjectIdentityFeature VectorCSE152, Spr 04 Intro Computer VisionExample: Face Detection• Scan window over image.• Classify window as either:– Face– Non-faceClassifierWindowFaceNon-faceCSE152, Spr 04 Intro Computer VisionThe Problem of RecognitionGiven an image Given an image IIand a database of and a database of kkobjects and a objects and a representation representation RRjjfor object for object jjin the database, recognitionin the database, recognitioncan be expressed as:can be expressed as:where where c(Rc(Rj,j,,I,I))is a function which gives the compatibilityis a function which gives the compatibilityor consistency of representation or consistency of representation RRjjwith the image.with the image.),(minarg],,1[IRcijkj L∈=CSE152, Spr 04 Intro Computer VisionImage as a Feature Vector• Consider an n-pixel image (window) to be a point in an n-dimensional space, x ∈ Rn.• Each pixel value is a coordinate of x.x1x2x34CSE152, Spr 04 Intro Computer VisionSimplest Recognition Scheme••RRjjis an image.••c(Rc(Rjj, I) , I) is Euclidean distance.Nearest NeighborClassifierx1x2x3RR11RR22IICSE152, Spr 04 Intro Computer VisionComments• Sometimes called “Template Matching”• Variations on distance function (e.g. L1, robust distances)• Multiple templates per class- perhaps many training images per class.• Expensive to compute k distances, especially when each image is big (N dimensional).• May not generalize well to unseen examples of class.• Some solutions:– Dimensionality reduction– Bayesian classificationCSE152, Spr 04 Intro Computer VisionEigenfaces: Linear Projection•An n-pixel image x∈Rncan be projected to a low-dimensional feature space y∈Rmbyy = Wxwhere W is an m by n matrix.• Recognition is performed using nearest neighbor in Rm.• How do we choose a good W?CSE152, Spr 04 Intro Computer VisionEigenfaces: Principal Component Analysis (PCA)Some details: Use Singular value decomposition, “trick” described in appendix of text to compute basis when n<<dCSE152, Spr 04 Intro Computer VisionMeanFirst Principal ComponentDirection of Maximum VarianceCSE152, Spr 04 Intro Computer VisionEigenfaces• Modeling1. Given a collection of n labeled training images,2. Compute mean image and covariance matrix.3. Compute k Eigenvectors (note that these are images) of covariance matrix corresponding to k largest Eigenvalues. (Or perform using SVD!!)4. Project the training images to the k-dimensional Eigenspace.• Recognition1. Given a test