Active Appearance ModelsOverviewWhat are we trying to do?Appearance ModelsFirst approach: Active Shape Model (ASM)First Approach: ASM (cont.)Lessons learnedCombined Appearance ModelsHow to generate a CAMHow to generate a CAM (cont.)How to generate a CAM (cont.)How to generate a CAM (cont.)CAM PropertiesCAM Properties (cont.)CAM Properties (cont.)CAM Properties (cont.)How to interpret unseen exampleHow to interpret unseen example (cont.)AAM: TrainingAAM: Training (cont.)AAM: Training (cont.)AAM: Training (cont.)AAM: SearchStarting approximationStarting approximation (cont.)AAM: Search (cont.)Experimental resultsExperimental results (cont.)Experimental results (cont.)Experimental results (cont.)Experimental results (cont.)Experimental results (cont.)Experimental results (cont.)Constrained AAMsConstrained AAMsConstrained AAMs (cont.)Constrained AAMsConclusionsConclusions (cont.)Active Appearance ModelsEdwards, Taylor, and CootesPresented by Bryan RussellOverview Overview of Appearance Models Combined Appearance Models Active Appearance Model Search Results Constrained Active Appearance ModelsWhat are we trying to do? Formulate model to “interpret” face images– Set of parameters to characterize identity, pose, expression, lighting, etc.– Want compact set of parameters – Want efficient and robust modelAppearance Models Eigenfaces (Turk and Pentland, 1991)– Not robust to shape changes– Not robust to changes in pose and expression Ezzat and Poggio approach (1996)– Synthesize new views of face from set of example views– Does not generalize to unseen facesFirst approach: Active Shape Model (ASM) Point Distribution ModelFirst Approach: ASM (cont.) Training: Apply PCA to labeled images New image– Project mean shape– Iteratively modify model points to fit local neighborhoodLessons learned ASM is relatively fast ASM too simplistic; not robust when new images are introduced May not converge to good solution Key insight: ASM does not incorporate all gray-level information in parametersCombined Appearance Models Combine shape and gray-level variation in single statistical appearance model Goals:– Model has better representational power– Model inherits appearance models benefits– Model has comparable performanceHow to generate a CAM Label training set with landmark points representing positions of key features Represent these landmarks as a vector x Perform PCA on these landmark vectorsHow to generate a CAM (cont.) We get: Warp each image so that each control point matches mean shape Sample gray-level information g Apply PCA to gray-level dataHow to generate a CAM (cont.) We get: Concatenate shape and gray-level parameters (from PCA) Apply a further PCA to the concatenated vectorsHow to generate a CAM (cont.) We get:CAM Properties Combines shape and gray-level variations in one model– No need for separate models Compared to separate models, in general, needs fewer parameters Uses all available informationCAM Properties (cont.) Inherits appearance model benefits– Able to represent any face within bounds of the training set– Robust interpretation  Model parameters characterize facial featuresCAM Properties (cont.) Obtain parameters for inter and intra class variation (identity and residual parameters) – “explains” faceCAM Properties (cont.) Useful for tracking and identification– Refer to: G.J.Edwards, C.J.Taylor, T.F.Cootes. "Learning to Identify and Track Faces in Image Sequences“. Int. Conf. on Face and Gesture Recognition, p. 260-265, 1998. Note: shape and gray-level variations are correlatedHow to interpret unseen example Treat interpretation as an optimization problem– Minimize difference between the real face image and one synthesized by AAMHow to interpret unseen example (cont.) Appears to be difficult optimization problem (~80 parameters) Key insight: we solve a similar optimization problem for each new face image Incorporate a-priori knowledge for parameter adjustments into algorithmAAM: Training Offline: learn relationship between error and parameter adjustments Result: simple linear modelAAM: Training (cont.) Use multiple multivariate linear regression– Generate training set by perturbing model parameters for training images– Include small displacements in position, scale, and orientation– Record perturbation and image differenceAAM: Training (cont.) Important to consider frame of reference when computing image difference– Use shape-normalized representation (warping)– Calculate image difference using gray level vectors:AAM: Training (cont.) Updated linear relationship: Want a model that holds over large error range Experimentally, optimal perturbation around 0.5 standard deviations for each parameterAAM: Search Begin with reasonable starting approximation for face Want approximation to be fast and simple Perhaps Viola’s method can be applied hereStarting approximation Subsample model and image Use simple eigenface metric:Starting approximation (cont.) Typical starting approximations with this methodAAM: Search (cont.) Use trained parameter adjustment Parameter update equation:Experimental results Training:– 400 images, 112 landmark points– 80 CAM parameters– Parameters explain 98% observed variation Testing:– 80 previously unseen facesExperimental results (cont.) Search results after initial, 2, 5, and 12 iterationsExperimental results (cont.) Search convergence: – Gray-level sample error vs. number of iterationsExperimental results (cont.) More reconstructions:Experimental results (cont.)Experimental results (cont.) Knee images:– Training: 30 examples, 42 landmarksExperimental results (cont.) Search results after initial, 2 iterations, and convergence:Constrained AAMs Model results rely on starting approximation Want a method to improve influence from starting approximation Incorporate priors/user input on unseen image– MAP formulationConstrained AAMs Assume:– Gray-scale errors are uniform gaussian with variance– Model parameters are gaussian with diagonal covariance– Prior estimates of some of the positions in the image along with covariancesConstrained AAMs (cont.) We get update equation:where:Constrained AAMs Comparison of constrained and unconstrained AAM searchConclusions Combined Appearance Models