ECE 738 Paper presentation Paper Active Appearance Models Author T F Cootes G J Edwards and C J Taylor Student Zhaozheng Yin Instructor Dr Yuhen Hu Date Feb 14 2005 Note some slides copyrighted by the original authors 1 Papers T F Cootes G J Edwards and C J Taylor Active Appearance Models IEEE PAMI Vol 23 No 6 pp 681 685 2001 T F Cootes G J Edwards and C J Taylor Active Appearance Models in Proc European Conference on Computer Vision 1998 Vol 2 pp 484 498 Springer 1998 Best paper prize 2 Flexible models Statistical Shape Models Active Shape Models ASM Combined Appearance Models Active Appearance Models AAM Shape model ASM AAM 3 Flexible models Shape Shape is the geometric information invariant to a particular class of transformations translation rotation scaling Appearance 4 Applications Flexible models can be used to Locate examples of structures in new images Classify objects found in images Filter images to pick out interesting features Practical problems Face recognition industrial inspection and medical image analysis 5 Flexible models Statistical Shape Models Active Shape Models ASM Combined Appearance Models Active Appearance Models AAM Shape model ASM AAM 6 Statistical Shape Models Given sets of training images build a statistical shape model Each shape in the training set is represented by a set of n labeled landmark points which must be consistent from one shape to the next Ex The outline of a hand is represented by 72 labeled points 6 5 4 3 2 1 7 Statistical Shape Models Each shape is represented by a 2n 1 vector X x1 xn y1 yn Using Principal Component Analysis PCA or eigen analysis the shape model is X X P b where P is a 2n t matrix whose columns are unit vectors along principle axes or basis vector b is a t 1 vector of shape parameters or weight Ex Vary the first three parameters of the shape vector b one at a time 8 Aligning Two Shapes Procrustes analysis Find transformation which minimizes x1 T x 2 2 Resulting shapes have approximately the same scale and orientation 9 Aligning a Set of Shapes Generalized Procrustes Analysis Find the transformations Ti which minimise 2 m T x i i Where 1 m Ti x i n Under the constraint that m 1 10 Dimensionality Reduction b1 p1 x x x x p1b1 11 Dimensionality Reduction Data lies in subspace of reduced dim x x p1b1 p nbn However for some t i b j 0 if j t Variance of b j is j t i 12 Statistical Shape Models Another example Shape of the facial structures with 68 points 13 Flexible models Statistical Shape Models Active Shape Models ASM Combined Appearance Models Active Appearance Models AAM 14 Active Shape Models Suppose we have a statistical shape model Trained from sets of examples How do we use it to interpret new images Use an Active Shape Model Iterative method of matching model to image 15 Active Shape Models ASM Assume we have an initial estimate for the pose and shape parameters eg the mean shape X X P b 16 Active Shape Models ASM Iterative algorithm Look along normals through each model point to find the best local match for the model of the image appearance at that point eg strongest nearby edge Update the pose and shape parameters to best fit the model instance to the found points Repeat until convergence Initial pos 5th iterations convergence 17 ASM Search Overview Local optimization Initialize near target Search along profiles for best match X Update parameters to match to X X i Yi 18 Active Shape Models ASM Performance improvement Multi resolution implementation coarse tofine approach we start searching on a coarse level of a Gaussian image pyramid and progressively refine This leads to much faster more accurate and more robust search 19 Flexible models Statistical Shape Models Active Shape Models ASM Combined Appearance Models Active Appearance Models AAM 20 Combined Appearance Models Idea Statistical Shape Model models the shape change of an object construct a similar statistical model to represented the intensity variation across a region Think skeleton and muscle 21 Combined Appearance Models Method Given a set of training images labeled with land mark points we can use image warping to deform each image so that the object has the mean shape then build a statistical model of the grey levels across the object Ex The central image is the mean 22 Building Appearance Models For each example extract shape vector Shape x x1 y1 xn yn T Build statistical shape model x x Ps b s 23 Building Appearance Models For each example extract texture vector Shape x x1 y1 xn yn T Warp to mean shape Texture g 24 Warping texture Problem Given corresponding points in two images how do we warp one into the other Two common solutions 1 Piece wise linear using triangle mesh 2 Thin plate spline interpolation 25 Interpolation using Triangles Control points xi yi Warped points xi yi Region of interest enclosed by triangles Moving nodes changes each triangle Just need to map regions between two triangles 26 Barycentric Co ordinates c c x a a b x a b c x b x a b c 1 x is inside the triangle if 0 1 and 0 1 27 Building Texture Models For each example extract texture vector Warp to mean shape Texture g Normalise vectors as for eigenfaces Build eigen model g g Pg b g 28 Combined Models Shape and texture often correlated When smile shadows change texture and shape changes Learning this correlation leads to more compact and specific model 29 Combined Appearance Models x x Ps b s In this paper g g Pg b g x x Q s c g g Q g c Varying c changes both shape and texture 30 Flexible models Statistical Shape Models Active Shape Models ASM Combined Appearance Models Active Appearance Models AAM 31 Active Appearance Models Suppose we have a statistical appearance model Trained from sets of examples How do we use it to interpret new images Use an Active Appearance Model Iterative method of matching model to image 32 Interpreting Images Place model in image Measure Difference Update Model Iterate Active Appearance Models AAM AAM vs ASM The Active Appearance Model AAM is a generalization of the widely used Active Shape Model approach but uses all the information in the image region covered by the target object rather than just that near modeled edges 34 Quality of Match Residual difference r p I m p I im p p all parameters eg p c X c Yc s Ideally find and optimize p p r p r p pT p Bayes rule p p r E p r p r p p r Cannot usually know p r 35 Quality of Match Usually attempt to maximize p r p p p 1 This is equivalent to maximizing log p r p p log p p 2 Which is equivalent