Face Modeling15-463: Computational PhotographyAlexei Efros, CMU, Fall 2006Portrait of Piotr Gibas© Joaquin Rosales GomezThe Power of AveragingFigure-centric averagesAntonio Torralba & Aude Oliva (2002)Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images.More by Jason SalavonMore at: http://www.salavon.com/“100 Special Moments” by Jason SalavonWhyblurry?Computing MeansTwo Requirements:• Alignment of objects• Objects must span a subspaceUseful concepts:• Subpopulation means• Deviations from the meanImages as Vectors= mnn*mVector Mean: Importance of Alignment= mnn*m= n*m½ + ½ = mean imageHow to align faces?http://www2.imm.dtu.dk/~aam/datasets/datasets.htmlShape Vector=43Provides alignment!Average Face1. Warp to mean shape2. Average pixelshttp://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/Objects must span a subspace(1,0)(0,1)(.5,.5)ExampleDoes not span a subspacemeanSubpopulation meansExamples:• Happy faces• Young faces• Asian faces•Etc.• Sunny days• Rainy days•Etc.•Etc.Average maleAverage femaleDeviations from the mean--==Image XMean XΔX = X - XDeviations from the mean+=+ 1.7=XΔX = X - XManipulating Facial Appearance through Shape and ColorDuncan A. Rowland and David I. PerrettSt Andrews UniversityIEEE CG&A, September 1995Face ModelingCompute average faces (color and shape)Compute deviationsbetween male and female (vector and color differences)Changing genderDeform shape and/or color of an input face in the direction of “more female”original shapecolor bothEnhancing gendermore same original androgynous more oppositeChanging ageFace becomes “rounder” and “more textured” and “grayer”original shapecolor bothBack to the SubspaceLinear Subspace: convex combinations∑==miiiXaX1Any new image X can beobtained as weighted sum of stored “basis” images. Our old friend, change of basis!What are the new coordinates of X?The Morphable Face ModelThe actual structure of a face is captured in the shape vector S = (x1, y1, x2, …, yn)T, containing the (x, y)coordinates of the n vertices of a face, and the appearance (texture) vector T = (R1, G1, B1, R2, …, Gn, Bn)T, containing the color values of the mean-warped face image. Shape SAppearance TThe Morphable face modelAgain, assuming that we have m such vector pairs in full correspondence, we can form new shapes Smodeland new appearances Tmodelas:If number of basis faces m is large enough to span the face subspace then:Any new face can be represented as a pair of vectors (α1, α2, ... , αm)Tand (β1, β2, ... , βm)T!∑==miiimodela1SS∑==miiimodelb1TTIssues:1. How many basis images is enough?2. Which ones should they be?3. What if some variations are more important than others?• E.g. corners of mouth carry much more information than haircutNeed a way to obtain basis images automatically, in order of importance! But what’s important?Principal Component AnalysisGiven a point set , in an M-dim space, PCA finds a basis such that• coefficients of the point set in that basis are uncorrelated•firstr < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r)x1x0x1x01stprincipal component2ndprincipal componentPCA via Singular Value Decomposition[u,s,v] = svd(A);http://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/Principal Component AnalysisChoosing subspace dimensionr:• look at decay of the eigenvalues as a function of r• Larger r means lower expected error in the subspace data approximationrM1eigenvaluesEigenFacesFirst popular use of PCA on images was for modeling and recognition of faces [Kirby and Sirovich, 1990, Turk and Pentland, 1991] Collect a face ensemble Normalize for contrast, scale, & orientation. Remove backgrounds Apply PCA & choose the first N eigen-images that account for most of the variance of the data.mean facelighting variationFirst 3 Shape BasisMean appearancehttp://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/Using 3D Geometry: Blinz & Vetter, 1999show SIGGRAPH
View Full Document