1CSE190a Fall 06Face Recognition:LightingBiometricsCSE 190-aLecture 17Why is Face Recognition Hard? CS252A, Winter 2005 Computer Vision IFisherfaces: Class specific linear projectionP. Belhumeur, J. Hespanha, D. Kriegman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, PAMI, July 1997, pp. 711--720.•An n-pixel image x∈Rncan be projected to a low-dimensional feature space y∈Rmbyy = Wxwhere W is an n by m matrix.• Recognition is performed using nearest neighbor in Rm.• How do we choose a good W?CS252A, Winter 2005 Computer Vision IPCA & Fisher’s Linear Discriminant • Between-class scatter• Within-class scatter• Total scatter•Where– c is the number of classes– μiis the mean of class χi–| χi| is number of samples of χi..TiiciiBS ))((1μμμμχ−−=∑=∑∑=∈−−=cixTikikWikxS1))((χμμμWBcixTkkTSSxSik+=−−=∑∑=∈1))((χμμμμ1μ2μχ1χ2CS252A, Winter 2005 Computer Vision IPCA & Fisher’s Linear Discriminant • PCA (Eigenfaces) Maximizes projected total scatter• Fisher’s Linear Discriminant Maximizes ratio of projected between-class to projected within-class scatterWSWWTTWPCAmaxarg=WSWWSWWWTBTWfldmaxarg=χ1χ2PCAFLDCS252A, Winter 2005 Computer Vision IHarvard Face Database1515oo4545oo3030oo6060oo• 10 individuals• 66 images per person• Train on 6 images at 15o• Test on remaining images2CS252A, Winter 2005 Computer Vision IRecognition Results: Lighting Extrapolation0510152025303540450-15 degrees 30 degrees 45 degreesLight DirectionError RateCorrelation Eigenfaces Eigenfaces (w/o 1st 3) Fisherface© Jain, 2004(2D) Model-based–Active Appearance Model• Model Construction (linear)labeled image landmarks shape-free texture© Jain, 2004Active Appearance Model (AAM)• Shape model• Appearance model• Combined modelTnnyxyxs ),,...,,(11=ssbPss +=ggbPgg +=TmIIg ),...,(1=⎟⎟⎠⎞⎜⎜⎝⎛−−=⎟⎟⎠⎞⎜⎜⎝⎛=)()(ggPssPWbbWbTgTssgssQcb =PCA© Jain, 2004AAM• Model fitting– Minimize the objective function (gray level difference between the given image and the stored model– Searching by learning• Annotated model (true model parameters)• Relation: known model displacements ↔ observed difference vector• Use multivariate multiple regression to learn the relation and predict the displacement during searching2Iδ=Δ© Jain, 2004AAMInitial 3 its 8 its 11 its Converged OriginalThe challenge caused by lighting variabilitySame Person or Different People3Same Person or Different PeopleSame Person or Different People4Illumination & Image Set• Lack of illumination invariants[Chen, Jacobs, Belhumeur 98]• Set of images of Lambertian surface w/o shadowing is 3-D linear subspace [Moses 93], [Nayar, Murase 96], [Shashua 97]• Empirical evidence that set of images of object is well-approximated by a low-dimensional linear subspace [Hallinan 94], [Epstein, Hallinan, Yuille 95]• Illumination cones – [Belhumeur, Kriegman 98]• Spherical harmonics lighting & images– [Basri, Jacobs 01], [Ramamoorthi, Hanrahan 01]• Analytic PCA of image over lighting – [Ramamoorthi 02]Some Background Issues1. What is the image space?2. What is lighting?3. What is reflectance?The Space of Images• Consider an n-pixel image to be a point in an n-dimensional space, x ∈ Rn.• Each pixel value is a coordinate of x.∈x1x2xn• Many results will apply to linear transformations of image space (e.g. filtered images)• Other image representations (e.g. Cayley-Klein spaces, See Koenderink’s “pixel f#@king paper”)x1xnx2LightingGenerally, arbitrary lighting can be viewed as a non-negative function on a 4-D space.For any object and any image, there exists a lighting condition which could have produced that image Typically make limiting assumptions• Distant lighting (non-negative function on sphere)• Point light sources (delta function)• Diffuse lighting (constant function)BRDFBi-directional Reflectance Distribution Functionρ(θin, φin ; θout, φout)• Ratio of incident irradiance to emitted radiance•Function of– Incoming light direction:θin, φin– Outgoing light direction: θout, φout^n(θin,φin)(θout,φout)()()()ωφφθφθφθφθρdxLxLxinininioutoutooutoutinincos,;,;,;,; =The Illumination Invariance QuestionLet I(O,L) be the image of object O under lighting L.Does there exist a recognition function c(I1,I2) of two images I1and I2such that ∀O, L1, L2C(I(O,L1), I(O, L2) = 0and∃Oi, Oj, L1, L2 C( I(Oi,L1), I(Oj, L2) ≠ 0NO!5Assumptions• Lambertian reflectance functions.• Objects have convex shape.• Light sources at infinity.• Orthographic projection.• Note: many of these can be relaxed….Lambertian SurfaceAt image location (u,v), the intensity of a pixel x(u,v) is:x(u,v) = [a(u,v) n(u,v)] [s0s ]= b(u,v) swhere• a(u,v) is the albedo of the surface projecting to (u,v).• n(u,v) is the direction of the surface normal.•s0is the light source intensity.• s is the direction to the light source.^n^s^^..ax(u,v)Lambertian Assumption with shadowing:x =where • x is an n-pixel image vector• B is a matrix whose rows are unit normals scaled by the albedos• s ∈R3is vector of the light source direction scaled by intensity- bT1 -- bT2 -...….- bTn-n x 3Model for Image Formationmax(B s, 0) B = b1b2b3sN-dimensional Image Spacex1x23-D Linear subspaceThe set of images of a Lambertian surface with no shadowing is a subset of 3-D linear subspace.[Moses 93], [Nayar, Murase 96], [Shashua 97]xnL = {x | x = Bs, ∀s ∈R3 }where B is a n by 3 matrix whose rows are product of the surface normal and Lambertian albedoLL0Set of Images from a Single Light Source•Let Li denote the intersection of L with an orthant i of Rn.•Let Pi(Li) be the projection of Lionto a “wall” of the positive orthant given by max(x, 0).x2P0(L0)P4(L4)P3(L3)P2(L2)P1(L1)x3x1L0Lx2x1x3U = U Pi(Li) Mi=0b1b2b3sS1S0S2S3S1S5Then, the set of images of an object produced by a singlelight source is:N-dimensional Image SpaceThe Illumination ConeTheorem:: The set of images of any object in fixed posed, but under all lighting conditions, is a convex cone in the image space.(Belhumeur and Kriegman, IJCV, 98) Single light source images lie on cone boundary2-light source imagex1x2xnIllumination Cone6Some natural ideas & questions• Recogntion: Is a test image within an object’s cone?• Can the cones of two different objects intersect?• Can