Eigenfaces 1An Overview of Face Recognition Using EigenfacesAcknowledgements: Original Slides from Prof. Matthew Turk-- also notes from the web-Eigenvalues and Eigenvectors-PCA-EigenfacesMonday, February 22, 2010OutlineWhy automated face recognition?Eigenfaces and appearance-based approaches to recognition– Motivation– ReviewWhy Eigenfaces?Why not Eigenfaces?Where shall we go from here?Monday, February 22, 2010Why Automated Face Recognition?It is a very vital and compelling human ability– Faces are important to us– Severe social problem for people who lack this ability It’s fun to work on– Better than recognizing tanks and sprocketsGood, paradigmatic vision problemIt may actually be useful– Biometrics, HCI, surveillance, …They can do it in the movies!Monday, February 22, 2010Commercial InterestImage and video indexingBiometrics, e-commerce– Visionics, Viisage, eTrue, …Surveillance– Casinos, Super Bowl, Tampa, FLMonday, February 22, 2010Automated Face RecognitionTypical formulations:– Given an image of a face, who is it? (recognition)– Is this an image of Joe Schmoe? (verification)Why isn’t this easy?Monday, February 22, 2010The ProblemThe human face is an extremely complex object, highly deformable, with both rigid and non-rigid components that vary over time, sometimes quite rapidly and sometimes quite slowlyThe “object” is covered with skin, a non-uniformly textured material that is difficult to model either geometrically or photometrically Monday, February 22, 2010The ProblemTime-varying changes include:– The growth and removal of facial hair, wrinkles and sagging of the skin brought about by aging, skin blemishes, changes in skin color and texture caused by exposure to sun, etc.Plus many common artifact-related changes:– Glasses, makeup, jewelry, piercings, cuts and scrapes, bandages, etc.Not to mention facial expressions, changes in hairstyle, etc.Monday, February 22, 2010The ProblemIn general, object recognition is difficult because of the immense variability of object appearance Several factors are all confounded in the image data – Shape, reflectance, pose, occlusion, illumination Human faces add more factors– Expression, facial hair, jewelry, etc.So… one may argue that face recognition is harder than most object recognition tasksMonday, February 22, 2010The ProblemOvercoming these difficulties will be a significant step forward for the computer vision communitySo, face recognition has been considered a challenging problem in computer vision for some time nowThe amount of effort in the research community devoted to this topic has increased significantly over the years.Real-time performance is key!Monday, February 22, 2010What is Face Recognition?What can be observed via the face?– Identity, emotion, race, age, sex, gender, attractiveness, lip reading, character(?)Does face recognition include hair? Ears?Are people really very good at face recognition?– Driver’s license photos– Models in catalogs– Colleagues at ICCVPerhaps we don’t do it all that often– Clothes, gait, voice, contextMonday, February 22, 2010Monday, February 22, 2010Monday, February 22, 2010The Context of Face RecognitionFace recognition (in humans and machines) often coexists with other face processing tasks:– Face (and head) detection– Face (and head) tracking– Face pose estimation– Facial expression analysis– Facial feature detection, recognition, and trackingIt may be unnatural to separate face recognition from these other tasks– But we will anyway…Monday, February 22, 2010Eigenfaces: MotivationFirst generation of FR systems– Locate features, measure distances and angles, create feature vector for classification– Bledsoe 1966, Kelly 1970, Kanade 1973Mid-to-late 1980s – Is there a different approach to recognition, perhaps making use of all the image data (not just isolated features)?features ???Monday, February 22, 2010Revisionary HistoryThe “Eigenfaces” approach, based on PCA, was never intended to be the definitive solution to face recognition. Rather, it was an attempt to re-introduce the use of information “between the features”; that is, it was an attempt to swing back the pendulum somewhat to counterbalance the focus on isolated features. Monday, February 22, 2010Motivation: Biological VisionFor decades, vision researchers have been investigating mechanisms of human face recognition– Debate: Are faces special? I.e., does face processing have a different neural substrate from other visual recognition?– Debate: Configural vs. holistic processing (feature-driven vs. whole stimulus integration)– Evidence from many sources: psychophysics, single-cell recording, neuroimaging, neurophysiological case studiesMonday, February 22, 2010Example: Two face cells in monkeyMonday, February 22, 2010Eigenfaces: MotivationBiological face recognition– Is face recognition configural or holistic?– Previous approaches had all been configuralSo… let’s try an appearance-based approach to face recognition!Appearance models are complementary to shape models – Not a replacementMonday, February 22, 2010Levels of Recognition/MatchingShapeFeaturesPixelsShapeFeaturesPixelsModel ExampleAbstractionMonday, February 22, 2010Eigenfaces origins1980s Burt et al. pyramid-based FR work1987 Sirovich and Kirby paper– PCA-based encoding of face imagesReal-time motivationIs this a suitable representation for face recognition? Detection? Multiple scales? Multiple views? Is it computationally feasible?Monday, February 22, 2010So What is (are) Eigenfaces?Uses Principal Component Analysis (PCA) to construct a “Face Space” from a training set of face images– Subspace of all possible images– Encodes only face images– Some choice in dimensionalityTest image is projected into the Face Space– Projection distance determines “faceness”– Classify according to projection coefficientsEfficient implementation for face detection and recognitionExplicitly handle scale and pose (simply)Implicitly handle lighting, expression, etc.Monday, February 22, 2010IntuitionImage space is vastly large–8x8 binary image → 264 image points (distinct images)–1 billion images per second →Assumptions:– Images of particular objects (faces) may occupy a relatively small but distinct region