Automatic Recognition of Facial Actions inSpontaneous ExpressionsMarian Stewart Bartlett1, Gwen C. Littlewort1, Mark G. Frank2, Claudia Lainscsek1,Ian R. Fasel1, Javier R. Movellan11Institute for Neural Computation, University of California, San [email protected], [email protected], [email protected], [email protected],[email protected] of Communication, University at Buffalo, State University of New [email protected]—Spontaneous facial expressions differ from posedexpressions in both which muscles are moved, and in the dy-namics of the movement. Advances in the field of automaticfacial expression measurement will require developmentand assessment on spontaneous behavior. Here we presentpreliminary results on a task of facial action detection inspontaneous facial expressions. We employ a user indepen-dent fully automatic system for real time recognition offacial actions from the Facial Action Coding System (FACS).The system automatically detects frontal faces in the videostream and coded each frame with respect to 20 Actionunits. The approach applies machine learning methods suchas support vector machines and AdaBoost, to texture-basedimage representations. The output margin for the learnedclassifiers predicts action unit intensity. Frame-by-frameintensity measurements will enable investigations into facialexpression dynamics which were previously intractable byhuman coding.I. INTRODUCTIONA. The facial action coding systemIn order to objectively capture the richness and com-plexity of facial expressions, behavioral scientists havefound it necessary to develop objective coding standards.The facial action coding system (FACS) [17] is the mostwidely used expression coding system in the behavioralsciences. A human coder decomposes facial expressionsin terms of 46 component movements, which roughlycorrespond to the individual facial muscles. An exampleis shown in Figure 1.FACS provides an objective and comprehensive lan-guage for describing facial expressions and relating themback to what is known about their meaning from thebehavioral science literature. Because it is comprehensive,FACS also allows for the discovery of new patterns relatedto emotional or situational states. For example, what arethe facial behaviors associated with driver fatigue? Whatare the facial behaviors associated with states that arecritical for automated tutoring systems, such as interest,boredom, confusion, or comprehension? Without an ob-jective facial measurement system, we have a chicken-and-egg problem. How do we build systems to detectcomprehension, for example, when we don’t know forcertain what faces do when students are comprehending?Having subjects pose states such as comprehension andconfusion is of limited use since there is a great deal ofevidence that people do different things with their faceswhen posing versus during a spontaneous experience (e.g.[8], [14]). Likewise, subjective labeling of expressionshas also been shown to be less reliable than objectivecoding for finding relationships between facial expressionand other state variables. Some examples of this arediscussed below, namely the failure of subjective labelsto show associations between smiling and other measuresof happiness, as well as failure of naive subjects to differ-entiate deception and intoxication from facial expression,whereas reliable differences were shown with FACS.Objective coding with FACS is one approach to theproblem of developing detectors for state variables suchas comprehension and confusion, although not the onlyone. Machine learning of classifiers from a databaseof spontaneous examples of subjects in these states isanother viable approach, although this carries with itissues of eliciting the state, and assessment of whetherand to what degree the subject is experiencing the desiredstate. Experiments using FACS face the same challenge,although computer scientists can take advantage of a largebody of literature in which this has already been done bybehavioral scientists. Once a database exists, however, inwhich a state has been elicited, machine learning can beapplied either directly to image primitives, or to facialaction codes. It is an open question whether intermediaterepresentations such as FACS are the best approach torecognition, and such questions can begin to be addressedwith databases such as the one described in this paper.Regardless of which approach is more effective, FACSprovides a general purpose representation that can beuseful for many applications. It would be time consumingto collect a new database and train application-specific de-tectors directly from image primitives for each new appli-cation. The speech recognition community has convergedon a strategy that combines intermediate representationsfrom phoneme detectors plus context-dependent featurestrained directly from the signal primitives, and perhapsa similar strategy will be effective for automatic facialexpression recognition.There are numerous examples in the behavioral scienceliterature where FACS enabled discovery of new relation-ships between facial movement and internal state. Forexample, early studies of smiling focused on subjectivejudgments of happiness, or on just the mouth movement(zygomatic major). These studies were unable to show areliable relationship between expression and other mea-sures of enjoyment, and it was not until experiments withFACS measured facial expressions more comprehensively,that a strong relationship was found: Namely that smileswhich featured both orbicularis oculi (AU6), as well as zy-gomatic major action (AU12), were correlated with self-reports of enjoyment, as well as different patterns of brainactivity, whereas smiles that featured only zygomaticmajor (AU12) were not (e.g. [16]). Research based uponFACS has also shown that facial actions can show differ-ences between genuine and faked pain [8], and betweenthose telling the truth and lying at a much higher accuracylevel than naive subjects making subjective judgments ofthe same faces [26]. Facial Actions can predict the onsetand remission of depression, schizophrenia, and otherpsychopathology [20], can discriminate suicidally fromnon-suicidally depressed patients [27], and can predicttransient myocardial ischemia in coronary patients [42].FACS has also been able to identify patterns of facialactivity involved in alcohol intoxication that observers nottrained in FACS failed to note [44].Figure 1.