Affective Computing : Computational Approaches in Emotion Recognition Abe Kazemzadeh, Chi-Chun (Jeremy) Lee, Angeliki Metallinou SAIL lab: sail.usc.edu Univ. of Southern CaliforniaEmotional Speech • Speech conveys rich emotional informationMultimodality • Emotions are often expressed multimodally – e.g.,sarcasm: conflicting multimodal cuesComplex Representation of Emotion • Emotions have variable intensity and clarity • Categorical descriptions may not give the full pictureComplex Representation of Emotion • Emotions have variable intensity and clarity • Categorical descriptions may not give the full pictureEmotion Evolution • Emotions generally happen in context • Of a situation • Of a conversation topic, e.g lost luggage • Of an emotional history, e.g speaker was angry until nowMotivation • Study of human emotions quantitatively • Human-computer interface – Education – Entertainment – Dialog system – Personalized application/software – Virtual agent – … • Behavioral informaticsOutline • Emotional Representations • Collecting Emotional Databases • Multimodal Feature Extraction • Methods for Emotion Recognition • Beyond Recognizing Emotions • Conclusions and Open QuestionsEmotion Representations Categorical and Dimensional • Categorical Representations – Description using discrete categories, e.g angry, happy, sad etc – Six ‘basic’ emotions * anger, disgust, fear, happiness, sadness, surprise – Choice of affective states could be application driven, e.g interest, frustration... • Dimensional Representations ** – Activation, Valence, Dominance – Description of attributes (dimensions) of an emotion * P. Ekman (1999), “Basic Emotions”, in Handbook of Cognition and Emotion ** H. Schlosberg, “Three dimensions of emotion”, in Psychology ReviewEmotion Representations Continuous DimensionalContinuous Dimensional Representations • Feeltrace Tool for continuous annotations* • Agreement on the trends of emotional curves • Rather that absolute values • Easier to rate emotions in relative terms ** * Feeltrace: an instrument for recording perceived emotion in real time, Cowie etal **Ranking-based emotion recognition for music organization and retrieval, Yang and ChenEmotion Representations Challenges • Emotional descriptions are subjective • perceptual differences among individuals • vague or subtle emotional expressions (real life emotions) • Ground truth for recognition task may be ambiguous • Level of detail of emotional descriptions • How many emotional categories need to be considered? • How many levels of valence and activation?Natural Language Descriptions EMO20Q Q: Do you feel this emotion at Disneyland? A: no Q: Do you feel this emotion when you run over a dog? A: possibly, yes. Q: is it remorse? A: no Q: Do you feel this when someone close dies? A: Not necessarily, but you could I suppose Q: When stealing something from a friend do you feel like this? A: I think so, but I don't usually steal stuff though. Q: There is a sound that does not let you sleep at night at your apartment do you feel like this in reaction to this noise? A: yes Q: is it annoyed? A: no Q: You are walking through South Central very late with your very expensive laptop and you see a stranger quickly moving towards you, do you feel like this when that happens? A: yes, getting closer. Q: Fear? A: no but close Q: Nervousness? A: no, that's a near synonym but I think it's slightly different Q: Do you feel like this when there is a big event coming up and you "cant wait" for it to happen A: no, actually the opposite... you don't want it to happen. Q: how about anxious? A: yes, it's anxious... I think i'll count it, but that wasn't the exact word. Do you know it?Data Collection • IEMOCAP * and CreativeIT ** – Multimodal emotional databases collected by SAIL • Use of improvisations and theatrical techniques – Elicitation of naturalistic emotions – Dyadic settings • Collected data – Detailed MoCap information of face or body – Microphones and cameras – Dialog transcriptions * IEMOCAP: Interactive emotional dyadic motion capture database, Busso etal ** The USC CreativeIT database: A multimodal database of theatrical improvisation, A. Metallinou, C.-C. Lee, C. Busso, S. Carnicke, S. NarayananA clip from CreativeIT database • A scene from Chekhov’s play ’Uncle Vanya’Feature ExtractionSpeech Production And Perception (Gray's Anatomy via wikipedia.org)MRI Recordings (SAIL Realtime MRI Corpus)Speech Processing: Spectrogram (Rob Hagiwara, http://home.cc.umanitoba.ca/~robh/howto.html)Text Features • Bag of words (unigrams) • N-gram language models • Emotion dictionaries • Lattices • Orthography (punctuation, capitalization, emoticons) • Wordnet • Syntax • Semantic roles • World knowledgeASR: HMM (http://en.wikipedia.org/wiki/Hidden_Markov_model)ASR: Lattice (Georgiou et al., ACII 2011)Facial Feature Extraction • Facial expressions convey emotional information • Facial Action Coding System (FACS) and Action Units (AU) * • Extract Facial Features – FACS based approaches – Data-driven approaches – Statistical functionals over low level face features *Facial Action Coding System Manual, P. Ekman and W. FriesenBody Language Feature Extraction • Body language expresses rich emotional information * – Body movement, gestures and posture – Relative behavior, e.g., approach/avoidance, looking/turning away, touching • Extract detailed features from MoCap * The new handbook of methods in nonverbal behavior research, J. Harrigan, R. Rosenthal and K. SchererExamples of Features * Tracking Changes in Continuous Emotion States using Body Language and Prosodic Cues, A. Metallinou, A. Katsamanis, Y. Wang and S. NarayananEmotion RecognitionEmotion Recognition turn by turn recognition • Recognition Task – recognize emotion in IEMOCAP database • Extracted Features – audio features (384 dimensions) • Emotion Representation – 4 categorical emotional labels: Angry, Happy, Sad, Neutral • Technological Difficulties – multiclass emotion labels classification – audio features only – database specificHierarchical Tree Classification • Easily adaptable to other databases (AIBO databases) • Flexible framework • Exploit expert knowledge Chi-Chun Lee, Emily Mower, Carlos Busso, Sungbok Lee and Shrikanth S. Narayanan, Emotion recognition