March 5th, 20091Emotion RecognitionCarlos BussoProf. Shri Narayananfrom speechMarch 5th, 2009 CSCI 534March 5th, 20092Some examples.. Lost baggage call centerMarch 5th, 20093More examplesChild-machine InteractionsCONFIDENT vs. UNCERTAINMarch 5th, 20094More examples Visualizing using MRI..neutralangrysadhappyMarch 5th, 20095Human Communication• Human communication involves a complex orchestration of cognitive, physiological, physical, social processes• Information resides at multiple time scales, through multiple cues– Inherently multimodal: natural communication involves speech, facial/hand gestures, head movement, postures,..• Spoken language carries crucial information: intent, desires, emotionsDecoding Human Communication Cues is a Multi-level Mapping ProblemMarch 5th, 20096Automatic Speech Processing Solutions:mapping speech to words, and beyondEmotionsMeta FeaturesProsodyA tightly integrated approach to speech processing: Recognize •What: spoken language content •Who: speaker identity, and •How: speaking style and emotions automatically from spoken languageSignificance: Natural spoken language is the primary means of human communication: to negotiate, to seek information, to issue orders and to resolve conflictsMarch 5th, 20097• Emotions play a crucial role in human interaction• Knowing the user’s emotional state should help to adjust system performance• User can be more engaged and have a more effective interaction with the system• Crucial for understanding and modeling both individual and social cognition– Emotional (vs. cognitive) reasoning– Emotion is reflected in our body– Our emotions change the minds of others– People rely on emotion for making decisionsWhy study emotion or attitude?March 5th, 20098Applications • Call centers– Quality of service– Coping with frustrated users• Robots – Sense and convey emotions• Artificial animated agents – Sense and convey emotions• Education– Detect frustration • Games– Expressive characters • Observational practices – (e.g. therapy sessions)– Diagnosis and coachingSynthesisEmotional speech synthesisManipulation of body/facial movementExpressive facial animationRecognitionEmotion recognitionAnalysis & perceptionEmotional perceptionAppraisal theoryMarch 5th, 20099Emotion Research @ SAIL• SAIL: Signal Analysis & Interpretation Lab.– http://sail.usc.edu• Speech and emotions– Analysis, recognition, synthesis• Speech production• Multimodal processingMarch 5th, 200910Work in collaboration with USC SAIL members & graduates• Dr. Shri Narayanan, Dr. Sungbok Lee• Matt Black, Jeannette Chang, Michael Grimm, Abe Kazemzadeh, Sam Kim, Chi-Chun (Jeremy) Lee , Emily Mower, Angeliki Metallinou, Ilene Rafii, Michelle Dee, Carlos Busso• SAIL PhD grads/alumni: Murtaza Bulut, Michael Grimm, Dagen Wang, Serdar Yildirim, Chul Min LeeMarch 5th, 200911Emotion recognitionfocus on speechMarch 5th, 200912Outline• Overview• Challenges in emotion recognition• Proposed approaches to emotion recognition• ConclusionsOverviewMarch 5th, 200913Automatic emotion recognition from speechOverviewMarch 5th, 200914Speech: a multimodal signalOverviewMarch 5th, 200915Emotion recognition in the lab• Databases– Acted data– Categorical representation of emotions– Few speakers• Limited data• Features– Many features are selected– Feature set is reduced (pca, fisher linear discriminant, sequential forward feature selection, etc…)• Results – From 50% - 85% depending on the task [Pantic_2003, Cowie_2001]OverviewMarch 5th, 200916• Too much variability– Speaker dependency– Emotional descriptors– Acoustic confusion between categories–Differences in acoustic environments • Results are strongly dependent on the recording condition• Models are not easily generalized to other databases or on-line recognition taskEmotional models do not generalize!!!Emotion recognition in real applicationsOverviewMarch 5th, 200917Outline• Overview• Challenges in emotion recognition– Representation– Databases– Speech normalization– Features– Models• Proposed approaches to emotion recognition• ConclusionsChallenges in emotion recognitionMarch 5th, 200918Brunswik’s lens model [Scherer, 2003]How to describe emotions? (1/4)• Expression and perception of emotion is a complex process–Intended emotion ! perceived emotion– Representation depends on the listenersRepresentationEncoding Transmission RepresentationDistal indicatorsProximal indicatorsAttributionTrait/stateMarch 5th, 200919How to describe emotions? (2/4)• One Pragmatic Approach: Categorical Emotional States– Six basic emotions (happiness, sadness, fear, anger, surprise, disgust)– Mixed emotion, in the order of hundreds (e.g., content, amused, etc…)– Tradeoff between inter-evaluator agreement and description accuracy• Define domain/application-dependent emotional states:– Negative and non-negative in Call Center data– Frustration, politeness, attention for child-machine interaction systems– Cooperation in negotiation tasks, Like/dislike in opinion polling– Hot spots, engagement in meetingsRepresentationMarch 5th, 200920How to describe emotions? (3/4)• Another approach: “Primitives based”• Dimensional attributes – Valence, activation/arousal, dominance/control– Better inter-evaluator agreement– Can help track dynamic variation– Not very useful for certain applicationsRepresentationMarch 5th, 200921How to describe emotions? (4/4)• Real emotional label or attribute values are unknown• Need to use human evaluators– Who, Where, How many• It should be view as an approximation– Perceived emotion may differ from intended emotion [Busso, 2008]?Conventional machine learning problem Emotion recognition Boundaries are blurred!RepresentationMarch 5th, 200922Examples [fru; ()] [ang; ()] [neu; ()][fru; ()] [oth; (exasperated)] [neu; ()][fru; ()] [ang; ()] [oth; (annoyed)][ang; ()] [ang; ()] [ang; ()]RepresentationMarch 5th, 200923Time scale• The best time scale to evaluate/analyze the emotion is not clear– Emotional content may change within a turn– Sentence, chunk, words• Continuous evaluation of emotion– FEELTRACE [Cowie, 2000]RepresentationMarch 5th, 200924Emotional databases• Availability of appropriate emotional databases is a major limitation for scientific research and technology development– Genuine realizations– Integrated