Nonverbal Behavior Generation for Virtual HumansCS534 Affective ComputingJina LeePhD [email protected] Functions of Nonverbal Behaviors Nonverbal Behavior Generator Data-driven Approach for Modeling Head Movements Conclusions and Future Work3Background:Function of Nonverbal Behavior Nonverbal behaviors serve functionsINTERACTIONALAwareness/RecognitionInitiate/Break contactTake/Give turnsPROPOSITIONALEmphasize/ContrastReferDepict featureChange topicRequest/Give feedbackAFFECTIVEExpress Emotion, AttitudesReveal Traits, CultureRaise eyebrowsGaze towards Posture NodSmile Shake head BeatPoint Gaze away GestureLower eyebrows Toss headBody orientation PauseNonverbal Behaviors for Virtual Humans5Nonverbal Behavior Generator - Problem Challenge – To find the mapping between utterance and function– To model the nonverbal behavior generation for virtual humans using this mapping without a rich markupEmphasizeContrastEmotional ExpressionRegulate TurnsReferComplement…FunctionSmileBrow LoweredBrow RaiseHead NodHeadshake…BehaviorsPsychologicalliteratureUtteranceI’m glad to hear that.We may not trust each other well.I can’t believe you did that.…?Goals of NVBG Robust NVB generation that can use markup of communicative function if provided, but can also extract/infer it if not Extraction that leverages syntactic and semantic analysis of text Use open-source tools Use evolving standards for markup– SAIBA framework (FML & BML)– Clear distinction of function and behaviorNVBG ArchitectureCommunicative Function DerivationBehavior SuggestionBehavior RealizationFunction RulesFML + BMLAgentReasoning,Emotion,LanguageSmartBodyBehavior DescriptionNVB RulesNVBGBMLCacheNatural Language ParserParse TreeSurface TextSituation, Agent, Intention, Behavior, and Animation (SAIBA) Framework International effort to unify multimodal behavior generation framework [Kopp et al., 2006]  A distinction between communicative function and behavior Wiki page: http://wiki.mindmkaers.org/projects:SAIBA:m ainBehavior Markup Language (BML)Elements roughly correspond to the parts involved in the behavior– BODY, GESTURE , HEAD, FACE, GAZE, LIPS, SPEECHBehaviorsFunctionEmphasize ContrastExpress Emotion ReferRegulate Turns complement…SmileBrow LoweredBrow RaiseHead NodHeadshake…Function Markup Language (FML)Specifies the communicative and expressive intent of the agent.- AFFECT, INTENT, TURN- Persistent Features: PERSONALITY, CULTURE, GENEDERSmartBody Goals of the SmartBody project:– Exploit a range of animation techniques that best address VH requirements– Support community-wide research in animation techniques for VH – Foster collaboration on components for developing VH applications– Support reuse across a range of projects– Lower the barrier of entry for application development (today’s focus)• SmartBody: an open source modular framework for animating virtual humans in real-timehttp://smartbody.wiki.sourceforge.net/Example of Nonverbal Behavior GenerationSurface Text: first_VP RulePriority 5emo_negative RulePriority 1Head nod, Lowered BrowsBeat gestureFunction Rules: Behavior Rules: first_NP RulePriority 5Head nodI was mad at him.Me RulePriority 5Head nodPossible Approaches How do we extract the communicative function from linguistic features?– Information from the natural language generator (Communicative Intent / affect) e.g. Multi-modal NLG [Krenn et al., 2002]– Machine learning techniques using a gesture corpora– Top down analysis of video dataPsychological Literature Literature on NVB– e.g. Ekman, Hadar, Kendon, McClave, etc.Function BehaviorSigns of affirmation Head nodsBackchannel (response) requests Head nodSelf correction Head shakeConcepts of inclusivity (i.e. everyone, all)Lateral sweep or head shakeListing Head moves with succeeding itemsUncertainty (I guess, I think…) Lateral shakesNegative expression Head shakeSuperlative or intensified expression (i.e. very, really)Head shake, Brow frownMark Contrast Head movementAnalysis of Video Data To validate what’s found in the literature Find out the dynamic properties of behaviors– speed, repetition, span of behaviors (word/phrase, cross-syntactic boundaries) To see what the actual NVB look like– Do head nods across different functions appear differently?  Relation between the behavior and linguistic properties of the surface text– Guide rule construction Sensitive Artificial Listener [HUMAINE, 2004]Nonverbal Behaviors Observed Head – Nod, shake, tilt, moved to the side, pulled back, pulled down Eyebrow– Raised, frowned (lowered), flash Eyes / Gaze– Look up, look down, look away, squinted, squeezed, rolled Others– Shoulder shrug, mouth pulled on one sideBreakdown of the number of utterances with corresponding functionFunction # of utterances (out of 223)Negation 62Intensification 62Affirmation 36Assumption 28Word Search 23Contrast 23Interjection 13Response Request 10Listing 9Obligation 9Inclusivity 7Construction of Nonverbal Behavior RulesVideo AnalysisNVB RulesPsychological Literatureon NVBsNonverbal Behavior Rules – from Surface TextDerivationNo, not, nothing, cannot, noneReally, very, quite, great, absolutely, gorgeous…Yes, yeah, I do, We have, It’s true, OKI guess, I suppose, I think, maybe, probably, perhaps, couldBut, howeverEverything, all, whole, several, plenty, full…FunctionNegationIntensificationAffirmationAssumption / PossibilityContrastInclusivityBehaviorHead shakes on phraseHead nod and brow frown on wordHead nods and brow raise on phraseHead nods on phraseHead moved to side and brow raiseLateral head sweep and brow flash on wordNonverbal Behavior Rules – from Parse TreeDerivationINT(interjection)First <NP>(noun phrase)<NP>(noun phrase)First <VP>(verb phrase)FunctionInterjectionBelievabilityBelievabilityBelievabilityBehaviorHead nod on wordBig head nod on start of the noun phraseHead nod on start of noun phraseHead nod and beat gesture on start of the first verb phrasePriorities of NVB Rules Example She wouldn’t be really happy. -> Head shakes over the whole sentence -> negation overrides intensification Algorithm1. Find all the utterances where two or more rules co-occur2. Mark which rule overrides the other (looking at the behavior) in the matrix of rules and count the frequencies of these cases Result1. Interjection2. Negation3.