1!Virtual Human Behavior!!and NVBG / SmartBody!!"Stacy Marsella 02/14/2011!Stacy Marsella, PhD.!!The projects or efforts depicted were or are sponsored by the U.S. Army Research, Development, and Engineering Command (RDECOM) Simulation Training and Technology Center (STTC). The content or information presented does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.2!Requirements: Virtual Human Body"• Goal: Virtual human behavior that supports face-to-face interaction with humans"• Basic Physical Behavior"– Walking, grasping!• Nonverbal, expressive behavior"– Gestures, facial expressions, gaze!• Spontaneous, interactive"– Responsiveness to events!– Constraints on realization!3!Challenge: Complex Function of Nonverbal Behaviors"Interactional Start/end contact Show attention Hold conversation turn Request/provide feedback Discourse Refer Emphasize Depict shape/feature Socio-emotional Express emotions Show status Interpersonal relationship Impression management Physical Look lively, tired Smile Beat gesture Posture Shift Brow Raise Point Nods Shake Gaze aversion Body orientation Breathing head tilt … Functions Behaviors § Many-to-many mapping between comm. functions and behaviors!4!Interactional Start/end contact Show attention Hold conversation turn Request/provide feedback Discourse Refer Emphasize Depict shape/feature Socio-emotional Express emotions Show status Interpersonal relationship Impression management Physical Look lively, tired Smile Beat gesture Posture Shift Brow Raise Point Nods Shake Gaze aversion Body orientation Breathing head tilt … Functions Behaviors § Many-to-many mapping between comm. functions and behaviors!Challenge: Complex Function of Nonverbal Behaviors"5!Interactional Start/end contact Show attention Hold conversation turn Request/provide feedback Discourse Refer Emphasize Depict shape/feature Socio-emotional Express emotions Show status Interpersonal relationship Impression management Physical Look lively, tired Smile Beat gesture Posture Shift Brow Raise Point Nods Shake Gaze aversion Body orientation Breathing head tilt … Functions Behaviors § Many-to-many mapping between comm. functions and behaviors!Challenge: Complex Function of Nonverbal Behaviors"6!Challenge: Realization Matters"Dynamics matters"§ E.g., Gaze, glance, gape, glare, glower, look askew!– Different dynamics, postures & interpretations!– (Lance & Marsella 08;09;10)!§ E.g., Facial expression dynamics impact sense of trustworthiness (Krumhuber et al. 2007)!"Timing Matters"§ Nods, eyebrow flashes & gestures emphasize words!– Different timing, different emphasis!§ Poorly timed gestures influence attitudes about speaker!7!7 Why do we care: Reeves & Nass’s Media Equation • People tend to relate to media as they do to people • Confirmed by multiple experimental studies • Virtual Humans exploit this tendency8!Functions of Nonverbal Behaviors"• Socio-emotional Functions"– Approx. 60-65% of social meaning is derived from NVBs [Burgoon 1994]!– Mechanism of social control (status, persuasion, impression management) [Patterson 1990]!– Communicate emotion and interpersonal attitude [Bente et al. 2008]!– Relational communication (social support, comforting, conflict management) [Burgoon and Bacue 2003]!– Individual impact!• NVBs of virtual agents"– Affects learning performance [Rickenberg and Reeves 2000]!– Agent with human-like gaze rated more positively on usability, naturalness, satisfaction [Heylen et al. 2002]!– Process-oriented gestures (e.g. turn taking, beat gestures) lead to smoother interaction, better speech comprehension [Cassell and Thorisson, 1999]!– Mediate interpersonal trust [Bente et al. 2008]!– Greater sense of rapport [Wang et al. 2010]!– More fluent speech by participant [Von der Putten et al. 2009]!9!FACE"10!Psychological theories about facial expressions"• Facial Action Programs (Ekman)"– Emotion categories have associated facial action programs!• Basic emotions culturally universal (anger, surprise, disgust, sadness, happiness and fear)!– Expression is mediated by display rules!• Culture specific!• Component Models (Smith & Scherer)"– Facial Actions tied to appraisals!– Temporal course of appraisal revealed in expression dynamics!– Virtual human researchers have studied this mapping (Lisetti; Martin; )!• Communicative Act (Fridlund)"– Facial expressions are social messages not involuntary eruptions of emotional state!11!Components of Facial Expressions"• Duchenne: Studied muscles that control facial movement"• Used electrical stimulation!• Darwin’s work on the expression of emotion"• The Expression of Emotion in Man and Animals, 1872!• Built on Duchenne’s work!• FACS: Ekman and Friesen"• Derived the Facial Action Coding System, 1978!• Categorizes facial behavior as Action Units, tied to muscles!• Standard in psychological research!• Increasingly used in animation and virtual human research!11 11!12!FACS Action Units"• Example Action Units"• AU1 : Inner Brow Raiser AU2: Outer Brow Raiser!• AU4: Brow Lowerer AU5: Upper Lid Raiser!"• From Expression to Action Units"• Action Units 1+2+4+5!• Scared!12!13!• Dynamics and Emotion Perception – Genuine smiles have longer onset/offset times (Hess&Kleck90) – Smiles with longer onset judged more trustworthy, more attractive, and less dominant (Krumhuber&Kappas, 2005) – Smiles with long apex judged less authentic • Emotion Perception and decision making – Job applicants with “inauthentic smiles” were rated lower (Krumhuber&Manstead 2006) Apex Apex Facial Expression Dynamics14!Gesture"15!16!Gestures: Categories & Structure"• Types: Beat, metaphoric, iconic, deictic, emblems"• Timing and Structure"17!GAZE"18!Role of Gaze"• Attention"• Dialog Regulation"• Cognitive Load Regulation"• Deictic Reference"• Convey intentions, cognitive and emotional state"19!Expressive