A Data-Driven Paradigm to Understand Multimodal Communication in Human-Human and Human-Robot InteractionIntroductionData Collection – A Multimodal Sensing SystemCoding of Multimodal DataKnowledge Discover and Data MiningVisual Data Mining SystemInformation-Theoretic MeasuresInteractive Data MiningConclusionsReferencesP.R. Cohen, N.M. Adams, and M.R. Berthold (Eds.): IDA 2010, LNCS 6065, pp. 232–244, 2010. © Springer-Verlag Berlin Heidelberg 2010 A Data-Driven Paradigm to Understand Multimodal Communication in Human-Human and Human-Robot Interaction Chen Yu, Thomas G. Smith, Shohei Hidaka, Matthias Scheutz, and Linda B. Smith Psychological and Brain Scienecs and Cognitive Science Program, 1101 East 10th Street, Indiana University, Bloomington, IN, 47405 {chenyu,thgsmith,shhidaka,mscheutz,smith4}@indiana.edu Abstract. Data-driven knowledge discovery is becoming a new trend in various scientific fields. In light of this, the goal of the present paper is to introduce a novel framework to study one interesting topic in cognitive and behavioral stud-ies -- multimodal communication between human-human and human-robot in-teraction. We present an overall solution from data capture, through data coding and validation, to data analysis and visualization. In data collection, we have developed a multimodal sensing system to gather fine-grained video, audio and human body movement data. In data analysis, we propose a hybrid solution based on visual data mining and information-theoretic measures. We suggest that this data-driven paradigm will lead not only to breakthroughs in under-standing multimodal communication, but will also serve as a successful case study to demonstrate the promise of data-intensive discovery which can be ap-plied in various research topics in cognitive and behavioral studies. Keywords: Scientific Discovery, Cognitive and Behavioral Studies, Human-Human Interaction, Human-Robot Interaction, Information Visualization, Data mining. 1 Introduction With advances in computing and sensing technologies, the dominant methodology of science has been changing over recent years. Bell et al. [1] predicted that the first three paradigms in science – empirical, theoretical and computational simulation – have successfully carried us to where we are and will continue to make incremental progress, but meanwhile dramatic breakthroughs will be achieved by the next fourth paradigm of science – data-intensive science, which will help bring about a profound transformation of scientific research (see also [2]). In brief, a vast volume of scientific data captured by new instruments in various labs is likely to be substantially publically accessible for the purposes of continued and deeper data analysis. This analysis will result in the development of many new theories from such data mining efforts. Indeed, data-driven discovery has already happened in various research fields, such as earth sciences, medical sciences, biology and physics, to name a few. However, cognitive and behavioral studies still mostly rely on traditional experimental paradigmsA Data-Driven Paradigm to Understand Multimodal Communication 233 (reviewed below). The goal of the present paper is to introduce a contemporary frame-work to study one interesting topic in behavioral studies -- multimodal communication in human-human and human-robot interaction. We present an overall solution from data capture, through data coding and validation, to data analysis and visualization. We suggest that this data-driven paradigm will not only lead to breakthroughs in under-standing multimodal communication but also more generally serve as a successful case study to demonstrate the promise of this data-intensive approach which can be applied in many other research topics in cognitive and behavioral studies. Everyday human collaborative behavior (from maintaining a conversation to jointly solving a physical problem) seems so effortless that we often notice it only when it goes awry. One common cognitive explanation of how we (typically) manage to work so well together is called “mind-reading” [3]. The idea is that we form mod-els of and make inferences about the internal states of others; for example, along the lines of “He is pointing at the object, so he must want me to pick it up.” Accordingly, previous empirical methods on human-human communication are rather limited. For example, survey-based methods have been widely used to study human social interaction. This kind of measure relies on participants to recall and self-report their experiences, and although these reports may be predictive and diagnostic, they need not be objectively correct, and thus are at best an imperfect indication of what makes for “good” versus “not good” social interactions. Another popular approach is based on video coding in which human researchers code and interpret video data of human-human everyday interaction based on the prior notions about what is worth counting. But this rather subjective method may confirm what we already know but overlook important aspects of social interactions that we do not yet know – the ultimate goal of scientific discovery. It is not at all clear that mind-reading theories about the states of others – and in-ferences from such internal representations – can explain the real-time smooth fluidity of such collaborative behaviors as everyday conversation or joint action. The real-time dynamics of the behaviors of collaborating social partners involve micro-level behaviors, such as rapid shifts of eye movements, head turns, and hand gestures, and how the partners co-organize in an interaction. These behaviors seem to be composed of coordinated adjustments that happen on time scales of fractions of seconds and that are highly sensitive to the task context and to changing circumstances. Previous sur-vey and video-coding approaches don’t have access to such fine-grained behavioral data, to say nothing of interpreting such micro-level behaviors. An understanding of micro-level real-time behaviors, however, has potential applications to building effec-tive teams that can solve problems effectively, to building better social environments to facilitate human-human communication, to helping people that have various com-munication problems (e.g. autism), to building artificial agents (intelligent robots, etc.) that work seamlessly with people through human-like communication, and to building