1 Introduction2 Types of Interaction3 Multimodal Environments3.1 Visual3.2 Auditory3.3 Haptic3.4 Olfactory3.5 Software Architecture4 Dynamic Environments4.1 The Basics4.2 Rigid and Deformable Body Dynamics4.3 Fluid Dynamics4.4 Fracture4.5 Benefits5 GEMINI5.1 Previous Work5.1.1 The Services5.1.2 The Communication Service5.2 GEMINI5.2.1 Flexible Communication5.2.2 Flexible Protocols5.3 Collaborative Interaction6 Conclusions and Future Work7 Acknowledgements8 ReferencesInteraction within Multimodal Environments in a Collaborative SettingGlenn A. Martin, Jason Daly and Casey ThurstonInstitute for Simulation & Training, University of Central Florida3280 Progress Dr., Orlando, FL 32826{martin, jdaly, cthursto}@ist.ucf.eduAbstractMuch attention has been given to the topic of multimodal environments as of late. Research in haptics is thrivingand even some olfactory research is appearing. However, while there has been research in collaborativeenvironments in the general sense, minimal focus has been given to the interaction present in multimodalenvironments within a collaborative setting. This paper presents research into a network architecture forcollaborative, multimodal environments with a focus on interaction.Within a dynamic world three types of human interaction are possible: user-to-user, world-to-user, and user-to-world. The first two form what are fundamentally multimodal environments; the user receives feedback usingvarious modalities both from other users and the world. The last one forms the area of dynamic environments wherethe user can affect the world itself. The architecture presented here provides a flexible mechanism for studyingmultimodal environments in this distributed sense. Within such an architecture, latency and synchronization are keyconcepts when bringing multimodal environments to a collaborative setting. 1 IntroductionVirtual environments have existed for many years. However, despite numerous distributed simulations, includingtraining uses in many domains, there still is relatively little interaction within these virtual environments. Mostallow only navigation and perhaps some minimal interaction. For example, military simulations allow basiccombat-oriented interaction (shooting, throwing a grenade, etc.), but only few of those incorporate advancedmodalities (such as haptics or olfactory) or dynamic changes to the environment itself.2 Types of InteractionWithin a distributed, interactive virtual environment there are many components to address: the user, theenvironment and other users. In categorizing the types of interaction we have split them into four categories: user-to-user, world-to-user, user-to-world and world-to-world. The first two types form what are fundamentallymultimodal environments. Whether from another user or from the world, the user gets feedback from the variousmodalities. In the case of user-to-user it may be grabbing a shoulder or shooting an opponent. For world-to-user itmay be bumping into a piece of furniture in the virtual environment.The latter two types of interaction form what we call dynamic environments. For example, the user can act upon theworld by moving an object or changing the characteristics of an object (e.g. watering a virtual plant making the soildamper). Similarly, within a world-to-world interaction a rain model might change the soil moisture. In this paperwe will focus on the user-to-world interactions, however.3 Multimodal EnvironmentsIn terms of virtual environments, a multimodal environment is one that the user experiences and interacts with usingmore than one sense or interaction technique. Strictly speaking, an environment that provides visual and auditoryfeedback is a multimodal environment; however, the more interesting environments are those that employ hapticand/or olfactory feedback in addition to visual and auditory. Gustatory, or taste feedback, is also possible, but thereis almost no evidence of this in the literature, so we will not discuss it here. On the interaction side, systems thattake input using more than one technique can also be considered multimodal. For example, a system that provides ajoystick for locomotion as well as a speech recognition system for accepting commands to alter the environment canbe called a multimodal environment. In this section, we focus on the senses and how the user experiences amultimodal environment, as well as a software architecture that supports the modeling and design of multimodalenvironments in a seamless way. Though it is difficult to develop a clear metric to determine whether adding moresenses contributes to a greater sense of presence or immersion, there have been studies conducted that suggested this(Meehan, Insko, Whitton & Brooks, 2002). Also, Stanney et al. have shown that adding sensory channels allows theuser to process more data than the same amount of information presented over a single sensory channel (2004). 3.1 VisualApart from a few specialized applications, a virtual environment always provides visual feedback. The visual senseis essential for navigation and detailed exploration of the environment. While other senses may introduce an event,entity, or object to investigate, it is ultimately the visual sense that will be used to analyze and deal with the object ofinterest. For this reason, an environment designed to be multimodal cannot ignore or diminish the importance ofvision, and an appropriate amount of resources should be spent on creating high-fidelity visuals. Visual feedback isessential to all types of collaborative interaction. Signals, gestures, and other visual interaction allow user-usercollaboration. The primary means for the user to experience the world is through visual exploration. Finally, theability for the user to visually perceive the world is a prerequisite for user-world interaction.3.2 AuditoryMost modern environments also provide some level of auditory feedback. It may be as simple as a beep or clickwhen a certain event happens, or it may involve realistic, spatialized sounds coordinated with virtual objects andenhanced with environmental effects. The auditory sense is a very useful and natural channel for informing the userof events that occur outside his or her field of view. Simple speech is
View Full Document
Unlocking...