Christoph W. BorstCenter for Advanced ComputerStudiesUniversity of Louisiana at LafayetteLafayette, [email protected] A. VolzDepartment of Computer ScienceTexas A&M UniversityPresence, Vol. 14, No. 6, December 2005, 677– 696© 2005 by the Massachusetts Institute of TechnologyEvaluation of a Haptic MixedReality System for Interactionswith a Virtual Control PanelAbstractWe present a haptic feedback technique that combines feedback from a portableforce-feedback glove with feedback from direct contact with rigid passive objects.This approach is a haptic analogue of visual mixed reality, since it can be used tohaptically combine real and virtual elements in a single display. We discuss devicelimitations that motivated this combined approach and summarize technologicalchallenges encountered. We present three experiments to evaluate the approachfor interactions with buttons and sliders on a virtual control panel. In our first ex-periment, this approach resulted in better task performance and better subjectiveratings than the use of only a force-feedback glove. In our second experiment, vi-sual feedback was degraded and the combined approach resulted in better perfor-mance than the glove-only approach and in better ratings of slider interactions thanboth glove-only and passive-only approaches. A third experiment allowed subjectivecomparison of approaches and provided additional evidence that the combinedapproach provides the best experience.1 IntroductionOne of the most promising advances for virtual environments (VEs) isthe development of displays that provide force or tactile feedback during inter-actions with virtual objects. These displays, called haptic displays, can increasethe realism of VEs and communicate information that improves user perfor-mance or understanding. There is a growing interest in developing haptic dis-plays and in understanding their effect on users.1.1 TerminologyWe use the adjectives active and passive to describe haptic devices thatuse computer-controlled actuators and those that do not, respectively. Notethat the terms are being used to describe a feedback device characteristic andnot to refer to Gibson’s classification of a human’s mode of touching the de-vice (Gibson, 1962, 1966).Examples of active devices include joysticks with force feedback, pin arraysfor tactile feedback to fingertips, vibrotactile devices using piezo elements orsmall motors, thermal displays using Peltier modules, and specialized trainingdevices such as syringe bodies with embedded force-feedback components.Conventional input devices encountered on a daily basis are passive. TheBorst and Volz 677click of a mouse button or keyboard key is a type of pas-sive haptic feedback that can be carefully designed toprovide useful information to a user. Terms such as pas-sive haptics, static haptics, tactile augmentation, and in-strumented objects have been used to refer to approachesusing rigid objects in the real world to provide tactilestimulation to users interacting with VEs (Boud, Baber,& Steiner, 2000; Hoffman, 1998; Insko, Meehan,Whitton, & Brooks, 2001; Lindeman, Sibert, & Hahn,1999). Benefits of these approaches over active approachesinclude low cost and low mechanical complexity.1.2 A Mixed Haptic FeedbackApproachWe present a combination of passive hapticswith an active force-feedback device. Specifically, weinvestigate the combined use of a glove-mountedforce-feedback device and a passive panel for the vir-tual control panel application shown in Figure 1. TheVE consists of a room and a control panel located ona table in front of the user. The panel includes sliders,buttons, and LED readouts. Figure 2 shows an exter-nal view of a user performing interactions with thevirtual control panel (the specific interactions are notexactly identical to those seen in Figure 1). The userwears a head-mounted display and a force-feedbackglove that provides forces of interaction for sliderhandles and short force pulses that indicate the reac-tion of buttons. The panel surface itself is felt as aresult of contact with a real panel that is spatially reg-istered with the panel in the visual display. The realtable surface is also registered with its virtual counter-part.1.3 The Mixed Approach as anExtension of Mixed RealityWe present the combination of passive hapticswith an active feedback device as a form of mixed real-ity. It is the haptic analogue of visually mixed displays,since it can allow real and virtual components to becombined haptically. It is also a logical extension to ex-isting mixed reality systems, which do not yet allowboth real and virtual objects to be felt. Although weevaluated the technique with purely virtual visual feed-back, it can be integrated with a visual mixed reality dis-play in the future to produce a system that combinesreal and virtual elements both visually and haptically.1.4 The Mixed Approach as anEnhancement to Other ApproachesThe mixed approach overcomes some limitationsof glove-only and passive-only approaches. PortableFigure 1. Virtual visual environment.678 PRESENCE: VOLUME 14, NUMBER 6glove-mounted devices can provide internal forces ofgrasping, but they have a number of limitations, as de-tailed in Section 3.2 (we use the term portable, as usedby Burdea [1996], to refer to body-grounded inter-faces). For our virtual control panel application, theycannot provide proper interaction forces at the virtualpanel surface. On the other hand, a passive approachusing static objects is promising during contact withstatic virtual objects such as the panel surface, but astatic object does not provide feedback for contact withdynamic slider handles or for button reactions (buttonclicks). A passive panel with real sliders and buttons ispossible, but this increases panel complexity and doesnot support a highly reconfigurable virtual panel. Thecombination of the active and passive approaches hasthe potential to overcome glove limitations without re-quiring additional devices that limit mobility or arecumbersome for the user. The cost and mechanicalcomplexity of introducing the passive component isminimal. The force-feedback glove can provide feedbacknot available with the passive approach alone.1.5 Applications of a Mixed ApproachA mixed haptic feedback approach is useful in avariety of applications. For example, during ergonomicdesign of dashboards, real dashboard components couldprovide feedback for completed design portions while ahaptic glove