Stitching: Pen Gestures that Span Multiple Displays Ken Hinckley1, Gonzalo Ramos1,2, Francois Guimbretiere1,3, Patrick Baudisch1, and Marc Smith1 1Microsoft Research One Microsoft Way Redmond, WA 98052 +1 425 703 9065 {kenh, baudisch, masmith} @microsoft.com 2University of Toronto 40 St. George St. Toronto, ON M5S 2E4 +1 (416) 946-8491 [email protected] 3University of Maryland 3267 AV Williams Bldg College Park, MD 20742 +1 (301) 405 7952 [email protected] ABSTRACT Stitching is a new interaction technique that allows users to combine pen-operated mobile devices with wireless networking by using pen gestures that span multiple displays. To stitch, a user starts moving the pen on one screen, crosses over the bezel, and finishes the stroke on the screen of a nearby device. Properties of each portion of the pen stroke are observed by the participating devices, synchronized via wireless network communication, and recognized as a unitary act performed by one user, thus binding together the devices. We identify the general requirements of stitching and describe a prototype photo sharing application that uses stitching to allow users to copy images from one tablet to another that is nearby, expand an image across multiple screens, establish a persistent shared workspace, or use one tablet to present images that a user selects from another tablet. We also discuss design issues that arise from proxemics, that is, the sociological implications of users collaborating in close quarters. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: Input Devices and Strategies, Interaction Styles General Terms Human Factors, Design Keywords pen computing, mobile devices, co-located collaboration, proxemics, synchronous gestures, spontaneous device sharing 1. INTRODUCTION With wirelessly networked mobile devices rapidly becoming ubiquitous, it is increasingly important to find effective methods for dynamically binding together mobile devices in support of sharing information, collaborating and communicating with other persons, or performing complex tasks involving large documents and multiple sources of information. The spontaneous device sharing problem poses the following question: how can a user dynamically forge a purposeful connection between two or more target devices that do not a priori know one another’s network address? This is not only a significant research problem for system implementation, interaction design, and social and behavioral observation, but also is becoming a problem with a pressing need for practical solutions. Researchers have recently discovered a new class of techniques that use synchronous user actions (known as synchronous gestures) to address the spontaneous device sharing problem without forcing the user to manually enter network addresses. Examples of these techniques include holding two devices together and shaking them [13], bumping a pair of devices together [12], or simultaneously pressing a button on each device [19]. However, the design space of synchronous gestures is still not well understood and needs further exploration. path taken by the pentransferred picturespath taken by the pentransferred picturespath taken by the pentransferred pictures Figure 1. Here, a user gives some photos to another user by stitching from the top tablet to the bottom tablet. Here, we contribute stitching, a new type of synchronous gesture that uses commonplace pen input capabilities to establish and manage serendipitous connections between pen-operated mobile devices. A stitching gesture is a pen gesture that spans multiple displays, consisting of a continuous pen motion that starts on one device, skips over the bezel of the screen, and ends on the screen of another device (Fig. 1). We show how stitching serves as the basis for a flexible connection architecture that can be 23 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AVI 2004, Gallipoli (LE), Italy © 2004 ACM 1-58113-867-9/04/0500..$5.00 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AVI '04, May 25-28, 2004, Gallipoli (LE), Italy © 2004 ACM 1-58113-867-9/04/0500..$5.00implemented on a variety of pen-operated devices. To explore the pragmatics of stitching, we implement a prototype photo sharing application for the Tablet PC that supports operations such as copying images from one tablet to another that is nearby, establishing a persistent shared workspace for collaboration, expanding an image across multiple screens, or using one tablet to display a slideshow of images that a user selects from another tablet. Preliminary usability testing suggests that users can easily make pen gestures that span displays, and find it compelling to have a straightforward means to perform cross-device operations. In addition to the new technique of stitching, more generally we contribute six requirements for the spontaneous device sharing problem, phrased as design questions: • Connection: How is a connection established? • Command: What type of connection is required? • Operands: What information is shared? • Geometry: What is the spatial relationship between devices? • Coexistence: How do connection gestures coexist with traditional interactions or naturally occurring user behaviors? • Proxemics: How do users share physical space? The literature presently lacks such a discussion of general requirements. In this sense our research represents a framework that considers cross-device operations in general, including the system, interaction, and social issues that arise. 2. RELATED WORK Our work is the first to address all six requirements listed above. Several previous systems foster collaboration