ABSTRACTINTRODUCTIONCrayons OverviewUSER INTERFACECRAYONS CLASSIFIERMachine LearningFeaturesEVALUATIONRELATED WORKCONCLUSIONREFERENCESA Design Tool for Camera-based Interaction Jerry Alan Fails, Dan R. Olsen Computer Science Department Brigham Young University Provo, Utah 84602 {failsj, olsen}@cs.byu.edu ABSTRACT Cameras provide an appealing new input medium for interaction. The creation of camera-based interfaces is outside the skill-set of most programmers and completely beyond the skills of most interface designers. Image Processing with Crayons is a tool for creating new camera-based interfaces using a simple painting metaphor. A transparent layers model is used to present the designer with all of the necessary information. Traditional machine learning algorithms have been modified to accommodate the rapid response time required of an interactive design tool. Categories & Subject Descriptors: H.1.2 [Information Systems]: User/Machine Systems — Human factors; H.5.2 [Information Interfaces and Presentation]: User Interfaces — interaction styles, prototyping, theory and methods; I.4.8 [Image Processing and Computer Vision]: Scene Analysis — color, object recognition, tracking; I.4.9 [Image Processing and Computer Vision]: Applications; General Terms: Algorithms, Human Factors; Keywords: Image processing, classification, interaction, machine learning, perceptive user interfaces. INTRODUCTION Myron Krueger’s VideoPlace [20] established cameras as an interactive input medium and sparked an interest in user interfaces that function by watching what people do rather than requiring overt inputs by a user. Mark Weiser popularized the notion of ubiquitous computing [32] where computing moved into the environment as a whole rather than residing on a desk. Hiroshi Ishii and his Tangible Media Group [14,28,29] have pioneered demonstrations of how computing and interaction can be embedded in everyday things. Achieving these visions requires a much larger range of interactive sensors than can be found on common desktop systems. In this work we are interested particularly in camera-based interfaces. We chose the camera because of its ambient nature in that it passively watches without requiring the user to wear or carry anything special. Cameras are currently quite cheap with their costs rapidly approaching the cost of a keyboard and mouse. Projects such as the Gesture Pendant [26], Light Widgets [9], finger tracking [6], hand waving [11] and Laser Pointer Interaction [22] have demonstrated how cameras can form the primary sensor for an interactive behavior. The challenge, however, is that designing camera-based interfaces is quite difficult. Not only is sophisticated programming required, but also the mathematics of image processing and machine learning. These are far beyond the skill set required for Visual Basic. In our Image Processing with Crayons (Crayons) project we are focused on creating interface development tools that can be placed in the hands of ordinary designers. Since there is no extant design community for camera-based interfaces we have made several assumptions about the skill set such a community might posses. • Familiarity with visual design tools like Visual Basic, and how they integrate with other software, • No familiarity with image processing or machine learning, • An understanding that image regions must be classified to identify the items of interest to the interaction, but little understanding of how that might be done. At the heart of any camera-based interaction is a classifier that takes an image and identifies those pixels or groups of pixels that are of interest to the interaction. This is so because once a programmer knows where items of interest are found in an image, the remaining programming is very similar to mouse-based interaction. The classifier is the particular part of camera-based interaction that most programmers shy away from. Because UI designers rarely have detailed image processing or machine learning knowledge, they must learn these skills before even beginning to build such a classifier. Once the knowledge of how to build the classifier has been attained the implementation is not usually difficult, although there are still many tricks and nuances that require repetitive testing and alterations. Kernels, filters and machine learning algorithm parameters can be tricky and temperamental and generally require many modifications. In our experience, adjustments to a manually created Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2003, April 5-10, 2003, Ft. Lauderdale, Florida, USA. Copyright 2003 ACM 1-58113-630-7/03/0004...$5.00. Ft. Lauderdale, Florida, USA • April 5-10, 2003 Paper/Demos: Camera-based Input and Video Techniques Displays Volume No. 5, Issue No. 1 449classifier may take weeks, if not months to finally achieve satisfactory results. It is important to note that image classifiers can be difficult for even knowledgeable programmers to get right. In the traditional approach, a programmer must identify a set of features that are sufficient for the task. Most machine learning algorithms assume that substantial time and thought will go into feature selection before the classifier algorithm is even invoked. Such features may be as simple as the hue and saturation values for a particular pixel or more complicated convolution kernels for separating desired pixels from similar ones. A programmer can spend months researching, programming and trying different combinations of features. Such activities are quite beyond the skills of the average interface designer. However, understanding the interaction problem is well within the capacities of an interface designer. Consider the example of laser-pointer interaction. Once the laser spot is accurately located the remaining interactions are very similar to mouse interactions. Anyone looking at the camera images can readily identify the key problem as “find that red spot.” Finding the red spot, however, is not trivial due to the laser overdriving