Evaluating User Interface Systems Research Dan R. Olsen Jr. Brigham Young University Computer Science Department, Provo, Utah, USA [email protected], ABSTRACT The development of user interface systems has languished with the stability of desktop computing. Future systems, however, that are off-the-desktop, nomadic or physical in nature will involve new devices and new software systems for creating interactive applications. Simple usability testing is not adequate for evaluating complex systems. The problems with evaluating systems work are explored and a set of criteria for evaluating new UI systems work is presented. ACM Classification Keywords H.5.2 User Interfaces General Terms: Human Factors Author Keywords: User Interface Systems Evaluation INTRODUCTION In the early days of graphical user interfaces, the creation of new architectures for interactive systems was a lively and healthy area of research. This has declined in recent years. There are three reasons for this decline in new systems ideas. The first is that, unlike those early days, there are essentially three stable platforms (Windows, Mac, Linux) upon which virtually all software is built and those platforms have dictated the user interface architecture. This is in contrast to the state of UI research 15 years ago when there were many competing toolkits and platforms. The second is that the stability of these platforms has lead to a new generation of researchers who lack skills in toolkit or windowing system architecture and design. The third reason is the lack of appropriate criteria for evaluating systems architectures. This paper addresses the last question of “How should we evaluate new user interface systems so that true progress is being made?” WHY UI SYSTEMS RESEARCH? Before addressing the evaluation question we must first consider the value of user interface systems research. The systems we have are stable. Applications are being written. Work is progressing. The users are happy (sort of). Why then does the world need yet another windowing system? Forces for change A very important reason for new UI systems architectures is that many of the hardware and operating system assumptions that drove the designs of early systems no longer hold. Saving a byte of memory, the time criticality of dispatching an input event to the right window or lack of CPU power for geometric and image transformations are no longer an issue. Yet those assumptions are built into the functionality of existing systems. The constraints of screen size are rapidly falling and we are finding that interaction in a 10M pixel space is very different from interaction in a 250K pixel space. Our assumptions about users and their expertise have radically changed. Most of our windowing systems are designed to deal with a populace who had never used a graphical user interface. That assumption is no longer valid. The rising generation is completely comfortable with computing technology in a variety of forms and is increasingly comfortable with change. Our existing system models are barriers to the inclusion of many of the interactive techniques that have been developed. Research as shown that manipulating the mouse gain can improve selection in various spaces [1] yet this does not fit smoothly into any UI system model. Cameras and touch tables produce inputs that are the size of a hand or finger rather than a point, yet we force such techniques into the standard mouse point model because that is all that our systems support. Multiple input points and multiple users are all discarded when compressing everything into the mouse/keyboard input model. Lots of good research into input techniques will never be deployed until better systems models are created to unify these techniques for application developers. The advent of new interactive platforms also drives a need for new systems architectures. The WWW forms a huge base of interactive use, yet its interaction model is primitive and the toolkits built around it are difficult. People are increasingly moving their digital lives to PDAs, cell phones Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee. UIST’07, October 7–10, 2007, Newport, Rhode Island, USA. Copyright 2007 ACM 978-1-59593-679-2/07/0010...$5.00. 251and other portable platforms. Many people live and work across many platforms and interact with many people, yet our UI systems architectures support none of this. In the early days of the Mac and Windows, an industry leader said “Almost none of our customers own a mouse or a graphics card. The installed market is too large and too entrenched to change.” Barely 5 years later that company had fallen from market dominance to near irrelevance because the market had moved to screen/keyboard/mouse. We now stand at a similar position. Systems based on one screen, one keyboard and one mouse are the new equivalent of command-line interfaces. We need new UI systems solutions. Our current systems architectures are beginning to impede progress rather than empower it. Value added by UI systems architecture Before addressing the evaluation of research claims we should look at the value UI systems architectures bring to the table. Reduce development viscosity A good UI toolkit will reduce the time it takes to create a new solution. The faster new solutions can be created, the easier it is to try more solutions on users. The more solutions that users experience, the more effective the UI design process will be. Least resistance to good solutions UI programmers, like most programmers, are optimizers. They tend to follow the path of least resistance. Apple spent a lot of time and effort writing a style manual and evangelizing the value of a common look and feel. None of this effort had the impact of providing a standard widget set built into the system that was available for free. The toolkit made adoption of a common look and feel much cheaper and easier than custom solutions. A related concept is that toolkits can encapsulate and simplify expertise. When exploration of a space of possibilities finally settles on a few good solutions, these can