ABSTRACTGeneral Terms:Author KeywordsACM Classification KeywordsINTRODUCTIONAn ExamplePrior WorkThe Pervasive Annotation Architecture ProblemRequirements for an Annotation ToolCAPTURING ANNOTATIONSCOMMENTSHIGHLIGHT/IMAGE REGION ASSOCIATIONContinuity ImagesStroke/Region AssociationCircles and scribblesUnderlinesHorizontal highlightsMargin barsDocument Analysis AlgorithmsRegion Association Algorithm VerificationWORKING WITH CRAYONS AND NOTESViewing NotesAnimation of Note View TransitionsUSEFULNESS OF SCREEN CRAYONSREFERENCESScreenCrayons: Annotating Anything Dan R. Olsen Jr., Trent Taufer, Jerry Alan Fails Brigham Young University Computer Science Department, Provo, Utah, USA [email protected], ABSTRACT ScreenCrayons is a system for collecting annotations on any type of document or visual information from any application. The basis for the system is a screen capture upon which the user can highlight the relevant portions of the image. The user can define any number of topics for organizing notes. Each topic is associated with a highlighting “crayon.” In addition the user can supply annotations in digital ink or text. Algorithms are described that summarize captured images based on the highlight strokes so as to provide overviews of many annotations as well as being able to “zoom in” on particular information about a given note and the context of that note. General Terms: Human Factors Author Keywords Annotation, Screen capture, digital ink, image summarization. ACM Classification Keywords H.5.2 User Interfaces INTRODUCTION This paper describes the ScreenCrayons system for capturing and managing annotations for a variety of tasks using any application. The goal is for the system to be very flexible, lightweight and widely applicable. The fundamental metaphor for the modern office workstation has been paper. Beginning with the design of the Xerox Star, windows have been modeled as active sheets of paper. The most commonly used word-processing, drawing and spreadsheet applications all use paper as their metaphor. In the days before computers, creating documents was hard (typewriters are not error-friendly). Distributing paper documents was hard. Modifying paper documents was hard. Annotating paper documents by making marks on them was easy. Because of this disparity of labor, the focus of most office tools has centered on the creation and dissemination of documents that can readily be rendered onto paper. In this context the process of annotating the paper received much less attention despite its importance in actual use. The advent of very cheap storage, cheap communication via the Internet, standard formats such as PDF or HTML and the pervasive availability of computing has caused a shift in our usage of documents. For an increasing number of people, the majority of their reading experience is digital rather than paper. Email has rapidly replaced the paper letter for much correspondence. Scholars increasingly subscribe to digital libraries rather than print journals. Technical manuals and promotional materials increasingly come through the web. Adler et al [1] have reported that reading occupies 70% of document-related activity. However, for many subjects a substantial amount of reading time occurred in conjunction with writing. In the same study creation and updating of documents constitutes only 18% of writing while reading. On the other hand annotation and note taking consume 48% of the time. Schilit describes this as “active reading” [19] where the user is augmenting, filtering, highlighting, summarizing and organizing the information that they are reading. What we need are widely applicable computer-based tools that support this activity. We will first provide an extended example of the range of annotations that we expect of our system. We then will review prior work in annotation, followed by the architectural issues with being able to annotate anything and review our notes in a meaningful way. This is followed by a description of the note taking process along with algorithms for associating image regions with our highlight marks. Lastly we discuss how we use these highlights, regions and notes to provide summarized views. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UIST ’04, October 24–27, 2004, Santa Fe, New Mexico, USA. Copyright © 2004 ACM 1-58113-957-8/04/0010. . . $5.00. An Example Consider Fred the biologist. Fred is an expert on nematode genomics. When he starts work one morning, he begins to read his email and finds that one of his students has posted a copy of her thesis on a web site so that he can review it. Fred wants to complete his email so he makes a note of where the thesis is located and continues. He soon finds that 165Volume 6, Issue 2he is requested to do an urgent review for a journal paper. Again he makes a note of the paper and continues. He finds a message with a budget for a research proposal. He notes that the budget’s travel and fringe benefits are not correct and sends the note back to his colleague. He also finds a message from a student indicating the results of a successful sequencing analysis that he adds to his list. Fred finishes his email, reviews his “todo” notes and decides that his student’s experiments are the most important. Fred then runs a special sequencing program that shows the results of the latest supercomputer run. He is excited by the results but not completely happy with the software settings that were used. He makes a note on those settings and forwards it to his student. Fred then downloads the thesis and opens it in his word processor. While reading he makes notes of corrections. In the middle of his reading he thinks of a related paper that his student has not seen. He opens a web browser and begins a search for “nematodes.” While doing the search he sees a page for a new center for nematode genomics. He makes a note to remember to review the site later. He also encounters the “Nematode Songbook” [17] with a hilarious rendering of an old western tune redone with worms. He notes this in his list of “worm humor” to