Enhanced Exploration of Oral History Archives through Processed Video and Synchronized Text Transcripts Michael G. Christel, Scott M. Stevens, Bryan S. Maher Entertainment Technology Center Carnegie Mellon University Pittsburgh, PA 15213 USA +1 412 268 {7799, 7796, 8970} {christel, sms, bsm}@cs.cmu.edu Julieanna Richardson The HistoryMakers 1900 South Michigan Ave. Chicago, IL 60616 USA +1 312 674 1900 [email protected] ABSTRACT A digital video library of over 900 hours of video and 18000 stories from The HistoryMakers was used by 266 students, faculty, librarians, and life-long learners interacting with a system providing multiple search and viewing capabilities over a trial period of several months. User demographics and actions were logged with this multimedia collection, providing quantitative and qualitative metrics on system use. These transaction logs were complemented with heuristic evaluation, interviews, and contextual inquiry with representative users. Collectively, these mixed methods informed the development of the next generation web-based interface for the HistoryMakers video oral histories to improve access to and dissemination of this rich cultural resource. In particular, the feature of a synchronized text transcript in the video player for the narratives merited further investigation. Such an interface has not seen widespread use in digital video players available on the web, yet was valued highly by oral history archive viewers. A user study with 27 participants measured the utility of the HistoryMakers web interface incorporating the synchronized transcript video player for stated fact-finding and open-ended tasks. For life oral histories, an aligned text transcript is valued for both tasks, with the video rated significantly more useful for open-ended tasks over fact-finding. These results suggest a task-dependent role of modality in presentation of oral histories, with synchronized transcripts rated highly across tasks. Categories and Subject Descriptors H5.1 [Information Interfaces and Presentation]: Multimedia Information Systems. General Terms Design, Experimentation, Human Factors. Keywords Oral histories, video retrieval, exploratory search. 1. INTRODUCTION Oral history has long been recognized as offering a valuable method for recording and studying memories, with extensive life histories representing a particularly valuable dimension for preserving cultural heritage. Oral history has been embraced as a tool “for giving voice to those who have been excluded from the historical record” [5]. Oral histories can “add richness and personal perspective to the historical record and can engage students and scholars in a lively study of history” [6]. Contextual inquiry has found that oral history recordings are considered a central historical artifact [11]. While the value of oral history has been recognized, its access medium has often been purely text, rather than audio or video [11]. In discussing recorded speech, Vemuri and colleagues discuss one reason why: aural speech delivery presents unique challenges [17]. The average speech rate of an English speaker is over twice as slow as the average reading rate. This large disparity suggests that automatically transcribing audio and then accessing it as a written document would be most effective for information retrieval tasks. However, in reading a text transcript, the prosodic cues, which make speech rich in meaning and subtlety, are lost [17]. Furthermore, although video has a level of emotion not available in text transcripts, and the video recordings can help transcript readers clarify text and observe non-verbal cues, “these recordings sit unused after a written transcript is produced” [11]. De Jong, Oard, and colleagues note that significant hurdles in oral history access are brought upon by the linear nature of speech and the extended length of some interviews. They comment that multimedia processing and delivery technologies offer the potential to radically transform the way in which oral histories are made accessible, encouraging and promoting the use of audio and audiovisual records [10]. Automatic transcription, improved alignment of existing transcripts, and additional metadata, particularly at the story segment level, are suggested as means of overcoming the tediousness of navigating through thousands of hours of linear audio and video. The Informedia research group at Carnegie Mellon University has worked with an oral history archive, The HistoryMakers, to apply such automated techniques and generate time-aligned metadata for use in accessing the video narratives [3]. A lab study in 2007 showed the value of representing the oral histories in video form [4]. This paper reports on recent field studies conducted across a number of institutions and broad set of users, Empirical data of what happens through transaction logs is coupled with subjective reports on why, through interviews and contextual inquiry, leading to the development of a new web multimedia interface. One interface element in particular enabled archive users to have greater utility with the oral history videos: the display of a Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM’10, October 25–29, 2010, Firenze, Italy. Copyright 2010 ACM 978-1-60558-933-6/10/10...$10.00.synchronized transcript in the video player, which is investigated further in a study of a web portal interface. The value of improved navigation into linear media through text transcripts has been acknowledged for webcast lectures [15] and discussed in the context of experimentation with error-laden transcripts from automatic speech recognition (ASR) [17]. The inherent value of a searchable transcript for navigating into linear audio (or the narrative audio track of linear video) can be seen with