Lifelogging Memory Appliance for People with Episodic Memory Impairment Matthew L. Lee & Anind K. Dey Human-Computer Interaction Institute, Carnegie Mellon 5000 Forbes Avenue, Pittsburgh, PA 15213 USA {mllee, anind}@cs.cmu.edu ABSTRACT Lifelogging technologies have the potential to provide memory cues for people who struggle with episodic memory impairment (EMI). These memory cues enable the recollection of significant experiences, which is important for people with EMI to regain a sense of normalcy in their lives. However, lifelogging technologies often collect an overwhelmingly large amount of data to review. The best memory cues need to be extracted and presented in a way that best supports episodic recollection. We describe the design of a new lifelogging system that captures photos, ambient audio, and location information and leverages both automated content/context analysis and the expertise of family caregivers to facilitate the extraction and annotation of a salient summary consisting of good cues from the lifelog. The system presents the selected cues for review in a way that maximizes the opportunities for the person with EMI to think deeply about these cues to trigger memory recollection on his own without burdening the caregiver. We compare our system with another review system that requires the caregiver to repeatedly guide the review process. Our self-guided system resulted in better memory retention and imposed a smaller burden on the caregiver whereas the caregiver-guided approach provided more opportunities for caregiver interaction. Author Keywords Lifelogging, information overload, Alzheimer’s disease, memory impairment, caregiver burden. ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION Episodic memory impairment (EMI) is the main symptom associated with Alzheimer’s disease, a common neurodegenerative disease that affects over 26 million people worldwide, with this number expected to quadruple by 2050 [2]. Recent episodic memory supports our sense of self [4] by enabling us to mentally travel back in time and relive our pleasant experiences and to be socially and physically engaged in our lives. Recent episodic memory impairment dramatically changes the lives of individuals with the impairment. Individuals struggle with the need to constantly readjust their own expectations of what they can and can no longer do. EMI can lead to a loss in autonomy and control in individuals’ lives, resulting in feelings of uncertainty, irritation, and frustration as they attempt to compensate for their memory loss but repeatedly fail. They may withdraw from interacting socially with others to avoid appearing incompetent and even develop depression as a result [17]. They must rely on others for support and often must repetitively ask their family caregivers for information about current and recent events. The lives of their family caregivers are also dramatically changed as caregivers become overburdened with providing for both the cognitive and physical needs of their loved one and can themselves develop depression or burnout that leads to reduced quality of care [1]. Thus, individuals with EMI and their caregivers struggle for a sense of normalcy—how things were before the onset of the disease—in their lives [3, 4]. In other words, their ideal situation would be to turn back the clock and live in a time before the disease changed their lives, when their lives were “normal.” Restoring normalcy in their now changed lives involves restoring the memory abilities and independence that the disease took away from them. Ubiquitous lifelogging systems (e.g., [12], [13], [19]) use wearable or embedded sensor technologies such as cameras, audio recorders, location trackers, and physiological sensors to passively and automatically record a user’s personal experiences. Lifelogging technologies allow people with EMI to automatically record, review, and thus regain an awareness of meaningful personal experiences in their lives to maintain their sense of self [13]. Sellen et al. [15] showed that episodic details from a visual “lifelog” can be presented to users as memory cues to assist them in remembering the details of the original experience. Other successful systems leverage other modalities such as the Audio Memory Prosthesis [19] that records audio from Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UbiComp'08, September 21-24, 2008, Seoul, Korea. Copyright 2008 ACM 978-1-60558-136-1/08/09...$5.00. 44personal experiences and presents it as searchable and browsable text. The Personal Life Log system [18] uses a combination of location sensors, physiological sensors, and real-time voice annotation to identify potentially interesting scenes in a continuous video log. However, lifelogging technologies often automatically capture an overwhelmingly large amount of data that can be both tedious and difficult to review. In particular, people with EMI often have other cognitive impairments that make it extremely difficult to review and engage with vast amounts of content that most lifelogging systems generate [16]. To be most effective in supporting memory, lifelogging systems need to create a salient summary of the lifelog that provides the most important information to present to the user in an engaging way. One successful system, the Microsoft SenseCam, leverages sensors to selectively capture information and also leverages the caregiver to further select the important information [13]. The system uses a wearable digital camera that automatically takes photos triggered by onboard sensors. It was found to be helpful