SenSay: A Context-Aware Mobile Phone Daniel Siewiorek, Asim Smailagic, Junichi Furukawa, Neema Moraveji, Kathryn Reiger, and Jeremy Shaffer Human Computer Interaction Institute and Institute for Complex Engineered Systems Carnegie Mellon University Pittsburgh, PA 15213, USA {dps, asim}@cs.cmu.edu, {junichif, nmoravej, kr, jshaffer}@andrew.cmu.edu Abstract SenSay is a context-aware mobile phone that adapts to dynamically changing environmental and physiological states. In addition to manipulating ringer volume, vibration, and phone alerts, SenSay can provide remote callers with the ability to communicate the urgency of their calls, make call suggestions to users when they are idle, and provide the caller with feedback on the current status of the Sensay user. A number of sensors including accelerometers, light, and microphones are mounted at various points on the body to provide data about the user’s context. A decision module uses a set of rules to analyze the sensor data and manage a state machine composed of uninterruptible, idle, active and normal states. Results from our threshold analyses show a clear delineation can be made among several user states by examining sensor data trends. SenSay augments its contextual knowledge by tapping into applications such as electronic calendars, address books, and task lists. The phone alleviates cognitive load on users by various methods including detecting when the user is uninterruptible and automatically turning the ringer off. 1. Introduction Current commercial mobile phones impose additional cognitive load on their users by requiring them to be conscious of their phone’s states. Examples include remembering to turn the ringer on and off, handling missed calls, determining call priority, and worrying about inaudible ringer volume in a loud environment. This paper attempts to alleviate users of these inconveniences, creating a phone that can adapt to user’s context changes. SenSay (sensing & saying) is a context-aware mobile phone that modifies its behavior based on its user's state and surroundings. It adapts to dynamically changing environmental and physiological states and also provides the remote caller information on the current context of the phone user. To provide context information SenSay uses light, motion, and microphone sensors. The sensors are placed on various parts of the human body with a central hub, called the sensor box, mounted on the waist (see Figure 1). Figure 1. SenSay: sensor box mounted on the hip (left), the mobile phone (center), and voice and ambient microphones mounted on the user (right). SenSay introduces the following four states: Uninterruptible, Idle, Active, and the default state, Normal. A number of phone actions are associated with each state. For example, in the Uninterruptible state, the ringer is turned off. Some related work is reported in the following papers. In a much more limited context the idea of smart appliances and phones was explored in [1], [2], [4], and [5]. In [3] concepts of context-aware computing and wearable devices have been described. 2. SenSay Architecture 2.1 General Overview A closed architecture was adopted with five functional modules: the sensor box, sensor module, decision module, action module, and phone module. The following components are shown in Figure 3, from left to right: the sensor box collects physical sensor data, the software-based sensor module queries that data, the decision module determines the phone’s state, the action module sets that state, and the phone module provides access to the mobile phone operating system and user interface. In the current prototype, the decision, sensor, and action modules run on a notebook computer (henceforth called theFigure 2. SenSay architecture depicting the 5 modules.123457861. Voice & ambient microphones2. Debug circuits3. Voltage regulation4. PIC5. Oscillator circuits6. Serial interface7. Light sensor8. Accelerometers123457861. Voice & ambient microphones2. Debug circuits3. Voltage regulation4. PIC5. Oscillator circuits6. Serial interface7. Light sensor8. Accelerometers Figure 3. Layout of the sensors on the PCB. platform) running MS Windows 2000, which is connected to both the sensor box and the mobile phone via RS232 serial connections. In the next revision of SenSay, the mobile phone will include the sensor, decision, and action modules and communicate directly with the sensor box. 2.2 Sensor Box The sensor box includes a printed circuit board (PCB), as shown in Figure 3, housed in a plastic casing. The board circuitry consists of two subsystems, the sensors and the microcontroller. A PIC16F877 microcontroller is the heart of the sensor box. It provides eight 10-bit analog-digital conversion (ADC) channels as the interface to the sensors, as well as a port for serial communication to interface with the sensor module. The microcontroller is used to process the queries from the sensor module and return the requested sensor data as a 10-bit word. The specific sensors integrated in the sensor box are shown in Table 1. The two microphones are connected to the board through long (~1m) wires to allow mounting on various parts of the body. The light sensor is also connected through a long wire allowing it to be mounted on the phone. Three accelerometers are used to capture three (x, y, z) degrees of motion and are mounted directly on the PCB. Various support circuits are also included to condition the output signal of the sensors to insure the outputs remain in the desired range. 2.3 Sensor Module The sensor module is responsible for querying the sensor box periodically (currently once per second) and returning that data to the decision module. A simple communication protocol has been defined, supporting basic query/response commands. Communication between the microcontroller and the sensor module is established through the standard RS232 serial connection. A unique ID number identifies each Table 1. Specifications for sensors mounted in sensor box. Relevant SpecificationsHardware Componenthigh sensitivity to visible light, output varies linearly from 0 – 4 volts Visible LightPanasonic PNA4603H Photo IClinear output from base temperature +10mV/°C, with accuracy of +/0.5°C, range of +2°C to +50°C TemperatureNational LM35captures 3 axes of motion: outputs range from +/- 38 g, output changes by 50mV/g, basic op-amp circuit changing the reference point to 2.5V to scale output