1Platform-Based Design of Augmented Cognition SystemsLatosha Marshall & Colby RaleyENSE623 Fall 2004Design & implementation of Augmented Cognition systems:Modular design can make it possible ◊ Platform-based design makes it feasible11/23/2004 2Augmented Cognition Augmented Cognition systems use real-time cognitive state data to adapt systems to a user rather than forcing the user to adapt to a system Exploit recent technological advancements Neuroscience: sensor design, signal interpretation Signal processing: speed and accuracy “21stCentury Human Computer Interaction”211/23/2004 3Augmented Cognition EnablersNeuroscienceMath:Signal ProcessingArtifact DetectionEngineering:Mechanical ElectricalSystemsCognitive SystemsPsychology:Human FactorsNeuroergonomicsDisciplineSensors: development & placementGauges:DevelopmentEnsuring meaningful informationSystem Design:Physical ComponentsComponent CommunicationInformation Flow“Bringing the Human in the Loop”Operator Environment:Interface designInterface designSystem Aspect11/23/2004 4userenvironmenttasksensorsensorsensorcognitive stateenvironmental statetask statecognitive modelenvironmental modeltask modelAugmentation Managerstate + task + environment = mitigation strategy; apply mitigation strategyautonomousagentscommandinterfaceSystem Architecture311/23/2004 5System Architecture - Inputs User Any person interacting with an augmented cognition system In a driving environment: the driver In a learning environment: the student Environment The environment in which the system is being implemented In a driving environment: the car and its current surroundings In a learning environment: the classroom and any equipment being used Task The task that the user is completing; using the augmented cognition system to improve performance In a driving environment: driving (or navigating to an objective) In a learning environment: concept mastery11/23/2004 6System Architecture - Models Cognitive Model User Model Computational model for how people perform tasks and solve problems, based on psychological principles Enable the prediction of the time it takes for people to perform tasks, the kinds of errors they make, the decisions they make, etc… Context Models Environmental Model Incorporates known information about the task environment enable a context-aware environment Task Model Incorporates known information about the task/objective to understand and predict the task that a user is completing Necessary for accurate timing of mitigation strategy executionComparison of Context Classification Systems411/23/2004 7Platform-Based Design Platform-based design goes beyond modular design to incorporate information about the application environment into the design process. Platform-based design combines top-down and bottom-up design approaches Top Down Platform-mandated constraints Connections and communications between components Consideration of system-level goals Bottom Up Component-mandated constraints Benefits of platform-based design include Reuse of designed components Reduced design cycle time Component “swapping” during design processAlberto Sangiovanni Vincentelli. Defining Platform-based Design. EEDesign of EETimes, February 2002.11/23/2004 8Platforms of Interest Cockpit Driving Airplane Control Station Unmanned Vehicle Interface Air Traffic Control Command Post of the Future Learning Environment Virtual Reality Classroom511/23/2004 9Component Catalog Sensors Cognitive Direct Brain Measures EEG fNIR Psychophysiological Measures HR, EKG Pulse Ox Posture GSR Temperature EOG Pupilometry Gaze Tracking Environmental Platform Measures Location Internal Conditions Fuel Weapons External Measures Weather Presence of Chemical or Biological Agents Situational Awareness Hostility Obstacles Task Status Interfaces Visual Heads up display Traditional display Alert Warning Picture Text Auditory Voice Warning Spatially locatable Tactile Warning Directional cue11/23/2004 10Component Catalog - Driving Sensors Cognitive Direct Brain Measures EEG fNIR Psychophysiological Measures HR, EKG Pulse Ox Posture GSR Temperature EOG Pupilometry Gaze Tracking Environmental Platform Measures Location Internal Conditions Fuel Weapons External Measures Weather Presence of Chemical or Biological Agents Situational Awareness Hostility Obstacles Task Status Interfaces Visual Heads up display Traditional display Alert Warning Picture Text Auditory Voice Warning Spatially locatable Tactile Warning Directional cueDriving611/23/2004 11System Architecture – Driving Components11/23/2004 12drivercar & roadtask / objectiveCognitive Sensors: EEG, fNIRVehicle sensors: gas gauge, speedometer, odometerDistance, task completion, # tasks in progresscognitive stateenvironmental statetask statecognitive model of drivingenvironmental model of car and roadtask model: driving and secondary tasksAugmentation Managerstate + task + environment = mitigation strategy; apply mitigation strategy*Interfaces: audio system, cell phone system, visual alert system, automatic braking/ steering/ between-car-distanceSystem Architecture - DrivingPhysiological Sensors: EOG, EKG, HR, pupilometry, gaze tracking, PO, posture, body temp, GSRContext sensors: lane departure, obstacle detection (IR, visual), temperature, heading, wind speed, locationMitigation Strategies - Driving• Modality Switching• Task Ordering• Attention Directing711/23/2004 13Driving: Constraint-Based Requirements System must be compatible with automobile standard functions System shall not inhibit driver’s vision of the road and/or surroundings No system equipment/procedure shall require driver to migrate attention behind self No system equipment/procedure shall require the driver to move beyond driver’s seat No system equipment/procedure will require the driver to have both hands off of the steering wheel11/23/2004 14Driving: System Interfaces & CommunicationsModule Platform instance Input OutputLevel 1UserDriver InterfacePhysiological conditions:Pulse, temperature, gaze location, heart rate, moisture content,