Modeling the Brain s Operating System Dana H Ballard Computer Science Dept University of Austin Texas NY USA Embodied Cognition Brain Body World Maurice Merleau Ponty 1906 1961 Timescales 10 3 Memory encoding 10 sec 2 Activity time 10 1 Continuous Speed Chess minimum search Sentence generation 10 0 Attention Switching Time 10 1 Modal fixation time Shortest Recognition time 10 2 Round trip through Cortical Memory Discrete Visual Routines Roelfsema et al PNAS 2003 Multi tasking revealed by gaze sharing in human data Mid Block 40m 60 Stop sign Car Mid block sign Intersection sign Horizontal coordinate deg 50 Gaze tracking sign location 40 Eye 30 20 10 Gaze tracking car location 0 0 3 172 10 5 3 177 10 time sec 10 3 182 10 15 3 187 10 Shinoda Hayhoe Vision Research 2001 Human models are used to study multi tasking Pickup cans Stay on sidewalk Avoid obstacles Control of visuo motor routines active only 4 can run simultaneously 100 200ms update per behavior inactive Running Behaviors Eye Movement Trace QuickTime and a decompressor are needed to see this picture Three Levels of a Human Operating System Behaviors are scheduled from a pool 3 Behavior 1 Behaviors obtain sensory information 2 Behaviors compete for body s motor resources Microbehavior for Litter Cleanup d 1 Visual Routine 2a Policy 2b Value of Policy Q d Heading from Walter s perspective Learned Microbehaviors Litter Sidewalk Obstacles Which Microbehavior should get the gaze vector Before Observation Which Microbehavior should get the gaze vector After Observation obs side can obs can side Performance Comparison A human walks Walter s route Methods Human Ss walk Walter s route in Virtual Reality A curved path in real space Produces the perception of a straight path in visual space Their 6 dof head position and 2 dof gaze positions are continuously tracked Three subjects were used The resultant video and eye track signal are scored frame byframe Human Data Random Salient Locations Match No match Saliency Map vs Gaze courtesy of program provided by Dr Laurent Itti at the iLab USC Three trials Scheduling Context Walter Human Ss Approaching crosswalk Waiting for light On Crosswalk Walter and the human Ss all exhibit context sensitivities Human gaze locations are interpreted based on gaze location The actual internal state is unknown Human Experiments PBJ Basic Routines Routines locate bread remember loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread1 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 Modeling Tasks Subject A Subject B time Routine Segmentation locate bread remember loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread1 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 fixate loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread2 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 locate jar lid remember loc jarlid puthand right pickup right fixate 0 40 0 2 0 65 remember loc jarlid on table puthand right dropoff right locate pblid remember loc bottlelid puthand right pickup right fixate 0 3 0 2 0 65 r e m e m b e r loc bottlelid on table puthand right dropoff right locate khandle remember loc knife puthand right pickup right locate pbbottle remember loc bottle puthand right fixate 0 1 0 2 0 78 puthand right puthand right loc bottle fixate 0 1 0 2 0 78 puthand right locate jar remember loc jar puthand right fixate 0 1 0 2 0 78 puthand right fixate loc knife puthand right dropoff right locate jar fixate loc bread1 puthand right pickup right turnhand right 0 0 90 fixate loc bread2 puthand right dropoff right fixate loc bottlelid on table puthand right pickup right fixate loc bottlelid puthand right dropoff right fixate loc jarlid on table puthand right pickup right fixate loc jarlid puthand right dropoff right locate bread remember loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread1 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 fixate loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread2 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 locate jar lid remember loc jarlid puthand right pickup right fixate 0 40 0 2 0 65 remember loc jarlid on table puthand right dropoff right locate khandle remember loc knife puthand right pickup right locate jar remember loc jar puthand right fixate 0 1 0 2 0 78 puthand right locate pblid remember loc bottlelid puthand right pickup right fixate 0 3 0 2 0 65 r e m e m b e r loc bottlelid on table puthand right dropoff right locate pbbottle remember loc bottle puthand right fixate 0 1 0 2 0 78 puthand right puthand right loc bottle fixate 0 1 0 2 0 78 puthand right fixate loc bread1 puthand right pickup right turnhand right 0 0 90 fixate loc bread2 puthand right dropoff right fixate loc knife puthand right dropoff right locate jar fixate loc jarlid on table puthand right pickup right fixate loc jarlid puthand right dropoff right fixate loc bottlelid on table puthand right pickup right fixate loc bottlelid puthand right dropoff right Routine Segmentation locate bread remember loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread1 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 fixate loc loaf puthand right pickup right fixate 0 1 0 2 0 78 remember loc bread2 puthand right turnhand right 0 0 90 dropoff right turnhand right 0 0 0 locate jar lid remember loc jarlid puthand right pickup right fixate 0 40 0 2 0 65 remember loc jarlid on table puthand right dropoff right locate pblid remember loc bottlelid puthand right pickup right fixate 0 3 0 2 0 65 r e m e m b e r loc bottlelid on table puthand right dropoff right locate khandle remember loc knife puthand right pickup right locate pbbottle remember loc bottle puthand right fixate 0 1 0 2 0 78 puthand right puthand right loc bottle fixate 0 1 0 2 0 78 puthand right locate jar remember loc jar puthand right fixate 0 1 0 2 0 78 puthand right fixate loc knife puthand right dropoff right locate jar fixate loc bread1 puthand right pickup right turnhand right 0 0 90 fixate loc bread2 puthand right dropoff right fixate loc bottlelid on table puthand right pickup right fixate loc bottlelid puthand right dropoff right fixate loc jarlid on table puthand right pickup right fixate loc jarlid puthand right dropoff right locate bread remember loc loaf puthand right pickup right fixate 0 1 0 2 0
View Full Document
Unlocking...