Modeling the Brain’s OperatingSystemDana H. BallardComputer ScienceDept.University of AustinTexas, NY, USAEmbodied CognitionMaurice Merleau-Ponty 1906-1961WorldBodyBrainTimescales10 -210 -1 10 110 210 0 Round-trip through Cortical MemoryShortest Recognition timeModal fixation timeAttention Switching TimeSentence generationSpeed Chess minimum searchActivity time10 3Memory encodingsecContinuousDiscreteRoelfsema et alPNAS 2003Visual RoutinesMulti-taskingrevealed by gaze sharing in humandata0102030405060172 103177 103182 103187 103Mid-Block 40mHorizontal coordinate (deg)time(sec)Mid-block signCarIntersection signEyeStop sign0 5 10 15Shinoda & Hayhoe, Vision Research 2001Gaze tracking sign locationGaze tracking car locationHuman models are used to study multi-taskingPickup cansStay on sidewalkAvoid obstaclesControl of visuo-motor routines“active”“inactive”+ only ~4 can run simultaneously+ 100-200ms update per behaviorQuickTime™ and a decompressorare needed to see this picture.QuickTime™ and a decompressorare needed to see this picture.Running Behaviors: Eye Movement TraceBehaviors competefor body’s motorresourcesBehaviors obtainsensory informationBehaviors arescheduled from a poolThree Levels of a Human “Operating System”Behavior123Microbehaviorfor Litter Cleanup2b. Value of PolicyQθ,ddθ2a. Policy1. Visual RoutineHeading from Walter’s perspectiveLearned MicrobehaviorsLitter Sidewalk ObstaclesWhich Microbehavior should get the gaze vector?Before ObservationWhich Microbehavior should get the gaze vector?After ObservationobscansideobscansidePerformance ComparisonA human walks Walter’s routeA curvedpath in realspaceProducestheperceptionof astraightpath invisualspace Human Ss walkWalter’s route in VirtualReality.Their 6 dof headposition and 2 dof gazepositions arecontinuously tracked.Three subjects wereused. The resultant video andeye track signal arescored frame-by-frame.MethodsHuman DataRandom Salient LocationsSaliency Map vs Gaze courtesy of programprovided by Dr. Laurent Ittiat the iLab, USCMatch No matchThree trialsOn CrosswalkApproachingcrosswalkWalter and the human Ss all exhibit context sensitivities. Humangaze locations are interpreted based on gaze location. The actualinternal state is unknown.Waiting for lightWalterHuman SsScheduling ContextHuman ExperimentsPBJ: Basic RoutinesRoutineslocate breadremember loc_loafputhand rightpickup rightfixate -0.1 -0.2 0.78remember loc_bread1puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0Modeling TasksSubject A Subject BtimeRoutine Segmentationlocate breadremember loc_loafputhand rightpickup rightfixate -0.1 -0.2 0.78remember loc_bread1puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0fixate loc_loafputhand rightpickup rightfixate 0.1 -0.2 0.78remember loc_bread2puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0locate jar-lidremember loc_jarlidputhand rightpickup rightfixate -0.40 -0.2 0.65remember loc_jarlid_on_tableputhand rightdropoff rightlocate pblidremember loc_bottlelidputhand rightpickup rightfixate -0.3 -0.2 0.65r e m e m b e rloc_bottlelid_on_tableputhand rightdropoff rightlocate khandleremember loc_knifeputhand rightpickup rightlocate pbbottleremember loc_bottleputhand rightfixate -0.1 -0.2 0.78puthand rightputhand right loc_bottlefixate -0.1 -0.2 0.78puthand rightlocate jarremember loc_jarputhand rightfixate 0.1 -0.2 0.78puthand rightfixate loc_knifeputhand rightdropoff rightlocate jarfixate loc_bread1puthand rightpickup rightturnhand right 0 0 -90fixate loc_bread2puthand rightdropoff rightfixate loc_bottlelid_on_tableputhand rightpickup rightfixate loc_bottlelidputhand rightdropoff rightfixate loc_jarlid_on_tableputhand rightpickup rightfixate loc_jarlidputhand rightdropoff rightlocate breadremember loc_loafputhand rightpickup rightfixate -0.1 -0.2 0.78remember loc_bread1puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0fixate loc_loafputhand rightpickup rightfixate 0.1 -0.2 0.78remember loc_bread2puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0locate jar-lidremember loc_jarlidputhand rightpickup rightfixate -0.40 -0.2 0.65remember loc_jarlid_on_tableputhand rightdropoff rightlocate khandleremember loc_knifeputhand rightpickup rightlocate jarremember loc_jarputhand rightfixate 0.1 -0.2 0.78puthand rightlocate pblidremember loc_bottlelidputhand rightpickup rightfixate -0.3 -0.2 0.65r e m e m b e rloc_bottlelid_on_tableputhand rightdropoff rightlocate pbbottleremember loc_bottleputhand rightfixate -0.1 -0.2 0.78puthand rightputhand right loc_bottlefixate -0.1 -0.2 0.78puthand rightfixate loc_bread1puthand rightpickup rightturnhand right 0 0 -90fixate loc_bread2puthand rightdropoff rightfixate loc_knifeputhand rightdropoff rightlocate jarfixate loc_jarlid_on_tableputhand rightpickup rightfixate loc_jarlidputhand rightdropoff rightfixate loc_bottlelid_on_tableputhand rightpickup rightfixate loc_bottlelidputhand rightdropoff rightRoutine Segmentationlocate breadremember loc_loafputhand rightpickup rightfixate -0.1 -0.2 0.78remember loc_bread1puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0fixate loc_loafputhand rightpickup rightfixate 0.1 -0.2 0.78remember loc_bread2puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0locate jar-lidremember loc_jarlidputhand rightpickup rightfixate -0.40 -0.2 0.65remember loc_jarlid_on_tableputhand rightdropoff rightlocate pblidremember loc_bottlelidputhand rightpickup rightfixate -0.3 -0.2 0.65r e m e m b e rloc_bottlelid_on_tableputhand rightdropoff rightlocate khandleremember loc_knifeputhand rightpickup rightlocate pbbottleremember loc_bottleputhand rightfixate -0.1 -0.2 0.78puthand rightputhand right loc_bottlefixate -0.1 -0.2 0.78puthand rightlocate jarremember loc_jarputhand rightfixate 0.1 -0.2 0.78puthand rightfixate loc_knifeputhand rightdropoff rightlocate jarfixate loc_bread1puthand rightpickup rightturnhand right 0 0 -90fixate loc_bread2puthand rightdropoff rightfixate loc_bottlelid_on_tableputhand rightpickup rightfixate loc_bottlelidputhand rightdropoff rightfixate loc_jarlid_on_tableputhand rightpickup rightfixate loc_jarlidputhand rightdropoff rightlocate breadremember loc_loafputhand rightpickup rightfixate -0.1 -0.2 0.78remember loc_bread1puthand rightturnhand right 0 0 90dropoff rightturnhand right 0 0 0fixate loc_loafputhand
View Full Document
Unlocking...