The ICSI/Berkeley Neural Theory of Language ProjectMoving from Spatial Relations to VerbsCoordination of Pattern GeneratorsCoordinationPreshaping While Reaching to GraspPowerPoint PresentationInternal Model and Efference CopySlide 10Many areas code for motion parametersSlide 12Slide 13Slide 14Slide 15F5 Mirror NeuronsSlide 17Slide 19Slide 20Summary of Fronto-Parietal CircuitsSlide 22Slide 23FARS (Fagg-Arbib-Rizzolatti-Sakata) ModelSlide 26Modeling Motor SchemasAn Active Model of EventsModel Review: Stochastic Petri NetsSlide 30Slide 31Active representationsSlide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 71Representing concepts using triangle nodesSlide 73Simulation hypothesisSimulation SemanticsSimulation-based language understandingSimulation specificationLanguage Development in ChildrenSlide 79Regier Model LimitationsLearning Verb Meanings David BaileySlide 82Reasoning about Actions in Artificial Intelligence (AI)Grasping: the actionActionsProblems with action conceptsThe Frame ProblemFrame axioms are needed in logicActive Representations don’t need frame axiomsRamification ProblemThe ICSI/Berkeley Neural Theory of Language ProjectLearning early constructions (Chang, Mok)ECGMoving from Spatial Relations to Verbs•Open class vs. closed class–How do we represent verbs (say of hand motion)•Can we build models of verbs based on motor control primitives?•If so, how can models overcome central limitations of Regier’s system?–Inference –Abstract usesCoordination of Pattern GeneratorsCoordination•PATTERN GENERATORS, separate neural networks that control each limb, can interact in different ways to produce various gaits. –In ambling (top) the animal must move the fore and hind leg of one flank in parallel.–Trotting (middle) requires movement of diagonal limbs (front right and back left, or front left and back right) in unison. –Galloping (bottom) involves the forelegs, and then the hind legs, acting togetherPreshaping While Reaching to GraspInternal Model and Efference CopyMany areas code for motion parametersMultiple, chronically implanted, intracranial microelectrode arrays would be used to sample theactivity of large populations of single cortical neurons simultaneously. The combined activity ofthese neural ensembles would then be transformed by a mathematical algorithm into continuousthree-dimensional arm-trajectory signals that would be used to control the movements of arobotic prosthetic arm. A closed control loop would be established by providing the subject withboth visual and tactile feedback signals generated by movement of the robotic arm.Rizzolatti et al. 1998A New PictureA New PictureThe fronto-parietal networksRizzolatti et al. 1998F5 Mirror NeuronsF5 Mirror NeuronsGallese and Goldman, TICS 1998Category Loosening in Mirror Neurons (~60%)(Gallese et al. Brain 1996)Observed: A is Precision GripB is Whole Hand Prehension Action: C: precision gripD: Whole Hand PrehensionUmiltà et al. Neuron 2001A (Full vision)A (Full vision)B (Hidden)B (Hidden)C (Mimicking)C (Mimicking)D (HiddenMimicking)D (HiddenMimicking)F5 Audio-Visual Mirror NeuronsF5 Audio-Visual Mirror NeuronsKohler et al. Science (2002)Summary of Fronto-Parietal CircuitsMotor-Premotor/Parietal Circuits PMv (F5ab) – AIP Circuit“grasp” neurons – fire in relation to movements of hand prehension necessary to grasp objectF4 (PMC) (behind arcuate) – VIP Circuit transforming peri-personal space coordinates so can move toward objects PMv (F5c) – PF Circuit F5c different mirror circuits for grasping, placing or manipulating object Together suggest cognitive representation of the grasp, active in action imitation and action recognitionEvidence in Humans for Mirror, General Purpose, and Action-Location NeuronsMirror: Fadiga et al. 1995; Grafton et al. 1996;Rizzolatti et al. 1996; Cochin et al. 1998;Decety et al. 1997; Decety and Grèzes 1999;Hari et al. 1999; Iacoboni et al. 1999;Buccino et al. 2001.General Purpose: Perani et al. 1995; Martin et al.1996; Grafton et al. 1996; Chao and Martin 2000. Action-Location: Bremmer, et al., 2001.Itti: CS564 - Brain Theory and Artificial Intelligence. FARS ModelFARS (Fagg-Arbib-Rizzolatti-Sakata) ModelAIPF5dorsal/ventral streamsTask Constraints (F6)Working Memory (46)Instruction Stimuli (F2)Task Constraints (F6)Working Memory (46?)Instruction Stimuli (F2)AIPDorsalStream:AffordancesITVentralStream:RecognitionWays to grab this “thing”“It’s a mug”PFCAIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.MULTI-MODAL INTEGRATIONThe premotor and parietal areas, rather than havingseparate and independent functions, are neurally integratednot only to control action, but also to serve the function ofconstructing an integrated representation of:(a) Actions, together with (b) objects acted on, and (c) locations toward which actions are directed. In these circuits sensory inputs are transformed in order toaccomplish not only motor but also cognitive tasks, such asspace perception and action understanding.Modeling Motor Schemas•Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti)–Should model coordinated, distributed, parameterized control programs required for motor action and perception.–Should be an active structure.–Should be able to model concurrent actions and interrupts.–Should model hierarchical control (higher level motor centers to muscle extensor/flexors.•Computational model called x-schemas (http://www.icsi.berkeley.edu/NTL)An Active Model of Events•At the Computational level, actions and events are coded in active representations called x-schemas which are extensions to Stochastic Petri nets.•x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.Model Review: Stochastic Petri Nets312Basic Mechanism[1]Precondition arcResource arcInhibition arc[1]Firing function -- conjunctive -- logistic -- exponential family312Firing SemanticsModel Review11112Result of FiringModel ReviewActive representations•Many inferences about actions derive from what we know about executing them•Representation based on stochastic Petri nets