Slide 1Slide 2Slide 4Slide 5Slide 6Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 71Slide 72Slide 73Slide 74Slide 75Slide 76Slide 77Slide 78Slide 79Slide 80Slide 81Slide 82Slide 83Slide 84Slide 85Slide 87Slide 88Slide 89Slide 90Slide 94The ICSI/Berkeley Neural Theory of Language ProjectLearning early constructions (Chang, Mok)ECGMoving from Spatial Relations to Verbs•Open class vs. closed class–How do we represent verbs (say of hand motion)•Can we build models of verbs based on motor control primitives?•If so, how can models overcome central limitations of Regier’s system?–Inference –Abstract usesCoordination of Pattern GeneratorsCoordination•PATTERN GENERATORS, separate neural networks that control each limb, can interact in different ways to produce various gaits. –In ambling (top) the animal must move the fore and hind leg of one flank in parallel.–Trotting (middle) requires movement of diagonal limbs (front right and back left, or front left and back right) in unison. –Galloping (bottom) involves the forelegs, and then the hind legs, acting togetherPreshaping While Reaching to GraspInternal Model and Efference CopyMany areas code for motion parametersMultiple, chronically implanted, intracranial microelectrode arrays would be used to sample theactivity of large populations of single cortical neurons simultaneously. The combined activity ofthese neural ensembles would then be transformed by a mathematical algorithm into continuousthree-dimensional arm-trajectory signals that would be used to control the movements of arobotic prosthetic arm. A closed control loop would be established by providing the subject withboth visual and tactile feedback signals generated by movement of the robotic arm.Rizzolatti et al. 1998A New PictureA New PictureThe fronto-parietal networksRizzolatti et al. 1998F5 Mirror NeuronsF5 Mirror NeuronsGallese and Goldman, TICS 1998Category Loosening in Mirror Neurons (~60%)(Gallese et al. Brain 1996)Observed: A is Precision GripB is Whole Hand Prehension Action: C: precision gripD: Whole Hand PrehensionUmiltà et al. Neuron 2001A (Full vision)A (Full vision)B (Hidden)B (Hidden)C (Mimicking)C (Mimicking)D (HiddenMimicking)D (HiddenMimicking)F5 Audio-Visual Mirror NeuronsF5 Audio-Visual Mirror NeuronsKohler et al. Science (2002)Summary of Fronto-Parietal CircuitsMotor-Premotor/Parietal Circuits PMv (F5ab) – AIP Circuit“grasp” neurons – fire in relation to movements of hand prehension necessary to grasp objectF4 (PMC) (behind arcuate) – VIP Circuit transforming peri-personal space coordinates so can move toward objects PMv (F5c) – PF Circuit F5c different mirror circuits for grasping, placing or manipulating object Together suggest cognitive representation of the grasp, active in action imitation and action recognitionEvidence in Humans for Mirror, General Purpose, and Action-Location NeuronsMirror: Fadiga et al. 1995; Grafton et al. 1996;Rizzolatti et al. 1996; Cochin et al. 1998;Decety et al. 1997; Decety and Grèzes 1999;Hari et al. 1999; Iacoboni et al. 1999;Buccino et al. 2001.General Purpose: Perani et al. 1995; Martin et al.1996; Grafton et al. 1996; Chao and Martin 2000. Action-Location: Bremmer, et al., 2001.Itti: CS564 - Brain Theory and Artificial Intelligence. FARS ModelFARS (Fagg-Arbib-Rizzolatti-Sakata) ModelAIPF5dorsal/ventral streamsTask Constraints (F6)Working Memory (46)Instruction Stimuli (F2)Task Constraints (F6)Working Memory (46?)Instruction Stimuli (F2)AIPDorsalStream:AffordancesITVentralStream:RecognitionWays to grab this “thing”“It’s a mug”PFCAIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.MULTI-MODAL INTEGRATIONThe premotor and parietal areas, rather than havingseparate and independent functions, are neurally integratednot only to control action, but also to serve the function ofconstructing an integrated representation of:(a) Actions, together with (b) objects acted on, and (c) locations toward which actions are directed. In these circuits sensory inputs are transformed in order toaccomplish not only motor but also cognitive tasks, such asspace perception and action understanding.Modeling Motor Schemas•Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti)–Should model coordinated, distributed, parameterized control programs required for motor action and perception.–Should be an active structure.–Should be able to model concurrent actions and interrupts.–Should model hierarchical control (higher level motor centers to muscle extensor/flexors.•Computational model called x-schemas (http://www.icsi.berkeley.edu/NTL)An Active Model of Events•At the Computational level, actions and events are coded in active representations called x-schemas which are extensions to Stochastic Petri nets.•x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.Model Review: Stochastic Petri Nets312Basic Mechanism[1]Precondition arcResource arcInhibition arc[1]Firing function -- conjunctive -- logistic -- exponential family312Firing SemanticsModel Review11112Result of FiringModel ReviewActive representations•Many inferences about actions derive from what we know about executing them•Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions•Generative model: action, recognition, planning , languageWalking:bound to a specific walker with a direction or goalconsumes resources (e.g., energy)may have termination condition(e.g., walker at goal) ongoing, iterative actionwalker=Harrygoal=homeenergywalker at goalPreshaping While Reaching to GraspThe ICSI/Berkeley Neural Theory of Language ProjectLearning early constructions (Chang, Mok)ECGRepresenting concepts using triangle nodestriangle nodes:when two of the neurons fire, the third also firesGoods ~ thing Dir. ~ ANY Cost ~money emp~Gra Speed ~slow Seller ~person Taste ~sweet sid~002 Schema ~walk Buyer ~person Color