Embodied Construction Grammar in language (acquisition and) useState of the ArtWhat does language do?Language understandingLanguage understanding: analysis & simulationInterpretation: x-schema simulationSlide 8Slide 9Construction GrammarForm-meaning mappings for languageSlide 12Embodied Construction Grammar (Bergen and Chang 2002)Representing image schemasInference and Conceptual SchemasEarly Example Understanding News StoriesTaskI/O as Feature StructuresSlide 19Slide 20Slide 21Formal Cognitive LinguisticsEmbodied constructionsSchema FormalismA Simple ExampleSource-Path-GoalTranslational MotionConstruction FormalismRepresenting constructions: TOThe INTO constructionGrammatical Construction ExampleThe DIRECTED-MOTION constructionSemantic specificationLanguage Understanding ProcessConstructional analysisSemantic SpecificationSlide 38Simulation-based sense disambiguationSimulation-based inferenceWorld knowledge informs simulationGetting From the Utterance to the SemSpec Johno BryantEmbodied Construction GrammarTraditional Parsing Methods Fall ShortOur AnalyzerOur Analyzer (cont’d)System ArchitectureThe LevelsRecognizersRecognizer ExampleSlide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 65Slide 66Resulting SemSpecChunkingConstruction RecognizersChunk ChartCombining Partial ParsesStructure MergingStructure Merging ExampleSemantic DensitySummary: ECGIssues in Scaling up to LanguageSlide 78The Buy schemaThe Sell schemaExtending Inferential CapabilitiesSemantic WebThe ICSI/Berkeley Neural Theory of Language ProjectProbabilistic Relation InferenceStatus of PRMsArticulating ProjectsConclusionEmbodied Construction Grammar in language (acquisition and) useJerome Feldman([email protected])Computer Science Division, University of California, Berkeley, andInternational Computer Science InstituteState of the Art•Limited Commercial Speech Applications transcription, simple response systems •Statistical NLP for Restricted Tasks tagging, parsing, information retrieval•Template-based Understanding programs expensive, brittle, inflexible, unnatural•Essentially no NLU in HCI, QA SystemsWhat does language do?“Harry walked to the cafe.”“Harry walked into the cafe.”A sentence can evoke an imagined scene and resulting inferences:CAFE CAFE–Goal of action = at cafe–Source = away from cafe–cafe = point-like location–Goal of action = inside cafe–Source = outside cafe–cafe = containing locationLanguage understandingInterpretation(Utterance, Situation)Linguistic knowledgeConceptual knowledgeAnalysisLanguage understanding: analysis & simulation“Harry walked to the cafe.”Schema Trajector Goalwalk Harry cafeCafeLexiconConstructiconGeneral KnowledgeBelief StateAnalysis ProcessSemanticSpecificationUtteranceSimulationInterpretation: x-schema simulationConstructions can•specify which schemas and entities are involved in an event, and how they are related•profile particular stages of an event•set parameters of an eventenergywalker at goalwalker=Harrygoal=homeHarry is walking home.Phonetics SemanticsPragmatics Morphology SyntaxTraditional Levels of AnalysisPhonetics SemanticsPragmatics Morphology Syntax“Harry walked into the cafe.”UtteranceConstruction GrammartoblockwalkForm MeaningA construction is a form-meaning pair whose properties may not be strictly predictable from other constructions.(Construction Grammar, Goldberg 1995)SourcePathGoalTrajectorForm-meaning mappings for languageFormphonological cuesword orderintonationinflectionMeaningevent structuresensorimotor controlattention/perspectivesocial goals...Linguistic knowledge consists of form-meaning mappings:CafeConstructions as maps between relationsMover + Motion + Directionbefore(Motion, Direction)before(Mover, Motion)“is” + Action + “ing”before(“is”, Action)suffix(Action, “ing”)Mover + Motionbefore(Mover, Motion)Form MeaningProgressiveActionaspect(Action, ongoing)MotionEventmover(Motion, Mover)DirectedMotionEventdirection(Motion, Direction)mover(Motion, Mover)Complex constructions are mappings between relations in form and relations in meaning.Embodied Construction Grammar(Bergen and Chang 2002)•Embodied representations–active perceptual and motor schemas–situational and discourse context•Construction Grammar–Linguistic units relate form and meaning/function.–Both constituency and (lexical) dependencies allowed.•Constraint-based (Unification)–based on feature structures (as in HPSG)–Diverse factors can flexibly interact.schema ContainerrolesinteriorexteriorportalboundaryRepresenting image schemasInteriorExteriorBoundaryPortalSourcePathGoalTrajectorThese are abstractions over sensorimotor experiences.schema Source-Path-Goalrolessourcepathgoaltrajectorschema namerole nameInference and Conceptual Schemas•Hypothesis: –Linguistic input is converted into a mental simulation based on bodily-grounded structures.•Components:–Semantic schemas •image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations–Linguistic units •lexical and phrasal construction representations invoke schemas, in part through metaphor•Inference links these structures and provides parameters for a simulation engineEarly ExampleUnderstanding News StoriesFrance fell into recession. Pulled out by GermanyIn1991, India set out on a path of liberalization. The Government started to loosen its stranglehold on business and removed obstacles to international trade. Now the Government is stumbling in implementing the liberalization plan.Task•Interpret simple discourse fragments/blurbs–France fell into recession. Pulled out by Germany–Economy moving at the pace of a Clinton jog.–US Economy on the verge of falling back into recession after moving forward on an anemic recovery.–Indian Government stumbling in implementing Liberalization plan.–Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice.–The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.I/O as Feature Structures•Indian Government stumbling in implementing liberalization planLanguage understanding: analysis & simulation“Harry walked into the cafe.”Analysis ProcessSemanticSpecificationUtteranceConstructionsGeneral KnowledgeBelief