Language ComprehensionSpeech PerceptionMeaning RepresentationSpeech Comprehension and ProductionSpeech PerceptionSpectrogram: I owe you a yo-yoSpeech perception: two problemsHow Do Listeners Deal with Variability in Acoustic Input?Phonemic restorationMcGurk EffectPerception of auditory event affected by visual processingMcGurk EffectCategorical PerceptionExamplesIdentification: Discontinuity at BoundaryPairwise discriminationPairwise Discrimination(same/different)What Happened?Categorical Perception depends on languageNon-English ContrastsModels of Spoken Word IdentificationOnline word recognition: the cohort modelRecognizing Spoken Words: The Cohort ModelEvidence for activation of spurious wordsTRACE modelTRACE ModelHuman Eye Tracking DataHuman Eye Tracking DataRepresenting MeaningRepresenting MeaningCategory Specific Semantic DeficitsSummary of patient dataRepresenting MeaningImplicationsSensory-Functional ApproachA neural network model of category-specific impairmentsSimulating the Effects of Brain Damage by “lesioning” the modelLanguage ComprehensionSpeech PerceptionMeaning RepresentationSpeech Comprehension and ProductionSpeech Perception• The first step in comprehending spoken language is to identify the words being spoken, performed in multiple stages:1. Phonemes are detected (/b/, /e/, /t/, /e/, /r/, )2. Phonemes are combined into syllables (/be/ /ter/)3. Syllables are combined into words (“better”)4. Word meaning retrieved from memorySpectrogram: I owe you a yo-yoSpeech perception: two problems• Words are not neatly segmented (e.g., by pauses) • Lack of phoneme invariance• Coarticulation = consecutive speech sounds blend into each other due to mechanical constraints on articulators• Speaker differences; pitch affected by age and sex; different dialects, talking speeds etc.How Do Listeners Deal with Variability in Acoustic Input?• Use of visual cues:– McGurk effect• Use of semantic cues:– Phonemic restoration• Categorical perception: continuous changes in input are mapped on to discrete perceptsPhonemic restorationAuditory presentation PerceptionLegislature legislatureLegi_lature legi latureLegi*lature legislatureIt was found that the *eel was on the axle. wheel It was found that the *eel was on the shoe. heel It was found that the *eel was on the orange. peel It was found that the *eel was on the table. meal Warren, R. M. (1970). Perceptual restorations of missing speech sounds. Science, 167, 392-393.McGurk EffectPerception of auditory event affected by visual processingDemo 1AVI: http://psiexp.ss.uci.edu/research/teachingP140C/demos/McGurk_large.aviMOV: http://psiexp.ss.uci.edu/research/teachingP140C/demos/McGurk_large.movDemo 2MOV: http://psiexp.ss.uci.edu/research/teachingP140C/demos/McGurk3DFace.movHarry McGurk and John MacDonald in "Hearing lips and seeing voices", Nature 264, 746-748 (1976).McGurk Effect• McGurk effect in video: – lip movements = “ga”– speech sound = “ba”– speech perception = “da” (for 98% of adults)• Demonstrates parallel & interactive processing: speech perception is based on multiple sources of information, e.g. lip movements, auditory information.• Brain makes reasonable assumption that both sources are informative and “fuses” the information.Categorical Perception• Categorical perception: high level cognitive processes (i.e., categorization) can influence perceptual processesCategorizationcategorical perceptionPerception of Sounds/ImagesDifferences among items that fall into different categories are exaggerated, and differences among items that fall into the same category are minimized. (from Rob Goldstone, Indiana University)Examples• from “LAKE” to “RAKE”– http://www.psych.ufl.edu/~white/Cate_per.htm• from /da/ to /ga/Good /da/ Good /ga/1 2 3 4 5 6 7 8Identification: Discontinuity at Boundary100%% of /ga/ response50%1 2 3 4 5 6 7 80%TokenPairwise discriminationGood /da/ Good /ga/1 2 3 4 5 6 7 8Discriminate these pairsDiscriminate these pairsDiscriminate these pairs(straddle the category boundary)Pairwise Discrimination(same/different)% Correct Discrimination01020304050607080901001_2 2_3 3_4 4_5 5_6 6_7 7_8Pair of stimuliWhat Happened?1 2 3 4 5 6 7 8Physical WorldPerceptual Representation1 2 3 4 5 6 7 8Categorical Perception depends on language• In one language a difference in sound may make a difference between words; in another, it might not•Example– The Japanese language does not distinguish between /l/ and /r/– These sounds belong to the same category for Japanese listeners– They find it very hard to discriminate between them (Massaro, 1994)Non-English ContrastsSalish (Native North American—Canadian—language)HindiDental Stop Retroflex StopUvularVelarModels of Spoken Word Identification•The Cohort Model– Marslen-Wilson & Welsh, 1978– Revised, Marslen-Wilson, 1989•TheTRACE Model– Similar to the Interactive Activation model – McClelland & Elman, 1986Online word recognition: the cohort modelRecognizing Spoken Words: The Cohort Model• All candidates considered in parallel• Candidates eliminated as more evidence becomes available in the speech input• Uniqueness point occurs when only one candidate remainsEvidence for activation of spurious words• If we recognize words by recognizing a cohort of possibilities, then the cohort words should excert some influence• Shillcock (1990). Test for semantic priming of cohort words:“He picked up the trombone”“trom” “bone”Semantic priming for the word “rib”TRACE model• Similar to interactive activation model but applied to speech recognition• Connections between levels are bi-directional and excitatory Æ top-down effects• Connections within levels are inhibitory producing competition between alternatives (McClelland & Elman, 1986)TRACE ModelFeatures over timebeakeri s Rb i s Rbspeaker(McClelland & Elman, 1986)Human Eye Tracking Data‘Pick up the beaker’Eye tracking device to measurewhere subjects are lookingAllopenna, Magnuson & Tanenhaus (1998)Human Eye Tracking DataHuman eye tracking datahighly similar to TRACE predictions Allopenna, Magnuson & Tanenhaus (1998)Representing MeaningRepresenting Meaning• Mental representation of meaning as a