James DiCarloThe McGovern Institute for Brain ResearchDepartment of Brain and Cognitive SciencesMassachusetts Institute of Technology, Cambridge MAFrom feedforward vision to naturalvision:The impact of free viewing and clutter on monkeyinferior temporal object representations• Position • Size • Pose • Illumination • “Clutter” • Background scene • Other objects One needs an image representation that isselective for object identity, yet tolerant to suchtransformations. across this wide range of conditions? The core problem of object recognition How does the brain recognize each objectWe have some idea of where we can find such an image representation (IT). We can study it at themost appropriate levelof abstraction (neuronalspikes). Rhesus monkey modelDecision Memory Implicit Explicit Monkey visual system and actiontime Object identity or category isdirectly* available in the regardless of (e.g.) objectposition and scale. Isolated, single objects.Passive viewing. AIT contains a rapidly evoked, explicit object representation 100 ms 100 ms population response, Hung, Kreiman, Poggio and DiCarlo Science (2005) Gross et al. (1972), Perret et al. (1982), Desimone et al.. (1984), Tovee et al. (1984), Schwartz et al (1985), Ito et al. (1995), Logethetis et al. (1996), Op de Beeck and Vogles (2000), DiCarlo and Maunsell (2000), etc.Mechanisms ? Feedforward* representation (The Core) The Core is powerful. The Core is not yet understood. Role in “natural vision” ? The Core is fast. * First evoked pattern of IT activity when an image is presented to the eye (Is it generalizable?)The Core and “natural vision” What is “natural vision” ? “You know it when you see it.”The Core and “natural vision”The Core and “natural vision” of core vision? ) / Scene / Context: objectsappear among other objects andon backgrounds (e.g. feature andspatial attention, motorpreparation to act, arousal) 3) Goal directedHow does “natural vision” challenge the basic model 1) Eye movements ( “free viewing” 2) ClutterObject Response 20 degA B C Object Response “A” “B” D Object identification task “C” “D”DiscriminableDetectable Detectablefixation pointExample IT neuron(normalized) controlled free 0 1.0 1.8 n = 63 best target worst target IT Population summary Population average response DiCarlo and Maunsell, Nature Neuroscience, 3: 814-821 (2000)IT responses are nearly identical incontrolled and free viewing conditions DiCarlo and Maunsell, Nature Neuroscience, 3: 814-821 (2000) DiCarlo and Maunsell, J Neurophysiology (2005)The Core and “natural vision”How does “natural vision” challenge the basic modelof core vision?1) Eye movements ( “free viewing” )2) Clutter / Scene / Context: objectsappear among other objects andon backgrounds3) Goal directed (e.g. feature andspatial attention, motorpreparation to act, arousal)Not much to worryabout here.Sheinberg and Logothetis, 2001DiCarlo and Maunsell, 2000Natural vision: Clutter, scene, and context In the real world… In the lab…IT Receptive Field Natural vision: Clutter, scene, and contextLong term goal: Understand IT in clutter IT responses to object are typically reduced when additional objects are presented (Sato, 1989; Miller et al., 1993; Rolls and Tovee, 1995; Chelazzi et al., 1998; Missal et al., 1999) Single objects Pair of objects Preferred object Poor object RF IT responseFirst open questions … Object pairs: IT response: ? ? ? ? • Any systematic relationship between: – response to an object pair – responses to the constituent objects?overviewExperimental design • Davide Zoccolan and David Cox • Recorded IT neuronal responses to thepresentation of: – Single objects – Pairs of objects – Triplets of objects – In three monkeys – Using two complementary experimentsExperiment 1Experiment 2EXPERIMENT 1 EXPERIMENT 2 Stimulus conditions Single objects Object pairs 2 degEXPERIMENT 1 EXPERIMENT 2 Stimulus conditions Single objects Object triplets 2 degFixate • Stimuli presented at 5 per sec • Passive viewing Core response: Rapid visual presentation 300 ms 100 ms 100 ms 100 ms 100 ms 100 ms 104 neurons recorded in three monkeysspikes/s Example IT neuronSum of responses to single objects(spikes/s) Response to object pairs(spikes/s) Example IT neuronResponse to object pairs(spikes/s) Sum Average Sum of responses to single objects(spikes/s) Example IT neuron Zoccolan, Cox and DiCarlo, 2005Population analysis Response to multiple objects (spikes/s) (spikes/s) 120 80 40 0 12080400 rPairs (n Sum Average 150 50 0 150100500 rTriplets (n Sum Average Sum of responses to single objects = 0.92 = 79) = 0.91 = 48) 100 Zoccolan,Cox and DiCarlo, 2005Summary: The Core and multiple objects Under the conditions described here: • An “average rule” is a very good predictor of the response of individual IT neurons (explains ~63% of response variance  r ≈ 0.8) • => The response pattern of The Core can be predicted by the response pattern to each constituent object • => useful for supporting the simultaneous representation of multiple objectsThe Core and “natural vision”How does “natural vision” challenge the basic modelof core vision?1) Eye movements ( “free viewing” )2) Clutter / Scene / Context: objectsappear among other objects andon backgrounds3) Goal directed (e.g. feature andspatial attention, motorpreparation to act, arousal)Not much to worryabout here.Sheinberg and Logothetis, 2001DiCarlo and Maunsell, 2000Very important challenge.Beginnings of a systematic understanding.Zoccolan, Cox and DiCarlo,