Latency, duration and codes for objectsin inferior temporal cortexRecognition can be very fastStimulus presentationReading the neuronal codeMind reading  Neuronal readingCan we read out what the monkey is seeing?Input to the classifierAccurate read-out of object category and identity from a small populationLocal field potentials (the “input”) also show selectivityThe classifier extrapolates to new scales and positionsOther observationsWe can decode object information from the model unitsThe model shows scale and position invarianceLatency, duration and codes for objectsin inferior temporal cortexLatency, duration and codes for objectsin inferior temporal cortexGabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarloCenter for Biological and Computational LearningComputation and Systems Biology InitiativeMcGovern Institute for Brain ResearchMassachusetts Institute of TechnologyRecognition can be very fastStimulus presentation100 ms100 mstimePassive viewing(fixation task)5 objects presentedper second•Recordings of spiking activity from macaque monkeys•Recordings in an area involved in object recognition (inferior temporal cortex)•10-20 repetitions per stimulus• Presentation order randomized• 77 stimuli drawn from 8 pre-defined categoriesRecordings made by Chou Hung and James DiCarloReading the neuronal code12345Neuron 1 Neuron 2 Neuron 3 ObjectYes No No1Neuron 1 Neuron 2 Neuron 3 ObjectYes No No1Yes Yes No2Yes Yes Yes3Neuron 1 Neuron 2 Neuron 3 ObjectYes No No1Yes Yes No2Mind reading Æ Neuronal readingCan we read out what the monkey is seeing?xLearning from (x,y) pairsy ∈ {1,…,8}Input to the classifier100 ms0 ms 200 ms 300 mswMUA: spike counts in each binLFP: power in each binMUA+LFP: concatenation of MUA and LFPAccurate read-out of object category and identity from a small populationHung*, Kreiman*, Poggio, DiCarlo. Science 2005Decoding requires very few spikesw = 12.5 msHung*, Kreiman*, Poggio, DiCarlo. Science 2005Local field potentials (the “input”) also show selectivityKreiman*, Hung*, Kraskov, Quiroga, Poggio, DiCarlo. Neuron 2006MUA: multi-unit spiking activitySUA: single-unit spiking activityLFP: local field potentialsMUA & LFP: MUA combined with LFPThe classifier extrapolates to new scales and positions00.10.20.30.40.50.60.7n=31 n=24 n=9 n=9 n=17 n=10chanceTRAINTEST00.10.20.30.40.50.60.7n=31 n=24 n=9 n=9 n=17 n=10chanceTRAINTEST00.10.20.30.40.50.60.7n=31 n=24 n=9 n=9 n=17 n=10chanceTRAINTESTOther observations• We can decode information from local field potentials. MUA+LFP > MUA > LFP• Feature selection significantly improves performance. Choosing the “best” neurons >> randomly selecting neurons• We can decode the time of stimulus onset• We can also read out coarse “where” information• Decoding is robust to internal and external perturbations• The population can extrapolate to novel pictures within known categoriesWe can decode object information from the model unitsSerre, Kouh, Cadieu, Knoblich, Kreiman, Poggio. MIT AI Memo 2005The model shows scale and position invarianceLatency, duration and codes for objectsin inferior temporal cortexGabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarloCenter for Biological and Computational LearningComputation and Systems Biology InitiativeMcGovern Institute for Brain ResearchMassachusetts Institute of