TitleContentsIntroduction Quantitative framework for the ventral streamFeedforward architecture and operations in the ventral streamLearningLearning a universal dictionary of shape-tuned (S) units: from S2 to S4 (V4 to AIT)Task-dependent learning: from IT to PFCTraining the model to become an expert by selecting features for a specific set of objectsPerformance on Natural ImagesComparison with state-of-the-art AI systems on different object categoriesPredicting human performance on a rapid-categorization taskImmediate recognition and feedforward architectureTheory and humansResultsVisual areasV1 and V2V1V2V4Properties of V4Modeling Individual V4 ResponsesPredicting V4 ResponseModel Units Learned from Natural Images are Compatible with V4ITPaperclip experimentsMultiple object experimentsRead-out of object information from IT neurons and from model unitsPFCBiophysics of the 2 basic operations: biologically plausible circuits for tuning and maxNon-spiking circuitsNormalizationMaxSpiking circuits, wires and cablesNormalizationMaxSummary of resultsComparison with experimental dataFuture experimentsDiscussionA theory of visual cortexNo-go results from modelsExtending the theory and open questionsOpen questionsPredictionsExtending the theory to include backprojectionsA challenge for cortical physiology and cognitive scienceAppendicesDetailed model implementation and parametersComparing S1 and C1 units with V1 parafoveal cellsMethodsSpatial frequency tuningOrientation tuningTraining the model to become an expertComparison between Gaussian tuning, normalized dot product and dot productIntroductionNormalized dot product vs. GaussianCan a tuning behavior be obtained for p q and r=1?Dot product vs. normalized dot product vs. GaussianRobustness of the modelRBF networks, normalized RBF and cortical circuits in prefrontal cortexTwo Spot Reverse Correlation in V1 and C1 in the modelIntroductionTwo-spot reverse correlation experiment in V1Two-spot reverse correlation experiment in the modelDiscussionFitting and Predicting V4 ResponsesAn Algorithm for Fitting Neural ResponsesWhat Mechanisms Produce 2-spot Interaction Maps?A Common Connectivity Pattern in V4Fast readout of object information from different layers of the model and from IT neuronsMethodsFurther observationsPredictionsCategorization in IT and PFCBiophysics detailsPrimer on the underlying biophysics of synaptic transmissionNon-spiking circuitsSpiking circuitsBrief discussion of some frequent questionsConnectivity in the modelPosition invariance and localizationInvariance and broken linesConfigural informationInvariance and multiple objectsBibliographyA theory of object recognition: computations and circuits in the feedforward path of the ventral stream in primate visual cortex Thomas Serre, Minjoon Kouh, Charles Cadieu, Ulf Knoblich, Gabriel Kreiman and Tomaso Poggio1 Center for Biological and Computational Learning, McGovern Institute for Brain Research, Computer Science and Artificial Intelligence Laboratory, Brain Sciences Department, Massachusetts Institute of Technology Abstract We describe a quantitative theory to account for the computations performed by the feedforward path of the ventral stream of visual cortex and the local circuits implementing them. We show that a model instan-tiating the theory is capable of performing recognition on datasets of complex images at the level of human observers in rapid categorization tasks. We also show that the theory is consistent with (and in some case has predicted) several properties of neurons in V1, V4, IT and PFC. The theory seems sufficiently com-prehensive, detailed and satisfactory to represent an interesting challenge for physiologists and modelers: either disprove its basic features or propose alternative theories of equivalent scope. The theory suggests a number of open questions for visual physiology and psychophysics. This version replaces the preliminary “Halloween” CBCL paper from Nov. 2005. This report describes research done within the Center for Biological & Computational Learning in the Department of Brain & Cognitive Sciences and in the Computer Science and Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. This research was sponsored by grants from: Office of Naval Research (DARPA) under contract No. N00014-00-1-0907, National Science Foundation (ITR) under contract No. IIS-0085836, National Science Foundation (KDI) under contract No. DMS-9872936, and National Science Foundation under contract No. IIS-9800032 Additional support was provided by: Central Research Institute of Electric Power Industry, Center for e-Business (MIT), Eastman Kodak Company, DaimlerChrysler AG, Compaq, Honda R&D Co., Ltd., Komatsu Ltd., Merrill-Lynch, NEC Fund, Nippon Telegraph & Telephone, Siemens Corporate Research, Inc., The Whitaker Foundation, and the SLOAN Foundations. 1To whom correspondence should be addressed.Contents Contents 1 Introduction 4 2 Quantitative framework for the ventral stream 9 2.1 Feedforward architecture and operations in the ventral stream . . . . . . . . . . . . . . . . . . 9 2.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2.1 Learning a universal dictionary of shape-tuned (S) units: from S2 to S4 (V4 to AIT) . . 14 2.2.2 Task-dependent learning: from IT to PFC . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.3 Training the model to become an expert by selecting features for a specific set of objects 16 3 Performance on Natural Images 19 3.1 Comparison with state-of-the-art AI systems on different object categories . . . . . . . . . . . 19 3.2 Predicting human performance on a rapid-categorization task . . . . . . . . . . . . . . . . . . 21 3.3 Immediate recognition and feedforward architecture . . . . . . . . . . . . . . . . . . . . . . . 22 3.4 Theory and humans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4 Visual