© 1999 Nature America Inc. • http://neurosci.nature.com© 1999 Nature America Inc. • http://neurosci.nature.comnature neuroscience • volume 2 no 1 • january 1999 79Neurons that respond optimally to line segments of a partic-ular length were first reported in early studies of the cat andmonkey visual cortex1,2. These neurons, which are especiallyabundant in cortical layers 2 and 3, have the curious propertyof endstopping (or end-inhibition): a vigorous response to anoptimally oriented line segment is reduced or eliminated whenthe same stimulus extends beyond the neuron’s classical recep-tive field (RF). Such ‘extra-classical’ RF effects occur in sever-al visual cortical areas, including V1 (area 17; refs 2, 3), V2(area 18; refs 1, 4), V4 (ref. 5) and MT6. In most of these cases,neural responses are suppressed when stimulus properties atthe center, such as orientation, velocity or direction of motion,match those in the surrounding extra-classical RF.Why should a neuron that responds to a stimulus stopresponding when the same stimulus extends beyond the clas-sical RF? Some studies have postulated a role for ‘hypercom-plex’ endstopped neurons in the detection of visualcurvature1,7. Others have suggested a role for these cells indetecting corners and line terminations8, occlusion9, percep-tual grouping10and illusory contours11. However, a straight-forward extension of these arguments to extra-classical RFeffects in different cortical areas has been difficult. We havepreviously shown that a model12based on the principle ofKalman filtering can account for certain visual cortical respons-es in a monkey freely viewing natural images13. It was conjec-tured that a similar model might also account for endstoppingand other extra-classical effects.Here we show simulations suggesting that extra-classical RFeffects may result directly from predictive coding of naturalimages. The approach postulates that neural networks learnthe statistical regularities of the natural world, signaling devi-ations from such regularities to higher processing centers. Thisreduces redundancy by removing the predictable, and henceredundant, components of the input signal. Roots of this ideacan be found in early information-theoretic approaches to sen-sory processing14–16. More recently, it has been used to explainthe spatiotemporal response properties of cells in the reti-na17–19and lateral geniculate nucleus (LGN)20,21. Becauseneighboring pixel intensities in natural images tend to be cor-related, values near the image center can often be predictedfrom surrounding values. Thus, the raw image-intensity valueat each pixel can be replaced by the difference between a cen-ter pixel value and its spatial prediction from a linear weight-ed sum of the surrounding values. This decorrelates (orwhitens) the inputs17,19and reduces output redundancy, pro-viding a functional explanation for center–surround receptivefields in the retina and LGN. The values of a given pixel alsotend to correlate over time. A retinal/LGN cell’s phasic responsecan thus be interpreted as the difference between the actualinput and its temporal prediction based on a linear weightedsum of past input values19–21. Similarly, the responses of reti-nal photoreceptors sensitive to different wavelengths are oftencorrelated because their spectral sensitivities overlap. Thus,the L-cone (long-wavelength or ‘red’ receptor) response maypredict the M-cone (medium-wavelength or ‘green’ receptor)response, and the L- and M-cone responses together may pre-dict the S-cone (short-wavelength or ‘blue’ receptor) response.Thus, the color-opponent (red – green) and blue – (red +green) channels in the retina might reflect predictive codingin the chromatic domain similar to that of the spatial and tem-poral domains18.Using a hierarchical model of predictive coding, we showthat visual cortical neurons with extra-classical RF propertiescan be interpreted as residual error detectors, signaling the dif-ference between an input signal and its statistical predictionbased on an efficient internal model of natural images.articlesPredictive coding in the visual cortex: a functional interpretation of someextra-classical receptive-field effectsRajesh P. N. Rao1and Dana H. Ballard21The Salk Institute, Sloan Center for Theoretical Neurobiology and Computational Neurobiology Laboratory, 10010 N. Torrey Pines Road, La Jolla, California 92037, USA2Department of Computer Science, University of Rochester, Rochester, New York 14627-0226, USACorrespondence should be addressed to R.P.N.R. ([email protected])We describe a model of visual processing in which feedback connections from a higher- to a lower-order visual cortical area carry predictions of lower-level neural activities, whereas the feedforwardconnections carry the residual errors between the predictions and the actual lower-level activities.When exposed to natural images, a hierarchical network of model neurons implementing such amodel developed simple-cell-like receptive fields. A subset of neurons responsible for carrying theresidual errors showed endstopping and other extra-classical receptive-field effects. These resultssuggest that rather than being exclusively feedforward phenomena, nonclassical surround effects inthe visual cortex may also result from cortico-cortical feedback as a consequence of the visual systemusing an efficient hierarchical strategy for encoding natural images.© 1999 Nature America Inc. • http://neurosci.nature.com© 1999 Nature America Inc. • http://neurosci.nature.com80 nature neuroscience • volume 2 no 1 • january 1999ResultsHIERARCHICAL PREDICTIVE CODING MODELEach level in the hierarchical model network (except the lowestlevel, which represents the image) attempts to predict the respons-es at the next lower level via feedback connections (Fig. 1a). Theerror between this prediction and the actual response is then sentback to the higher level via feedforward connections. This errorsignal is used to correct the estimate of the input signal at eachlevel (see Methods and Fig. 1b), similar to some previous mod-els22–24(see also refs 15, 25, 26). The prediction and error-cor-rection cycles occur concurrently throughout the hierarchy, sotop-down information influences lower-level estimates, and bot-tom-up information influences higher-level estimates of the inputsignal. Lower levels operate on smaller spatial (and possibly tem-poral) scales, whereas