1999 Nature America Inc http neurosci nature com articles Predictive coding in the visual cortex a functional interpretation of some extra classical receptive field effects Rajesh P N Rao1 and Dana H Ballard2 1 The Salk Institute Sloan Center for Theoretical Neurobiology and Computational Neurobiology Laboratory 10010 N Torrey Pines Road La Jolla California 92037 USA 2 Department of Computer Science University of Rochester Rochester New York 14627 0226 USA 1999 Nature America Inc http neurosci nature com Correspondence should be addressed to R P N R rao salk edu We describe a model of visual processing in which feedback connections from a higher to a lowerorder visual cortical area carry predictions of lower level neural activities whereas the feedforward connections carry the residual errors between the predictions and the actual lower level activities When exposed to natural images a hierarchical network of model neurons implementing such a model developed simple cell like receptive fields A subset of neurons responsible for carrying the residual errors showed endstopping and other extra classical receptive field effects These results suggest that rather than being exclusively feedforward phenomena nonclassical surround effects in the visual cortex may also result from cortico cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images Neurons that respond optimally to line segments of a particular length were first reported in early studies of the cat and monkey visual cortex1 2 These neurons which are especially abundant in cortical layers 2 and 3 have the curious property of endstopping or end inhibition a vigorous response to an optimally oriented line segment is reduced or eliminated when the same stimulus extends beyond the neuron s classical receptive field RF Such extra classical RF effects occur in several visual cortical areas including V1 area 17 refs 2 3 V2 area 18 refs 1 4 V4 ref 5 and MT6 In most of these cases neural responses are suppressed when stimulus properties at the center such as orientation velocity or direction of motion match those in the surrounding extra classical RF Why should a neuron that responds to a stimulus stop responding when the same stimulus extends beyond the classical RF Some studies have postulated a role for hypercomplex endstopped neurons in the detection of visual curvature 1 7 Others have suggested a role for these cells in detecting corners and line terminations8 occlusion9 perceptual grouping10 and illusory contours11 However a straightforward extension of these arguments to extra classical RF effects in different cortical areas has been difficult We have previously shown that a model 12 based on the principle of Kalman filtering can account for certain visual cortical responses in a monkey freely viewing natural images13 It was conjectured that a similar model might also account for endstopping and other extra classical effects Here we show simulations suggesting that extra classical RF effects may result directly from predictive coding of natural images The approach postulates that neural networks learn the statistical regularities of the natural world signaling deviations from such regularities to higher processing centers This reduces redundancy by removing the predictable and hence nature neuroscience volume 2 no 1 january 1999 redundant components of the input signal Roots of this idea can be found in early information theoretic approaches to sensory processing14 16 More recently it has been used to explain the spatiotemporal response properties of cells in the retina 17 19 and lateral geniculate nucleus LGN 20 21 Because neighboring pixel intensities in natural images tend to be correlated values near the image center can often be predicted from surrounding values Thus the raw image intensity value at each pixel can be replaced by the difference between a center pixel value and its spatial prediction from a linear weighted sum of the surrounding values This decorrelates or whitens the inputs17 19 and reduces output redundancy providing a functional explanation for center surround receptive fields in the retina and LGN The values of a given pixel also tend to correlate over time A retinal LGN cell s phasic response can thus be interpreted as the difference between the actual input and its temporal prediction based on a linear weighted sum of past input values19 21 Similarly the responses of retinal photoreceptors sensitive to different wavelengths are often correlated because their spectral sensitivities overlap Thus the L cone long wavelength or red receptor response may predict the M cone medium wavelength or green receptor response and the L and M cone responses together may predict the S cone short wavelength or blue receptor response Thus the color opponent red green and blue red green channels in the retina might reflect predictive coding in the chromatic domain similar to that of the spatial and temporal domains18 Using a hierarchical model of predictive coding we show that visual cortical neurons with extra classical RF properties can be interpreted as residual error detectors signaling the difference between an input signal and its statistical prediction based on an efficient internal model of natural images 79 1999 Nature America Inc http neurosci nature com articles 1999 Nature America Inc http neurosci nature com a b Fig 1 Hierarchical network for predictive coding a General c architecture of the hierarchical predictive coding model At each hierarchical level feedback pathways carry predictions of neural activity at the lower level whereas feedforward pathways carry residual errors between the predictions and actual neural activity These errors are used by the predictive estimator PE at each level to correct its current estimate of the input signal and generate the next prediction b Components of a PE module composed of feedforward neurons encoding the synaptic weights UT neurons whose responses r maintain the current estimate of the input signal feedback neurons encoding U and conveying the prediction f Ur to the lower level and error detecting neurons computing the difference r rtd between the current estimate r and its top down prediction rtd from a higher level c A three level hierarchical network used in the simulations An input image was analyzed by three level 1 PE modules each predicting its own local image patch The responses r of all
View Full Document
Unlocking...