A Hyva rinen and P O Hoyer A Two Layer Sparse Coding Model Learns Simple and Complex Cell Receptive Fields and Topography from Natural Images presented by Hsin Hao Yu Department of Cognitive Science November 7 2001 An overview of the visual pathway 2 Basic V1 physiology Simple cells approximately linear lters localized oriented band pass phase sensitive Complex cells non linear phase insensitive Question Why do we have these neurons 3 The principle of redundancy reduction The Principle of redundancy reduction The world is highly structured The purpose of early sensory processing is to transform the redundant sensory input to an e cient code Barlow 1961 Two approaches have been developed to apply this idea to study the visual cortex 1 Sparse coding eg Olshausen and Field 2 Independent Component Analysis eg Bell and Sejnowski 4 Compact coding vs Sparse coding What does a e cient code means Strategy 1 Compact coding represents data with minimum number of units This requirement often produces solutions that s similar to Principal Component Analysis but the principal components do not resemble any receptive eld structures found in the visual cortex 5 Principal components of natural images Not localized and no orientational selectivity 6 Compact coding vs Sparse coding Strategy 2 Sparse coding represents data with minimum number of active units but the dimensionality of the representation is the same as or even larger than the dimensionality of the input data 7 Learning sparse codes image model We use the linear generative mode That is I x y ai i x y i where I x y is a patch of natural image and ai are coe cients to the basis functions i x y A neural network interpretation writing images as column vectors 1 a1 I an or I A Thus A W I where W 1 A is the output layer of a linear network and W is the weight matrix ie lters 8 Learning sparse codes algorithm Olshausen and Field 1996 For the image model I x y ai i x y i We require that the distributions of the coe cients ai are sparse This can be achieved by minimizing the following cost function E f idelity sparseness S x f idelity sparseness 2 x y I x y i ai i x y i S ai log 1 x2 9 Maximum likelihood and sparse codes The sparse coding algorithm can be interpreted as nding that maximizes the average log likelihood of the images under a sparse independent prior delity negative log likelihood of the image given and a assuming gaussian noise P I a sparseness 1 Z N e I a 2 2 2 N sparse independent prior for a P a i e S ai So E log P I a P a It can be shown that minimizing E is equal to maximizing P I given some approximation assumptions 10 Supergaussian distributions S ai log 1 a2i S ai ai 1 1 a2i Cauchy distribution P ai e x Laplace distribution P ai 11 Independent Component Analysis In the context of natural image analysis I x y ai i x y i where the number of ai equals to the dimensionality of I We require that ai as random variables are independent to each other That is P ai aj P ai In a more general context let I be a random vector The goal of the Independent Component Analysis is to nd a matrix W such that the components of A W I are non gaussian and independent to each other 12 The Infomax ICA Bell and Sejnowski 1995 derived a learning rule for ICA by maximizing the entropy of a neural network with logistic or Laplace neurons Similar or equivalent algorithms can be derived from many other frameworks Let H X be the entropy of X The joint entropy of a1 and a2 can be written as H a1 a2 H a1 H a2 I a1 a2 where I a1 a2 is the mutual information between a1 and a2 a1 a2 are independent to each other when I a1 a2 0 We approximate the solution by maximizing H a1 a2 13 Independent components of natural images Olshausen and Field 1996 Bell and Sejnowski 1996 16x16 basis patches 12x12 lters 14 More ICA applications 1 Direction selectivity van Hatern et al 1998 2 Flow eld templates Park and Jabri 2000 3 Color Hoyer 2000 Tailor 2000 Lee 2001 4 Binocular vision Hoyer 2000 5 Audition Bell and Sejnowski 1996 Lewicki 15 Complex cells and topography Hyva rinen and Hoyer 2001 uses a hierarchical network and the sparse coding principle to explain the emergence of complex cell like receptive elds and topographic structures of simple cells 16 from Hu bener et al 1997 17 The ice cube model of V1 layer 4c 18 Network architecture 19 20 Results summary simple cell physiology orientation freq selective phase position senstive simple cell topography orientation continuity but not phase orientation singularities or pinwheels blob grouping of low freq complex cells physiology orientation freq selective phase position insensitive 21
View Full Document
Unlocking...