Introduction to Neural NetworksSelf-organization and efficient neural codingInitializationIn[107]:=Off[SetDelayed::write];Off[General::spell1];In[109]:=scale256@image_D := Module@8a , b<,a = 255 êHMax@imageD - Min@imageDL;b = -a Min@imageD;Return@a image + bD;D;myhistogram@image_D := Module@8histx<,histx = BinCounts@Flatten@imageD, 80, 255, 1<D;Return@N@histx ê Plus üü histxDD;D;In[111]:=SetOptions@ArrayPlot, ColorFunction Ø "GrayTones", DataReversed Ø False,Frame Ø False, AspectRatio Ø Automatic, Mesh Ø False,PixelConstrained Ø True, ImageSize Ø SmallD;IntroductionUnsupervised learning--what does it mean to learn without a teacher?Statistical view:Learning as probability density estimation and exploitation.From N samples (e.g. images), can improve the representation and transmission of information? What does “improve” mean? Smaller number of dimensions? Noise resistance? fewer spikes per bit (less energy), fewer neurons?Straightforward to estimate probability with a single random variable, e.g. histogram, and to reduce dimensionality, e.g. mean, standard deviation, reduce dimensionality Less straightforward to estimate density as dimensions go up, i.e. a vector random variable: histogram, vector means, autocovariance, higher-order statistics,Less straightforward because of the problem of getting enough samples to fill all the bins in the histogram.But for natural patterns, it is typically the case that even tho' the data maybe n-dimensional vectors, the ensemble lives in a much smaller space. How can we find that space? Once density is found, what is it good for?We’ll first look at examples from vision and learn some basic principles of coding. Principles of efficient coding are general, and are applicable to other perceptual, and cognitive domains as well.Unsupervised learning--what does it mean to learn without a teacher?Statistical view:Learning as probability density estimation and exploitation.From N samples (e.g. images), can improve the representation and transmission of information? What does “improve” mean? Smaller number of dimensions? Noise resistance? fewer spikes per bit (less energy), fewer neurons?Straightforward to estimate probability with a single random variable, e.g. histogram, and to reduce dimensionality, e.g. mean, standard deviation, reduce dimensionality Less straightforward to estimate density as dimensions go up, i.e. a vector random variable: histogram, vector means, autocovariance, higher-order statistics,Less straightforward because of the problem of getting enough samples to fill all the bins in the histogram.But for natural patterns, it is typically the case that even tho' the data maybe n-dimensional vectors, the ensemble lives in a much smaller space. How can we find that space? Once density is found, what is it good for?We’ll first look at examples from vision and learn some basic principles of coding. Principles of efficient coding are general, and are applicable to other perceptual, and cognitive domains as well.‡Why a sigmoidal non-linearity at receptor level?In[112]:=2 Lect_21_SelfOrg.nbOut[112]=Lect_21_SelfOrg.nb 3‡Why spatial bandpass tuning? Why lateral inhibition?In[113]:=Out[113]=4 Lect_21_SelfOrg.nb‡Why would the V1 analyze spatial information in multiple channels tuned to spatial frequency and orientation?In[114]:=Grating[x_,y_,fx_,fy_,phase_] := Cos[(2.0 Pi (fx x + fy y) + phase)];GratingPatch[x_,y_,fx_,fy_,sig_,phase_] := Exp[-((x)^2 + (y)^2)/(2*sig^2)]*Grating[x,y,fx,fy,phase];kern[fx_, fy_, sig_,phase_] := Table[GratingPatch[x, y, fx, fy, sig,phase], {x, -1, 1, .05}, {y, -1, 1, .05}];Lect_21_SelfOrg.nb 5In[117]:=Manipulate@GraphicsRow@8ArrayPlot@kern@fr * Cos@thetaD, fr * Sin@ thetaD, sig, phaseDD<D,88fr, 1, "radial frequency"<, .1, 2<, 88theta, .4, "orientation"<, 0, Pi<,88sig, .4, "envelope width"<, .001, 1<, 88phase, 0, "phase"<, .0, 2 * Pi<DOut[117]=radial frequencyorientationenvelope widthphaseMotivation from visual coding1rst image statistics: Sigmoidal non-linearities, and histogram equalization by light receptors in the fly’s eyeIn 1981, Simon Laughlin published a paper in which he showed that the contrast response function of LMC interneurons (“large monopolar cells”) of the fly’s ommatidium had a sigmoidal non-linearity as shown above. This kind of non-linearity wasn’t new, and is common, especially in sensory response functions. It is a de facto standard point non-linearity in generic neural network models. But why the sigmoidal shape in sensory coding? The mechanistic explanation for this kind of sigmoidal shape was and has been that small signals tend to get suppressed (e.g. a “soft” threshold), and signals get saturated because of biophysical limitations at the high end.Laughlin came up with a different kind of answer, based on a functional argument that went along the following lines: the fly lives in a visual world in which big contrasts (local intensity relative to a global mean) (negative or positive) are less common than contrasts near zero, so neurons should devote more of their resolving capacity to the middle contrasts, i.e. those near zero. This kind of argument is based on information theory. In this lecture, we’ll use some basic tools of information theory to understand his model and others like it.In image processing jargon, the fly’s visual neuron is doing “histogram equalization”. Let’s plot the histogram for a natural image. 6 Lect_21_SelfOrg.nbIn 1981, Simon Laughlin published a paper in which he showed that the contrast response function of LMC interneurons (“large monopolar cells”) of the fly’s ommatidium had a sigmoidal non-linearity as shown above. This kind of non-linearity wasn’t new, and is common, especially in sensory response functions. It is a de facto standard point non-linearity in generic neural network models. But why the sigmoidal shape in sensory coding? The mechanistic explanation for this kind of sigmoidal shape was and has been that small signals tend to get suppressed (e.g. a “soft” threshold), and signals get saturated because of biophysical limitations at the high end.Laughlin came up with a different kind of answer, based on a functional argument that went along the following lines: the fly lives in a visual world in which big contrasts (local intensity relative to a global mean) (negative or positive) are less common than contrasts near zero, so neurons should devote more of their resolving