Introduction to Neural NetworksU. Minn. Psy 5038Sculpting energy landscapes: interpolation and gradient descent‡InitializationOff@SetDelayed::writeDOff@General::spell1DUsing energy and gradient descent to derive update rulesEarlier we studied how to set up the weights in a discrete response TLU network for the correspondence problem in stereopsis. The weights were determined by an analysis of the constraints needed to find a unique correspondence between the pixels in the left and right eyes. We didn't compute energy in that example, but pointed out that energy could play a useful role as an index to describe how well the network's state vector was moving towards the correct answer in state space. In particular, the energy function contains information about the stable points in state space.In effect, we sculpted the energy landscape by hand-wiring the weights according to the constraints that were determined heuristically.In the TIP examples, we reconstructed stored letters from partial information. In that case, the weights were determined by Hebbian learning. So the energy landscape was sculpted by state vectors to be stored.But we can also do things the other way around. Rather than figuring out the weights for a Hopfield-style network that has a known relationship to an energy function, we first specify the energy function, and then figure out an update rule that will descend the energy landscape. We followed an analogous strategy when we set up an error function in terms of weights, and then did gradient descent to find the weights that minimized the error to learn the weights. But one can also set up the analog to the error function (weights variable), that is an energy function of the state vector (weights fixed), and then use gradient descent to derive a rule to find minima of this energy function. From the point of view of neural networks, this update rule may look nothing like what neurons do. But it may be the best way to start--that is, by sculpting the energy function directly, not worrying about "weights", and then see what emerges in terms of an update rule.We are going to follow this strategy in this notebook on a simple problem of interpolation. It will turn out that our update rule is the simplest neural model--a linear summer. However, this kind of analysis provides a starting point for more complicated energy functions with correspondingly non-linear update rules.The energy function is sometimes referred to as a cost function or objective function.In the second main part of this notebook (Deriving learning rules), we return to the problem of deriving learning rules from cost functions over weights using gradient descent. We apply it to the problem of self-organization where the goal is to learn a decorrelating set of weight vectors, which (unlike PCA) are not necessarily orthogonal.Our main tool is gradient descent. Given an objective or energy function, there are often better tools for finding the minimum (e.g. see Hertz et al.). But gradient descent is a simple and intuitive starting point.2 Lect_20b_SculptEn.nbInterpolation problems in perceptionA major theoretical problem in vision has to do with the fact that local changes in image intensities are usually ambiguous in natural images. A change in shading can mean an change in shape (or a change in illumination, e.g. a cast shadow). A change in image color can mean a change in the reflectivity of a surface (or a change in the illumination--two quite different causes). Changes of intensity of pixels in time provide information about surface structure, and the viewer's relation to that surface. In order to solve problems such as those above, researchers have studied special cases: shape-from-shading, reflectivity from color (color constancy), optic flow field from the flow of intensities, and more. One recurring theme in these problems is that the data available in the image does not fully constrain the estimate of the surface or surface properties one would like to compute. Earlier we studied the random dot stereogram in which each image had many local features densely packed, but there was ambiguity in matching left eye pixels to those in the right. Here another stereo example. This time there are few features sparsely packed.Lect_20b_SculptEn.nb 3backwidth = 50; backheight = 50;x0 =backwidth2; y0 =backheight2;vwidth =backwidth4; vheight =backheight8;vxoff =backwidth2; vyoff =backheight2;hwidth =backwidth8; hheight =backheight4; hxoff =backwidth2;hyoff =backheight2;gleft =Show@Graphics@[email protected]`D, Rectangle@80, 0<, 8backwidth, backheight<D,[email protected]`D, Rectangle@8x0 - vwidth, y0 - vheight<,8x0 + vwidth, y0 + vheight<D, [email protected]`D,Rectangle@8x0 - hwidth, y0 - hheight<, 8x0 + hwidth, y0 + hheight<D<D,AspectRatio Ø Automatic, DisplayFunction Ø IdentityD;gright =Show@Graphics@[email protected]`D, Rectangle@80, 0<, 8backwidth, backheight<D,[email protected]`D, Rectangle@8x0 - vwidth - 1, y0 - vheight<,8x0 + vwidth - 1, y0 + vheight<D, [email protected]`D,Rectangle@8x0 - hwidth, y0 - hheight<, 8x0 + hwidth, y0 + hheight<D<D,AspectRatio Ø Automatic, DisplayFunction Ø IdentityD;Show@GraphicsRow@8gright, gleft<D, DisplayFunction Ø $DisplayFunctionD4 Lect_20b_SculptEn.nbTry a motion analog: Show[gleft,DisplayFunctionÆ $DisplayFunction]; Show[gright,DisplayFunctionÆ $DisplayFunction]; and then group the two pictures, and play as a slow movieIf you can cross your eyes, so that the left image is in the right eye, and the right image in the left, you will see a green horizontal bar floating out in front of a green vertical bar. So-called "free-fusing" isn't easy, but when you've got it, you should see a total of three green crosses. The one in the middle is the one in which the two images are fused by your brain--and this is the one we are talking about.The interesting point here is that even though there is no local information in the image to support the percept of a horizon-tal occluding bar, observers still see an illusory completion. It looks a bit like the figure below, except that the color of the horizontal bar is changed here slightly just for illustration.Show@Graphics@[email protected]`D, Rectangle@80, 0<, 8backwidth, backheight<D,[email protected]`D, Rectangle@8x0 - hwidth, y0 - hheight<,8x0 + hwidth, y0 + hheight<D, [email protected]`D,Rectangle@8x0 - vwidth, y0 - vheight<, 8x0 + vwidth, y0 + vheight<D<D,AspectRatio Ø Automatic, ImageSize Ø SmallDThe visual system seems to interpolate a surface between salient points--in this case the salient