Computational VisionU. Minn. Psy 5036Daniel KerstenLecture 10: Image processingInitialize‡Read in Add-in packages:In[14]:=Off@General::"spell1"D;<< "BarCharts`"; << "Histograms`"; << "PieCharts`"SetOptions@ArrayPlot, ColorFunction Ø "GrayTones", DataReversed Ø True,Frame Ø False, AspectRatio Ø Automatic, Mesh Ø False,PixelConstrained Ø True, ImageSize Ø SmallD;‡The input 64x64 image: faceIn[18]:=width = Dimensions@faceDP1T; size = width;hsize =width2; hfwidth = hsize; height = Dimensions@faceDP2T; face;gface = ArrayPlot@faceDOut[19]=OutlineOutlineLast timeSingle - channel spatial filteringA fixed choice of lower and upper bound spatial frequencies, r0 and r1, respectively.In[20]:=filter@r0_, r1_D :=Table@If@HHx - hsizeL^ 2 + Hy - hsizeL^ 2 > r0L &&Hx - hsizeL^ 2 + Hy - hsizeL^ 2 < r1, 1, 0D , 8x, 1, size<, 8y, 1, size<D;In[21]:=Manipulate@tt = Chop@InverseFourier@Fourier@filter@r0, r1DD Fourier@faceDDD;tt = RotateLeft@tt, 8hsize, hsize<D;GraphicsRow@8ArrayPlot@filter@r0, r1DD, ArrayPlot@ttD<D,88r0, 0<, 0, r1<, 88r1, hsize<, 0, 100<DOut[21]=r0r1Multiple spatial frequency channels2 10.ImageProcessing.nbMultiple spatial frequency channelsGlobal vs. local: Sinusoidal basis functions are global filtersPsychophysical experiments. ->Multi-resolution, but local filters->A model of the spatial filtering properties of neurons in the primary visual cortexMultiresolution analysis with local filtersInstead of sinusoidal basis functions, we can filter with localized "gabor function" filters:In[27]:=Grating[x_,y_,fx_,fy_,phase_] := Cos[(2.0 Pi (fx x + fy y) + phase)];GratingPatch[x_,y_,fx_,fy_,sig_,phase_] := Exp[-((x)^2 + (y)^2)/(2*sig^2)]*Grating[x,y,fx,fy,phase];kern[fx_, fy_, sig_,phase_] := Table[GratingPatch[x, y, fx, fy, sig,phase], {x, -1, 1, .1}, {y, -1, 1, .1}];10.ImageProcessing.nb 3In[31]:=Manipulate@GraphicsRow@8ArrayPlot@kern@fr * Cos@thetaD, fr * Sin@thetaD, sig, phaseDD,ArrayPlot@ListConvolve@kern@fr * Cos@thetaD, fr * Sin@thetaD,sig, phaseD, faceDD<D, 88fr, 1, "radial frequency"<, .1, 2<,88theta, .4, "orientation"<, 0, Pi<,88sig, .4, "envelope width"<, .001, 1<, 88phase, 0, "phase"<, .0, Pi ê2<DOut[31]=radial frequencyorientationenvelope widthphaseSelf-similarityBut another restriction is that the filters could all be the same shape, but just scaled versions of each other. The self - similar idea is important to vision because of the need for some kind of scale - invariance. Further, the self - similar aspect of these neural models bore a close resemblance to the emerging mathematical field of wavelet analysis. The emphases are different-- over - completeness may be important and vision does the projections in parallel (the serial algorithmic component of wavelet computation is integral to the mathematical interest).‡Human efficiency for detecting gabor patches4 10.ImageProcessing.nb‡Human efficiency for detecting gabor patchesBurgess, Wagner, Jennings and Barlow (1981) combined the SKE observer and spatial frequency analysis of human vision to find out how efficiently humans detected patterns. They showed in a 1981 Science article that narrowly windowed sinusoids were detected with high efficiency (>70%) when added to static visual noise. Further, these targets were detected more efficiently than disks of light.You basically have all the tools to replicate the experiment of Burgess et al. You can compute d' for the ideal observer for signal-known-exactly patterns. And you can generate Gaussian-windowed sinusoids and add them to gaussian white noise. If you measure the percent correct, and convert that to d' for the human observer, you can calculate the absolute efficiency for human detection--and contribute to answer the question of what the eye sees best.Watson, Barlow & Robson (1983) found that that a 7 c/deg grating drifting at 4 Hz, (with a narrow gaussian envelope in space and time) was detected more efficiently than other patterns. Further, the quantum efficiency was very low (<0.05%).Kersten (1984) measured efficiency for 1-d gratings (i.e. vertical) in temporal (1-d spatial) visual noise for various spatial frequencies and widths. Peak efficiency was found for patterns of about the same shape, regardless of spatial frequency. The cross-sectional profiles for high efficiency patterns corresponded to the diagonals in the above graph and looked like:Psychophysical measurements across spatial scale haven't been made systematically yet for various vertical sizes. One prediction would be that images of the following type would be efficiently detected in noise:10.ImageProcessing.nb 5Psychophysical measurements across spatial scale haven't been made systematically yet for various vertical sizes. One prediction would be that images of the following type would be efficiently detected in noise:When the filters have the same shape except for a change of scale (xØax), they are called self-similar.‡Bottom-line: image coding in terms of scale and orientation:A model for human spatial image representationAt each spatial location, project the image onto a collection of basis vectors (i.e. compute the dot product) that span a range of spatial scales and orientations:In general, these neural models of basis functions may be over-complete, and non-orthogonal. And there may be a range of phases. Above we show only the "sine-phase" or "edge-detectors" of Hubel and Wiesel.6 10.ImageProcessing.nbNeural image? Or neural image representation?We can view the response activities of a family of receptive fields of neurons as representing a filtered neural image of the input image. Although useful, this view can be misleading when we start to think about function, for "who is looking at the image"? Alternatively, thinking in terms of basis functions gives us another perspective. We can view the response activities of a family of receptive fields as a representation of the input image. If linear, an activity is the result of a projection of an image on to a basis function (receptive field weights). Given such a representation we can begin to ask questions like:1. Is the neural basis set complete? Can any image be represented? 2. Are image representations unique? Is any information lost? I.e. we do the inverse transformation, can the original input be reconstructed?Or are there "equivalent classes" of images--i.e. ones that all produce the same neural representation? Consider for exam-ple, a single-channel model with lateral inhibitory center-surround spatial filters--what is