116.891Computer Vision and ApplicationsProf. Trevor. DarrellLecture 2:– Linear Filters and Convolution (review)– Fourier Transform (review)– Sampling and Aliasing (review)Readings: F&P Chapter 7.1-7.62Recap: Cameras, lenses, and calibrationLast time:• Camera models• Projection equations• Calibration methodsImages are projections of the 3-D world onto a 2-D plane…3Recap: pinhole/perspectivePinole camera model -box with a small hole in it:Perspective projection:Forsyth&Ponce°°¯°°® zyfyzxfx''''4Recap: Intrinsic parameters00 )sin()cot( vzyvuzyzxu 6-6,,¸¸¸¸¸¹·¨¨¨¨¨©§¸¸¸¸¹·¨¨¨¨©§  ¸¸¸¹·¨¨¨©§1000100)sin(0)cot(1100zyxvuzvu6-6,,Using homogenous coordinates,we can write this as:or:PKzp 0 1 5Recap: Combining extrinsic and intrinsic calibration parametersForsyth&PonceWCWCWCOPRP  PORKzpWCCW 1 PKzp 0 1 PΜzp 1 IntrinsicExtrinsic6Other ways to write the same equation¸¸¸¸¸¹·¨¨¨¨¨©§¸¸¸¹·¨¨¨©§ ¸¸¸¹·¨¨¨©§1.........11321zyxTTTWWWmmmzvuPMzp 1PmPmvPmPmubbbb3231pixel coordinatesworld coordinatesz is in the camera coordinate system, but we can solve for that, since , leading to:zPmb3127Recap:Camera calibration0)(0)(3231b b iiiiPmvmPmumStack all these measurements of i=1…n points into a big matrix:¸¸¸¸¸¸¹·¨¨¨¨¨¨©§ ¸¸¸¹·¨¨¨©§¸¸¸¸¸¸¹·¨¨¨¨¨¨©§00000000321111111mmmPvPPuPPvPPuPTnnTnTTnnTTnTTTTTTPmPmvPmPmubbbb32318TodayReview of early visual processing– Linear Filters and Convolution– Fourier Transform– Sampling and AliasingYou should have been exposed to this material in previous courses; this lecture is just a (quick) review.Administrivia: – sign-up sheet– introductions9What is image filtering?• Modify the pixels in an image based on some function of a local neighborhood of the pixels.5 141 715 310Local image data7Modified image dataSome function10Linear functions• Simplest: linear filtering.– Replace each pixel by a linear combination of its neighbors.• The prescription for the linear combination is called the “convolution kernel”.5 141 715 3100.50.5 00100 00Local image data kernel7Modified image data11Convolution¦UlklkglnkmIgInmf,],[],[],[12Linear filtering (warm-up slide)original0Pixel offsetcoefficient1.0?313Linear filtering (warm-up slide)original0Pixel offsetcoefficient1.0Filtered(no change)14Linear filtering0Pixel offsetcoefficientoriginal1.0?15shift0Pixel offsetcoefficientoriginal1.0shifted16Linear filtering0Pixel offsetcoefficientoriginal0.3?17Blurring0Pixel offsetcoefficientoriginal0.3Blurred (filterapplied in both dimensions).18Blur examples0Pixel offsetcoefficient0.3original8filtered2.4impulse419Blur examples0Pixel offsetcoefficient0.3original8filtered484impulseedge0Pixel offsetcoefficient0.3original8filtered2.420Smoothing reduces noise• Generally expect pixels to “be like” their neighbours– surfaces turn slowly– relatively few reflectance changes• Generally expect noise processes to be independent from pixel to pixel• Implies that smoothing suppresses noise, for appropriate noise models• Scale– the parameter in the symmetric Gaussian– as this parameter goes up, more pixels are involved in the average– and the image gets more blurred– and noise is more effectively suppressed21The effects of smoothingEach row shows smoothingwith gaussians of differentwidth; each column showsdifferent realisations of an image of gaussian noise.22Linear filtering (warm-up slide)original02.0?01.023Linear filtering (no change)original02.001.0Filtered(no change)24Linear filteringoriginal02.000.33?525(remember blurring)0Pixel offsetcoefficientoriginal0.3Blurred (filterapplied in both dimensions).26Linear filteringoriginal02.000.33?27Sharpening original02.000.33Sharpened original28Sharpening examplecoefficient-0.3original8Sharpened(differences areaccentuated; constantareas are left untouched).11.21.7-0.25829Sharpeningbefore after30Gradients and edges• Points of sharp change in an image are interesting:– change in reflectance– change in object– change in illumination– noise• Sometimes called edge points• General strategy– linear filters to estimate image gradient– mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points).631Smoothing and Differentiation• Issue: noise– smooth before differentiation– two convolutions to smooth, then differentiate?– actually, no - we can use a derivative of Gaussian filter• because differentiation is convolution, and convolution is associative32The scale of the smoothing filter affects derivative estimates, and alsothe semantics of the edges recovered.1 pixel3 pixels7 pixels33Oriented filtersGabor filters (Gaussian modulated harmonics) at differentscales and spatial frequenciesTop row shows anti-symmetric (or odd) filters, bottom row thesymmetric (or even) filters.34Linear image transformations• In analyzing images, it’s often useful to make a change of basis.Fourier transform, orWavelet transform, orSteerable pyramid transformfUFVectorized imagetransformed image35Self-inverting transformsFUFUf 1Same basis functions are used for the inverse transformU transpose and complex conjugate36An example of such a transform: the Fourier transformdiscrete domain¦¦ 1010ln],[1],[MkNlNMkmienmFMNlkf5Inverse transform¦¦ 1010ln],[],[MkNlNMkmielkfnmF5Forward transform737To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part ---as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction. 38Here u and v are larger than in the previous slide.39And larger still...40Phase and Magnitude• Fourier transform of a real function is complex– difficult to plot, visualize– instead, we can think of the phase and magnitude of the transform• Phase is the phase of the complex transform• Magnitude is the magnitude of the complex transform•