Sampling and pixelsMarc LevoyComputer Science DepartmentStanford UniversityCS 178, Spring 2011Begun 4/14/11. Finished 4/19.! Marc LevoyWhy study sampling theory?!Why do I sometimes get moiré artifacts in my images?!What is an antialiasing filter?!How many megapixels is enough?!How do I compute circle of confusion for depth of field?!Is Apple’s “Retina Display” just hype?!What do MTF curves in lens reviews mean?!What does Photoshop do when you downsize/upsize?!What’s the difference between more pixels and more bits?2! Marc LevoyOutline!frequency representations of images•filtering, blurring, sharpening•MTF as a measure of sharpness in images!resolution and human perception•the spatial resolution of typical display media•the acuity of the human visual system•the right way to compute circle of confusion ( C )!sampling and aliasing•aliasing in space and time, 1D and 2D, images and audio•prefiltering using convolution to avoid aliasing•prefiltering and sampling in photography!sampling versus quantization3! Marc LevoyFrequency representations!a sum of sine waves, each of different wavelength ( frequency ) and height ( amplitude ), can approximate arbitrary functions!to adjust horizontal position ( phase ), replace with cosine waves, or use a mixture of sine and cosine waves4(Foley)! Marc Levoy!Fourier series: any continuous, integrable, periodic function can be represented as an infinite series of sines and cosines!Discrete Fourier transform (DFT):5Fourier analysisf (x) =a02+ ancos(nx) + bnsin(nx)[ ]n=1!"Xk= xne!2"iNk nn = 0N !1#, k = 0,..., N ! 1xn= Xke!2"iNk nk = 0N !1#, n = 0,..., N ! 1where ei n x= cos(nx) + i sin(nx)fast version is called the Fast Fourier Transform (FFT)! Marc LevoyFourier transforms of images6imageFFT(image)% In Matlab:image = double(imread('flower.tif'))/255.0;fourier = fftshift(fft2(ifftshift(image)));fftimage = log(max(real(fourier),0.0))/20.0;often called a spectrumcomplete spectrum is two images - sines and cosines, or real and imaginary components! Marc LevoyFourier transforms of images7image!r• sinusoids in intensity as a function of spatial position• ! gives angle of sinusoid• r gives spatial frequencyFFT(image)! Marc LevoyFourier transforms of images8imageFFT(image)! Marc LevoyBlurring in the Fourier domain9imageFFT(image)! Marc LevoySharpening in the Fourier domain10imageFFT(image)! Marc LevoySharpening in the Fourier domain11imageFFT(image)! Marc Levoy12imageFFT(image)Q. What does this filtering operation do?! Marc LevoyBlurring in x, sharpening in y13imageFFT(image)argh, astigmatism!! Marc LevoyDescribing sharpness in images:the modulation transfer function (MTF)!the amount of each spatial frequency that can be reproduced by an optical system14(imatest.com)! Marc LevoyTwo different MTF curves!in one curve, contrast stays high, but drops off at a relatively low resolution!in the other curve, higher-resolution features are preserved, but contrast is low throughout15! Marc LevoySharpness versus contrast16(imatest.com)(Canon)! Marc LevoyRecap!any image can be equivalently represented by its Fourier transform, a.k.a. frequency or spectral representation•weighted sum of sine and cosine component images•each having a frequency, intensity, and orientation in the plane!filtering, for example blurring or sharpening, can be implemented by amplifying or attenuating selected frequencies•i.e. brightening or darkening selected sine or cosine componentsrelative to others, while maintaining same average over all components•attenuating high frequencies ! low-pass-filtering ! blurring•attenuating low frequencies ! high-pass filtering ! sharpening•filtering this way is slow!MTF measures preservation of frequencies by an optical system•subjective image quality depends on both sharpness and contrast17Questions?! Marc LevoySpatial resolution of display media!Example #1: Macbook Pro (laptop)•900 pixels on 8” high display•" x = 8”/ 900 pixels = 0.0089”/pixel•1/" x = 112 dpi (dots per inch)!Example #2: Kindle 2•800 pixels on 4.8” high display•1/" x = 167 dpi!Example #3: iPad•768 pixels on 5.8” high display•1/" x = 132 dpi18pitch = " xdensity = 1/" xLine printers are 300 dpi.This is why we don’t likereading on laptops.! Marc LevoySpatial frequency on the retina!Example #1: Macbook Pro viewed at d = 18”•900 pixels on 8” high display, p = 2 ! 0.0089”•retinal arc ! = 2 arctan (p / 2d) = 0.057º•spatial frequency on retina 1/" = 17.6 cycles per degree19!!assume the minimumperiod p of a sine wave is a black-white pixel pairviewing distance dQ. What is the acuity of the human visual system?! Marc LevoyHuman spatial sensitivity(Campbell-Robson Chart)20(neurovision.berkeley.edu)! Marc LevoyHuman spatial sensitivity21(horizontal axis not comparable to image on previous slide)(psych.ndsu.nodak.edu)cutoff is at about 50 cycles per degree! Marc LevoySpatial frequency on the retina!Example #1: Macbook Pro viewed at d = 18”•900 pixels on 8” high display, so p = 2 ! 0.0089”•retinal arc ! = 2 arctan (p / 2d) = 0.057º•spatial frequency on retina 1/" = 17.6 cycles per degree22!!viewing distance dnot nearly as highas human acuityassume the minimumperiod p of a sine wave is a black-white pixel pair(original is 40K ! 20K pixels, Gates Hall print is 72” ! 36”)Balboa Park, San Diego(Graham Flint)! Marc LevoySpatial frequency on the retina!Example #1: Macbook Pro viewed at d = 18”•900 pixels on 8” high display, p = 2 ! 0.0089”•retinal arc ! = 2 arctan (p / 2d) = 0.057º•spatial frequency on retina 1/" = 17.6 cycles per degree!Example #2: gigapixel photo viewed at d = 48”•20,000 pixels on 36” high print, p = 2 ! 0.0018”•spatial frequency on retina 1/" = 232 cycles per degree24!!viewing distance dassume the minimumperiod p of a sine wave is a black-white pixel pairway beyondhuman acuity! Marc LevoyHuman acuity & circle of confusion!the maximum allowable circle of confusion ( C ) in a photograph can be computed from human spatial acuity projected onto the intended display medium!Example: photographic print from viewed at 12”•max human acuity on retina 1/" # 50 cycles per degree•minimum detectable retinal arc ! # 0.02º•minimum feature size p = 2 ! 12” ! tan (! / 2) = 0.0043” (0.1mm)!assume 5” ! 7” print and Canon 5D II (5616 ! 3744 pixels)•5” / 3744 pixels = 0.0017”/pixel (0.04mm)•therefore, circle of confusion can be 2.5 pixels wide before