Sampling and pixels CS 178 Spring 2011 Begun 4 14 11 Finished 4 19 Marc Levoy Computer Science Department Stanford University Why study sampling theory 2 Why do I sometimes get moir artifacts in my images What is an antialiasing filter How many megapixels is enough How do I compute circle of confusion for depth of field Is Apple s Retina Display just hype What do MTF curves in lens reviews mean What does Photoshop do when you downsize upsize What s the difference between more pixels and more bits Marc Levoy Outline frequency representations of images filtering blurring sharpening MTF as a measure of sharpness in images resolution and human perception the spatial resolution of typical display media the acuity of the human visual system the right way to compute circle of confusion C sampling and aliasing aliasing in space and time 1D and 2D images and audio prefiltering using convolution to avoid aliasing prefiltering and sampling in photography 3 sampling versus quantization Marc Levoy Frequency representations Foley 4 a sum of sine waves each of different wavelength frequency and height amplitude can approximate arbitrary functions to adjust horizontal position phase replace with cosine waves or use a mixture of sine and cosine waves Marc Levoy Fourier analysis Fourier series any continuous integrable periodic function can be represented as an infinite series of sines and cosines f x Discrete Fourier transform DFT Xk xn 5 a0 an cos nx bn sin nx 2 n 1 N 1 n 0 xn e N 1 k 0 Xk e 2 i kn N 2 i kn N fast version is called the Fast Fourier Transform FFT k 0 N 1 n 0 N 1 where ei n x cos nx i sin nx Marc Levoy Fourier transforms of images In Matlab image double imread flower tif 255 0 fourier fftshift fft2 ifftshift image fftimage log max real fourier 0 0 20 0 complete spectrum is two images sines and cosines or real and imaginary components image 6 often called a spectrum FFT image Marc Levoy Fourier transforms of images sinusoids in intensity as a function of spatial position gives angle of sinusoid r gives spatial frequency r image 7 FFT image Marc Levoy Fourier transforms of images image 8 FFT image Marc Levoy Blurring in the Fourier domain image 9 FFT image Marc Levoy Sharpening in the Fourier domain image 10 FFT image Marc Levoy Sharpening in the Fourier domain image 11 FFT image Marc Levoy Q What does this filtering operation do image 12 FFT image Marc Levoy Blurring in x sharpening in y image 13 argh astigmatism FFT image Marc Levoy Describing sharpness in images the modulation transfer function MTF 14 imatest com the amount of each spatial frequency that can be reproduced by an optical system Marc Levoy Two different MTF curves 15 in one curve contrast stays high but drops off at a relatively low resolution in the other curve higher resolution features are preserved but contrast is low throughout Marc Levoy Sharpness versus contrast 16 imatest com Canon Marc Levoy Recap any image can be equivalently represented by its Fourier transform a k a frequency or spectral representation weighted sum of sine and cosine component images each having a frequency intensity and orientation in the plane filtering for example blurring or sharpening can be implemented by amplifying or attenuating selected frequencies i e brightening or darkening selected sine or cosine components relative to others while maintaining same average over all components attenuating high frequencies low pass filtering blurring attenuating low frequencies high pass filtering sharpening filtering this way is slow MTF measures preservation of frequencies by an optical system 17 subjective image quality depends on both sharpness and contrast Que s t ions Marc Levoy Spatial resolution of display media pitch x Example 1 Macbook Pro laptop 900 pixels on 8 high display x 8 900 pixels 0 0089 pixel 1 x 112 dpi dots per inch density 1 x Line printers are 300 dpi This is why we don t like reading on laptops Example 2 Kindle 2 800 pixels on 4 8 high display 1 x 167 dpi Example 3 iPad 768 pixels on 5 8 high display 1 x 132 dpi 18 Marc Levoy Spatial frequency on the retina assume the minimum period p of a sine wave is a black white pixel pair viewing distance d Example 1 Macbook Pro viewed at d 18 900 pixels on 8 high display p 2 0 0089 retinal arc 2 arctan p 2d 0 057 spatial frequency on retina 1 17 6 cycles per degree Q What is the acuity of the human visual system 19 Marc Levoy Human spatial sensitivity Campbell Robson Chart 20 neurovision berkeley edu Marc Levoy Human spatial sensitivity horizontal axis not comparable to image on previous slide cutoff is at about 50 cycles per degree 21 psych ndsu nodak edu Marc Levoy Spatial frequency on the retina assume the minimum period p of a sine wave is a black white pixel pair viewing distance d Example 1 Macbook Pro viewed at d 18 900 pixels on 8 high display so p 2 0 0089 retinal arc 2 arctan p 2d 0 057 spatial frequency on retina 1 17 6 cycles per degree not nearly as high as human acuity 22 Marc Levoy Graham Flint Balboa Park San Diego original is 40K 20K pixels Gates Hall print is 72 36 Spatial frequency on the retina assume the minimum period p of a sine wave is a black white pixel pair viewing distance d Example 1 Macbook Pro viewed at d 18 900 pixels on 8 high display p 2 0 0089 retinal arc 2 arctan p 2d 0 057 spatial frequency on retina 1 17 6 cycles per degree Example 2 gigapixel photo viewed at d 48 way beyond human acuity 20 000 pixels on 36 high print p 2 0 0018 spatial frequency on retina 1 232 cycles per degree 24 Marc Levoy Human acuity circle of confusion the maximum allowable circle of confusion C in a photograph can be computed from human spatial acuity projected onto the intended display medium Example photographic print from viewed at 12 max human acuity on retina 1 50 cycles per degree minimum detectable retinal arc 0 02 minimum feature size p 2 12 tan 2 0 0043 0 1mm assume 5 7 print and Canon 5D II 5616 3744 pixels 5 3744 pixels 0 0017 pixel 0 04mm therefore circle of confusion can be 2 5 pixels wide before it s blurry C 6 4 per pixel 2 5 pixels 16 25 Marc Levoy Recap spatial resolution of display media is measured by pitch distance between dots or pixels or density dots per inch effect on human observers is measured by retinal angle degrees of arc or frequency cycles per degree depends on viewing distance human spatial acuity is about 50 cycles per degree depends on contrast convert back to pitch to obtain circle of confusion for depth of field 26 Que s t ions