6 098 Digital and Computational Photography 6 882 Advanced Computational Photography Refocusing Light Fields Fr do Durand Bill Freeman MIT EECS Final projects Send your slides by noon on Thrusday Send final report Wavefront coding Is depth of field a blur Depth of field is NOT a convolution of the image The circle of confusion varies with depth There are interesting occlusion effects If you really want a convolution there is one but in 4D space more soon From Macro Photography Wavefront coding CDM Optics U of Colorado Boulder The worst title ever A New Paradigm for Imaging Systems Cathey and Dowski Appl Optics 2002 Improve depth of field using weird optics deconvolution http www cdm optics com site publications php Wavefront coding Idea deconvolution to deblur out of focus regions Convolution filter e g blur sharpen Sometimes we can cancel a convolution by another convolution Like apply sharpen after blur kind of This is called deconvolution Best studied in the Fourier domain of course Convolution multiplication of spectra Deconvolution multiplication by inverse spectrum Deconvolution Assume we know blurring kernel k f f k F F K in Fourier space Invert by F F K in Fourier space Well known problem with deconvolution Impossible to invert for where K 0 Numerically unstable when K is small Wavefront coding Idea deconvolution to deblur out of focus regions Problem 1 depth of field blur is not shift invariant Depends on depth If depth of field is not a convolution it s harder to use deconvolution Problem 2 Depth of field blur kills information Fourier transform of blurring kernel has lots of zeros Deconvolution is ill posed Wavefront coding Idea deconvolution to deblur out of focus regions Problem 1 depth of field blur is not shift invariant Problem 2 Depth of field blur kills information Solution change optical system so that Rays don t converge anymore Image blur is the same for all depth Blur spectrum does not have too many zeros How it s done Phase plate wave optics effect diffraction Pretty much bends light Will do things similar to spherical aberrations Ray version Other application Single image depth sensing Blur depends A LOT on depth Passive Ranging Through Wave Front Coding Information and Application Johnson Dowski Cathey http graphics stanford edu courses cs448a 06 winter johnson ranging optics00 pdf Single image depth sensing Important take home idea Coded imaging What the sensor records is not the image we want it s been coded kind of like in cryptography Image processing decodes it Other forms of coded imaging Tomography e g http en wikipedia org wiki Computed axial tomogr aphy Lots of cool Fourier transforms there X ray telescopes coded aperture e g http universe gsfc nasa gov cai coded intr html Ramesh s motion blur and to some extend Bayer mosaics See Berthold Horn s course Plenoptic camera refocusing Plenoptic light field cameras Lipmann 1908 Window to the world Adelson and Wang 1992 Depth computation Revisited by Ng et al for refocusing The Plenoptic Function Back to the images that surround us How to describe and capture all the possible images around us The Plenoptic function Adelson Bergen 91 http web mit edu pe rsci people adelson pu b pdfs elements91 pd f From the greek total See also http www everythin g2 com index pl node id 989303 lastnode id 1102051 Plenoptic function 3D for viewpoint 2D for ray direction 1D for wavelength 1D for time can add polarization From McMillan 95 Light fields Idea Reduce to outside the convex hull of a scene For every line in space Store RGB radiance Then rendering is just a lookup Two major publication in 1996 Light field rendering Levoy Hanrahan http graphics stanford edu papers light The Lumigraph Gortler et al Adds some depth information http cs harvard edu sjg papers lumigraph pdf How many dimensions for 3D lines 4 e g 2 for direction 2 for intersection with plane Two plane parameterization Line parameterized by intersection with 2 planes Careful there are different isotopes of such parameterization slightly different meaning of stuv Let s make life simpler 2D How many dimensions for 2D lines Only 2 e g y ax b a b Let s make life simpler 2D 2 line parameterization View View View line in Ray space Kind of cool ray point and view around point line There is a duality Back to 3D 4D From Gortler et al Cool visualization From Gortler et al View 2D plane in 4D With various resampling issues Demo light field viewer Reconstruction antialiasing depth of field Slide by Marc Levoy Aperture reconstruction So far we have talked about pinhole view Aperture reconstruction depth of field better antiliasing Slide by Marc Levoy Small aperture Image Isaksen et al Big aperture Image Isaksen et al Light field sampling Chai et al 00 Isaksen et al 00 Stewart et al 03 Light field spectrum as a function of object distance Slope inversely proportional to depth http graphics cs cmu edu projects plenoptic sampling ps projectpage htm http portal acm org citation cfm id 344779 344929 From Chai et al 2000 Light field cameras Plenoptic camera For depth extraction Adelson Wang 92 http www bcs mit edu people jyawang demos plenoptic plenoptic html Camera array Willburn et al http graphics stanford edu papers CameraArray Camera arrays http graphics stanford edu projects array MIT version Jason Yang Bullet time Time splice http www ruffy com frameset htm Robotic Camera Image Leonard McMillan Image Levoy et al Flatbed scanner camera By Jason Yang Plenoptic camera refocusing Conventional Photograph Slide by Ren Ng Light Field Photography Capture the light field inside the camera body Slide by Ren Ng Hand Held Light Field Camera Medium format digital camera Camera in use 16 megapixel sensor Microlens array Slide by Ren Ng Light Field in a Single Exposure Slide by Ren Ng Light Field in a Single Exposure Slide by Ren Ng Light Field Inside the Camera Body Slide by Ren Ng Digital Refocusing Slide by Ren Ng Digital Refocusing Slide by Ren Ng Digitally stopping down stopping down summing only the central portion of each microlens Digital Refocusing by Ray Tracing x u Lens Sensor Slide by Ren Ng Digital Refocusing by Ray Tracing x u Imaginary film Lens Sensor Slide by Ren Ng Digital Refocusing by Ray Tracing x u Imaginary film Lens Sensor Slide by Ren Ng Digital Refocusing by Ray Tracing x u Imaginary film Lens Sensor Slide by Ren Ng Digital Refocusing by Ray Tracing x u Imaginary film Lens Sensor Slide by Ren Ng Results of Band Limited Analysis Assume a light field