Single shot Multidomain Camera Roarke Horstmeyer Gary Euliss Euliss Ravi Athale Athale The MITRE Corp Marc Levoy Stanford University 2009 Background Megapixel M i l race h has resulted lt d iin an excess off pixels i l Other types of useful information that a camera can capture Multispectral Polarization Images removed due to copyright restrictions See references 4 5 Images removed due to copyright restrictions See references 1 2 3 4 5 High Dynamic Range Images removed due to copyright restrictions See reference 6 1 3 6 Can all of this be captured in a single image 2 Background y Presented with a dimensionality mismatch during multidimensional information capture 7D 3D 4D light field Temporal Spectral Polarization Spatial T Temporal l There are many different methods of encoding decoding spectral and polarimetric information One of the most direct methods is through the temporal domain Full F ll 2D spatial ti l resolution l ti preserved d att the th expense of time response Kinemacolor 7 3 Images public domain Image removed due to copyright restrictions See reference 8 Pushbroom Hyperspectral 8 Background y There are many spatial encoding decoding schemes 1 At the focal plane Images removed due to copyright restrictions Tradeoffs Fixed Integration Bayer Filter 11 Polarization 10 Assorted Pixels 9 2 Multi aperture Tradeoffs Registration Alignment Images removed due to copyright restrictions PERIODIC 12 TOMBO 13 3 Code division multiplexing Tradeoffs Computation SNR Images removed due to copyright restrictions CASSI 14 Reduced spatial resolution number of samples CTIS 15 Background y Basic idea of a light field camera s t information only Sensor integrates over all rays originating from a particular object point 5 Background y Basic idea of a light field camera Misfocus s t and u v information mixed Misfocus reintroduces u v info but it is not distinguishable from structure of light from object 6 Our Approach Encode over ray angle spatial frequency variable Basic idea of a light field camera Pinhole is imaging pupil plane providing u v angular info Used for depth Adelson and Wang 16 refocusability Ng et al 17 glare removal Raskar et al 18 7 Our Approach Place a filter array in the camera s pupil plane Now each pinhole at a particular s t creates an image of the filter array in the u v plane 8 Our Approach Color information e g is available at each spatial location in s t from each filter array y image g Spatial resolution from pinholes filter resolution from filters 9 Camera Design g A conventional setup sets P and Q P 50 mm standard lens Q 1 14mm cover glass S R F M2 W 25 32 38 83 1 51mm 1 15 3 61cm Tradeoff between synthesized image resolution and number of filters similar to u v and s t tradeoff in refocusing light field systems Pinholes optics optics limited limited R pixel size 2009 IEEE Courtesy of IEEE Used with permission Source Horstmeyer R G W Euliss R A Athale and M Levoy Flexible Multimodal Camera Using a Light Field Architecture Proceedings of IEEE ICCP 2009 Image Reconstruction Process Filtered Synthetic Images 11 Portion of raw image roughly Close up images of filter array Combine similarly filtered resembles scene apparent 3x3 square array areas from every pinhole Camera Setup p Use conventional Nikon 50mm f 1 8 lens 10Mpix 9 CCD Pinhole arrays printed on transparencies varying size pitch Filters c cutt and arranged on laser laser cut c t plastic holders holders placed inside lens over aperture stop 2009 IEEE Courtesy of IEEE Used with permission Source Horstmeyer R G W Euliss R A Athale and M Levoy Flexible Multimodal Camera Using a Light Field Architecture Proceedings of IEEE ICCP 2009 12 Experimental Results Six S filters te s R G G B 0 0 45 5 90 90 pinhole p o e r 25 5 pitch p tc 200 00 Lambertian division RGB 13 LDOP Experimental Results layout Sixteen filters color NIR ND pol 14 2009 IEEE Courtesy of IEEE Used with permission Source Horstmeyer R G W Euliss R A Athale and M Levoy Flexible Multimodal Camera Using a Light Field Architecture Proceedings of IEEE ICCP 2009 Experimental Results An example of foveation RGB RGB HDR DOP NIR A total of 12 filters used to create color image extend dynamic range and find regions g of interest Error associated with low pixel values angular diversity of compared images Courtesy of SPIE Used with permission Source Horstemeyer R R A Athale and G Euliss Light Field Architecture for Reconfigurable Multimode Imaging Proc of SPIE 7468 August 2009 doi 10 1117 12 828653 Experimental Results An example of spectral imaging Courtesy of SPIE Used with permission Source Horstemeyer R R A Athale and G Euliss Light Field Architecture for Reconfigurable Multimode Imaging Proc of SPIE 7468 August 2009 doi 10 1117 12 828653 With Spectral filter in aperture have roughly 25 spectral channels per pixel 16 Another example Changing the lighting causes a shift in spectrum 17 Courtesy of SPIE Used with permission Source Horstemeyer R R A Athale and G Euliss Light Field Architecture for Reconfigurable Multimode Imaging Proc of SPIE 7468 August 2009 doi 10 1117 12 828653 References 18 1 Arete Associates Polarization Imaging Streak Tube LIDAR PISTL http www arete com index php view ms stil 2 http bobatkins com photography tutorials polarizers html 3 Davidhazy A Stressed Plastics by Polarization http people rit edu andpph text polarization html 4 http www classzone com books earth science terc content investigations esu101 esu101page07 cfm 5 Barnard R Gray B van der Gracht J Mirotznik M Mathews S PERIODIC state of the art array imaging technology Proc 45thh annual ACM southeast regional conference 544 545 2007 6 http en wikipedia org wiki High dynamic range imaging 7 Kinemacolor http www widescreenmuseum com oldcolor kinemaco htm 8 http www ebajapan jp images remotesenshing jpg http www ebajapan jp images remotesenshing jpg 9 Generalized Mosaicing http www1 cs columbia edu CAVE projects gen mos 10 C S L Chun and F A Sadjadi Polarimetric imaging system for automatic target detection and recognition DTIC No ADA392865 2000 11 http en wikipedia org wiki Bayer filter 12 12 S K S K Nayar and S S G G Narasimhan Narasimhan Assorted pixels multi multi sampled sampled imaging with structural models models Proc ECCV ECCV 2002 13 R J Plemmons S Prasad S Matthews M Mirotznik R Barnard G Gray V P Pauca T C Torgersen J van der Gracht and G Behrmann PERIODIC Integrated Computational Array Imaging Technology Adaptive Optics OSA Technical Digest CMA1 2007 14 M E Gehm R