Single-shot Multidomain Camera Roarke HorstmeyerGary Euliss Ravi Athale The MITRE Corp Gary Euliss, Ravi Athale, The MITRE Corp Marc Levoy, Stanford University 2009tBackground M i l h l d i f i l■ Megapixel race has resulted in an excess of pixels ■ Other types of useful information that a camera cancapture: Multispectral Polarization Images removed due to copyright restrictions. See references [4], [5]. High Dynamic Range [4,5]Images removed due to copyright restrictions. See references [1], [2], [3]. Images removed due to copyright restrictions. [1-3] [6] See reference [6]. 2 [6] ■ Can all of this be captured in a single image?Background y Presented with a dimensionality mismatch during y Presented with a dimensionality mismatch during multidimensional information capture: 7D 3D7D 4D light field Temporal Spectral 3D Spatial T l Spectral Polarization Temporal ■ There are many different methods of encoding/decoding spectral andpolarimetric informationand polarimetric information ■ One of the most direct methods is through the temporal domain Fll 2D ti l lti d t th ■ Full 2D spatial resolution preserved at the expense of time response 3 Kinemacolor [7] Images: public domain Pushbroom Hyperspectral [8] Image removed due to copyright restrictions. See reference [8].Background y There are many spatial encoding/decoding schemes: 1. At the focal plane: 1. At the focal plane: Images removed due to copyright restrictions. Tradeoffs: Fixed, Integration Bayer Filter [11] Polarization [10] Assorted Pixels [9] 2. Multi-aperture : Tradeoffs: Registration, Alignment Images removed due to copyright restrictions. PERIODIC [12] TOMBO [13] 3. Code-division multiplexing: Tradeoffs: Computation, SNR CASSI [14] CTIS [15] Images removed due to copyright restrictions. ■ Reduced spatial resolution number of samplesBackground y Basic idea of a light field camera: (s,t) information only ■ Sensor integrates over all rays originating from a particular 5 ■ Sensor integrates over all rays originating from a particular object pointBackground y Basic idea of a light field camera: Misfocus!Misfocus! (s,t) and (u,v)information mixed ■ Misfocus reintroduces (u,v) info, but it is not 6 ■ Misfocus reintroduces (u,v) info, but it is not distinguishable from structure of light from objectOur Approach: ■ Encode over ray angle (spatial frequency variable) ■ Basic idea of a light field camera: ■ Pinhole is imaging pupil plane, providing (u,v) angular info ■ Used for depth (Adelson and Wang [16]), refocusability (Ng 7 ■ Used for depth (Adelson and Wang [16]), refocusability (Ng et al. [17]) glare removal (Raskar et al. [18])Our Approach ■ Place a filter array in the camera’s pupil plane: ■ Now, each pinhole at a particular (s,t) creates an image of thefilter array in the (u,v) plane 8 filter array in the (u,v) planeOur Approach ■ Color information (e.g.) is available at each spatial location in (s,t) from each filter array image 9 ( ) y g ■ Spatial resolution from # pinholes, filter resolution from # filtersCamera Designg■ A conventional setup sets Pand Q: P' ≈ 50 mm (standard lens) Q = 1.14mm (cover glass) S R F M2 W 25.32µ 38.83µ 1.51mm 1.15 3.61cm ■ Tradeoff between synthesized image resolution and number of filters similar to (u,v) and (s,t) tradeoff in refocusing light field systems ■ Pinholes = optics-limited (R > pixel size) ■ Pinholes optics limited (R pixel size) © 2009 IEEE. Courtesy of IEEE. Used with permission. Source: Horstmeyer, R., G. W. Euliss, R. A. Athale, and M. Levoy. "Flexible Multimodal Camera Using a Light Field Architecture." Proceedings of IEEE ICCP, 2009.Image Reconstruction ProcessImage Reconstruction Process Filtered Synthetic ImagesFiltered Synthetic Images (Portion of) raw image roughly resembles scene Close up, images of filter array apparent (3x3 square array) Combine similarly filtered areas from every pinhole 11Camera Setupp ■ Use conventional Nikon 50mm f/1.8 lens, 10Mpix 9µ CCD ■ Pinhole arrays printed on transparencies, varying size + pitch Filters c t and arranged on laser c t plastic holders placed inside ■ Filters cut and arranged on laser-cut plastic holders, placed inside lens over aperture stop 1© 2009 IEEE. Courtesy of IEEE. Used with permission. Source: Horstmeyer, R., G. W. Euliss, R. A. Athale, and M. Levoy. "Flexible Multimodal Camera Using a Light Field Architecture." Proceedings of IEEE ICCP, 2009.2S te sG 0 5 90 o e tcExperimental Results ■ Six filters: R, G, B, 0º, 45º, 90º (pinhole r = 25µ, pitch = 200µ), , , , , (p 5µ, p 00µ) Lambertian division LDOPRGB 13Experimental Results layout ■ Sixteen filters: colorcolor NIR pol. ND 14 © 2009 IEEE. Courtesy of IEEE. Used with permission. Source: Horstmeyer, R., G. W. Euliss, R. A. Athale, and M. Levoy. "Flexible Multimodal Camera Using a Light Field Architecture." Proceedings of IEEE ICCP, 2009.Experimental Results ■ An example of foveationRGB RGB HDR DOP + NIR ■ A total of 12 filters used to create color image, extend dynamic range, and find “regions of interest”g■ Error associated with low pixel values, angular diversity of compared images Courtesy of SPIE. Used with permission. Source: Horstemeyer, R., R. A. Athale, and G. Euliss. "Light Field Architecture for Reconfigurable Multimode Imaging." Proc. of SPIE 7468, August 2009. doi: 10.1117/12.828653Experimental Results ■ An example of spectral imaging •With Spectral filter in aperture, have roughly 25 spectral channels per pixel 16 Courtesy of SPIE. Used with permission. Source: Horstemeyer, R., R. A. Athale, and G. Euliss. "Light Field Architecture for Reconfigurable Multimode Imaging." Proc. of SPIE 7468, August 2009. doi: 10.1117/12.828653■ Another example:example: Changing the lighting causes a shift in spectrum:spectrum: 17 Courtesy of SPIE. Used with permission. Source: Horstemeyer, R., R. A. Athale, and G. Euliss. "Light Field Architecture for Reconfigurable Multimode Imaging." Proc. of SPIE 7468, August 2009. doi: 10.1117/12.828653References 1. Arete Associates, “Polarization Imaging Streak Tube LIDAR (PISTL),” http://www.arete.com/index.php?view=ms_stil 2. http://bobatkins.com/photography/tutorials/polarizers.html 3. Davidhazy, A. , “Stressed Plastics by Polarization”, http://people.rit.edu/andpph/text-polarization.html 4. http://www.classzone.com/books/earth_science/terc/content/investigations/esu101/esu101page07.cfm