6.098 Digital and Computational Photography 6.882 Advanced Computational PhotographyHDR imaging and the Bilateral FilterBill FreemanFrédo DurandMIT - EECSAnnouncement• Why Matting Matters • Rick Szeliski• Monday at 2pm in Kiva/Patil• Image matting (e.g., blue-screen matting) has been a mainstay of Hollywood and the visual effects industry for decades, but its relevance to computer vision is not yet fully appreciated. In this talk, I argue that the mixing of pixel color values at the boundaries of objects (or even albedo changes) if a fundamental process that must be correctly modeled to make meaningful signal-level inferences about the visual world, as well as to support high-quality imaging transformations such as de-noising and de-blurring. Starting with Ted Adelson et al.'s seminal work on layered motion models, I review early stereo matching algorithms with transparency and matting (with Polina Golland), work on layered representations with matting (with Simon Baker and Anandan), through Larry Zitnick's 2-layer representation for 3D video. I then present our recent work (with Ce Liu et al.) on image de-noising using a segmented description of the image and Eric Bennett's et al.'s work on multi-image de-mosaicing, again using a local two-color model.ReferencesRefshttp://www.hdrsoft.com/resources/dri.htmlhttp://www.clarkvision.com/imagedetail/dynamicrange2/http://www.debevec.org/HDRI2004/http://www.luminous-landscape.com/tutorials/hdr.shtmlhttp://www.anyhere.com/gward/hdrenc/http://www.debevec.org/IBL2001/NOTES/42-gward-cic98.pdfhttp://www.openexr.com/http://gl.ict.usc.edu/HDRShop/http://www.dpreview.com/learn/?/Glossary/Digital_Imaging/Dynamic_Range_01.htmhttp://www.normankoren.com/digital_tonality.htmlhttp://www.anyhere.com/http://www.cybergrain.com/tech/hdr/Contrast reduction• Match limited contrast of the medium• Preserve details10-610610-6106Real worldPictureLow contrastHigh dynamic rangeHistogram• See http://www.luminous-landscape.com/tutorials/understanding-series/understanding-histograms.shtmlhttp://www.luminous-landscape.com/tutorials/expose-right.shtml• Horizontal axis is pixel value• Vertical axis is number of pixelsHighlights• Clipped pixels (value >255) • Pro and semi-pro digital cameras allow you to make them blink.Questions?Multiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeMultiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeMultiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeMultiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeMultiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeMultiple exposure photography• Sequentially measure all segments of the range10-610610-6106Real worldPictureLow contrastHigh dynamic rangeHow do we vary exposure? • Options:– Shutter speed– Aperture –ISO – Neutral density filterSlide inspired by Siggraph 2005 course on HDRTradeoffs• Shutter speed– Range: ~30 sec to 1/4000sec (6 orders of magnitude)– Pros: reliable, linear– Cons: sometimes noise for long exposure• Aperture– Range: ~f/1.4 to f/22 (2.5 orders of magnitude)– Cons: changes depth of field– Useful when desperate• ISO– Range: ~100 to 1600 (1.5 orders of magnitude)– Cons: noise– Useful when desperate• Neutral density filter– Range: up to 4 densities (4 orders of magnitude) & can be stacked– Cons: not perfectly neutral (color shift), not very precise, need to touch camera (shake)– Pros: works with strobe/flash, good complement when desperateSlide after Siggraph 2005 course on HDRQuestions?HDR image using multiple exposure• Given N photos at different exposure• Recover a HDR color for each pixelIf we know the response curve• Just look up the inverse of the response curve• But how do we get the curve? Pixel valuescene valueCalibrating the response curve• Two basic solutions– Vary scene luminance and see pixel values• Assumes we control and know scene luminance– Vary exposure and see pixel value for one scene luminance • But note that we can usually not vary exposure more finely than by 1/3 stop• Best of both: – Vary exposure– Exploit the large number of pixels•3••33•1••11•2••22ΔΔtt==1/100 sec1/100 sec•3••33•1••11•2••22ΔΔtt==1 sec1 sec•3••33•1••11•2••22ΔΔtt==1/1000 sec1/1000 sec•3••33•1••11•2••22ΔΔtt==10 sec10 secImage seriesImage seriesImage series•3••33•1••11•2••22ΔΔtt==1/10 sec1/10 secExposure = Radiance ×Δtlog Exposure = log Radiance + log ΔtPixel Value Z = f(Exposure)Slide adapted from Alyosha Efros who borrowed it from Paul DebevecΔ t don't really correspond to pictures. Oh well. The Algorithmlog ExposureAssuming unit radiancefor each pixelAfter adjusting radiances to obtain a smooth response curvePixel value333111222log ExposurePixel valueSlide stolen from Alyosha Efros who stole it from Paul DebevecResponse curve• Exposure is unknown, fit to find a smooth curve• Let g(z) be the discrete inverse response function• For each pixel site i in each image j, want:• Solve the overdetermined linear system:fitting termsmoothness term[]∑∑∑===′′+−Δ+maxminZZzNiPjijjizgZgtRadiance2112)()(loglogλ)(loglogijjiZgtRadiance=Δ+Slide stolen from Alyosha Efros who stole it from Paul DebevecThe MathSlide stolen from Alyosha Efros who stole it from Paul DebevecMatlab codefunction [g,lE]=gsolve(Z,B,l,w)n = 256;A = zeros(size(Z,1)*size(Z,2)+n+1,n+size(Z,1));b = zeros(size(A,1),1);k = 1; %% Include the data-fitting equationsfor i=1:size(Z,1)for j=1:size(Z,2)wij = w(Z(i,j)+1);A(k,Z(i,j)+1) = wij; A(k,n+i) = -wij; b(k,1) = wij * B(i,j);k=k+1;endendA(k,129) = 1; %% Fix the curve by setting its middle value to 0k=k+1;for i=1:n-2 %% Include the smoothness equationsA(k,i)=l*w(i+1); A(k,i+1)=-2*l*w(i+1); A(k,i+2)=l*w(i+1);k=k+1;endx = A\b; %% Solve the system using SVDg = x(1:n);lE = x(n+1:size(x,1));Recovered response curveRecovered response curvelog Exposurelog ExposurePixel valuePixel valueKodak DCS4601/30 to 30 secSlide