Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 65Image and Depth from a Conventional Camera with a Coded ApertureAnat Levin, Rob Fergus, Frédo Durand, William FreemanMIT CSAILSingle input image:Output #1: Depth mapOutput #2: All-focused imageLensCamera sensorPoint spread functionImage of a point light sourceLens and defocusFocal planeLens’ apertureLensObjectCamera sensorPoint spread functionImage of a defocused point light sourceLens and defocusLens’ aperture Focal planeLensCamera sensorPoint spread functionImage of a defocused point light sourceLens and defocusObjectLens’ aperture Focal planeLensCamera sensorPoint spread functionImage of a defocused point light sourceLens’ aperture Lens and defocusObjectFocal planeDepth and defocusDepth from defocus:Infer depth by analyzing local scale of defocus blurOut of focusIn focusill posedChallenges• Hard to discriminate a smooth scene from defocus blur • Hard to undo defocus blur Input Ringing with conventional deblurring algorithmOut of focus?Key contributions• Exploit prior on natural images - Improve deconvolution - Improve depth discrimination • Coded aperture (mask inside lens) - make defocus patterns different from natural images and easier to discriminateNatural UnnaturalRelated Work•Active methodse.g. Nayar et al. , Zhang and Nayar•Depth from (de)focus e.g. Pentland, Chaudhuri, Favaro et al. • Plenoptic/ light field cameras e.g. Adelson and Wang, Ng et al.• Wave front coding e.g. Cathey & Dowski•Blind Deconvolution e.g. Kundur and Hatzinakos , Fergus et al, LevinNever recover both depth AND full resolution image from a single imageDefocus as local convolutionInput defocused imageCalibrated blur kernels at different depthsDefocus as local convolutionxfykInput defocused imageLocal sub-windowCalibrated blur kernels at depthSharp sub-windowkxfykDepth k=3:Depth k=1:xfykxfykDepth k=2:Overview Correct scale Smaller scale Larger scale   Try deconvolving local input windows with different scaled filters:Somehow: select best scale.??????Challenges•Hard to identify correct scale:•Hard to deconvolve even when kernel is known Input Ringing with the traditional Richardson-Lucy deconvolution algorithm ??Correct scale Smaller scale ?Larger scale   •Hard to deconvolve even when kernel is known?yxf =Deconvolution is ill posedLinear, but rank deficientDeconvolution is ill posed?==?Solution 1:Solution 2:yxf Linear, but rank deficientImage Restoration Techniques•Frequency domain–Wiener filter, deconvwnr()–Direct inverse method•Spatial domain–Richardson-Lucy method, deconvlucy()–MAP estimator, iterative method–Slow convergence for high freq. (needs priors)?=XKernel BlurClearIdea 1: Natural images priorImagegradientNatural UnnaturalWhat makes images special?Histogram of image gradientsCharacteristic distribution with heavy tailsNatural Image StatisticsBlurry image has different statisticsHistogram of image gradientsDeconvolution with prior2|| minarg yxfx _iix )(2+_+2??Convolution error Derivatives priorHigh Low Equal convolution errorComparing deconvolution algorithmsInputRichardson-Lucy(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/Gaussian prior2)( xx “spread” gradientsSparse prior“localizes” gradients8.0)( xx Comparing deconvolution algorithmsInputRichardson-Lucy(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/Gaussian prior“spread” gradientsSparse prior“localizes” gradients2)( xx 8.0)( xx  ??Correct scale Smaller scale ?Larger scale   Try deconvolving local input windows with different scaled filters:Recall: OverviewChallenge: smaller scale not so different than correct Somehow: select best scale.Idea 2: Coded Aperture• Mask (code) in aperture plane - make defocus patterns different from natural images and easier to discriminateConventional apertureOur coded apertureLensCamera sensorPoint spread functionObjectSolution: lens with occluderFocal planeSolution: lens with occluderLens with coded apertureCamera sensorPoint spread functionImage of a defocused point light sourceAperture patternObjectFocal planeLens with coded apertureCamera sensorPoint spread functionImage of a defocused point light sourceAperture patternSolution: lens with occluderObjectFocal planeLens with coded apertureCamera sensorPoint spread functionImage of a defocused point light sourceAperture patternSolution: lens with occluderObjectFocal planeLens with coded apertureCamera sensorPoint spread functionImage of a defocused point light sourceAperture patternSolution: lens with occluderObjectFocal planeLens with coded apertureCamera sensorPoint spread functionImage of a defocused point light sourceAperture patternSolution: lens with occluderObjectFocal planeWhy coded?Conventional Coded Coded aperture- reduce uncertainty in scale identificationCorrect scale Smaller scale Larger scaleConvolution- frequency domain representation==0Sharp ImageSharp ImageSpatial convolution frequency multiplication0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0FrequencyspectrumFilter, 1st scaleFilter, 2nd scaleOutput spectrum has zeros where filter spectrum has zeros2nd observed image1st observed image??Coded aperture: Scale estimation and division by zero==0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0FrequencyspectrumEstimated image spatial ringingDivision by zeroEstimated imageFilter, wrong scaleFilter, correct scaleObserved image??=0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0Frequencyspectrum0FrequencyspectrumEstimated imageEstimated