Copyright © 2006 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail [email protected]. © 2006 ACM 0730-0301/06/0700- $5.00 0787Removing Camera Shake from a Single PhotographRob Fergus1Barun Singh1Aaron Hertzmann2Sam T. Roweis2William T. Freeman11MIT CSAIL2University of TorontoFigure 1: Left: An image spoiled by camera shake. Middle: result from Photoshop “unsharp mask”. Right: result from our algorithm.AbstractCamera shake during exposure leads to objectionable image blurand ruins many photographs. Conventional blind deconvolutionmethods typically assume frequency-domain constraints on images,or overly simplified parametric forms for the motion path duringcamera shake. Real camera motions can follow convoluted paths,and a spatial domain prior can better maintain visually salient im-age characteristics. We introduce a method to remove the effects ofcamera shake from seriously blurred images. The method assumesa uniform camera blur over the image and negligible in-plane cam-era rotation. In order to estimate the blur from the camera shake,the user must specify an image region without saturation effects.We show results for a variety of digital photographs taken frompersonal photo collections.CR Categories: I.4.3 [Image Processing and Computer Vision]:Enhancement, G.3 [Artificial Intelligence]: LearningKeywords: camera shake, blind image deconvolution, variationallearning, natural image statistics1 IntroductionCamera shake, in which an unsteady camera causes blurry pho-tographs, is a chronic problem for photographers. The explosion ofconsumer digital photography has made camera shake very promi-nent, particularly with the popularity of small, high-resolution cam-eras whose light weight can make them difficult to hold sufficientlysteady. Many photographs capture ephemeral moments that cannotbe recaptured under controlled conditions or repeated with differ-ent camera settings — if camera shake occurs in the image for anyreason, then that moment is “lost”.Shake can be mitigated by using faster exposures, but that can leadto other problems such as sensor noise or a smaller-than-desireddepth-of-field. A tripod, or other specialized hardware, can elim-inate camera shake, but these are bulky and most consumer pho-tographs are taken with a conventional, handheld camera. Usersmay avoid the use of flash due to the unnatural tonescales that re-sult. In our experience, many of the otherwise favorite photographsof amateur photographers are spoiled by camera shake. A methodto remove that motion blur from a captured photograph would bean important asset for digital photography.Camera shake can be modeled as a blur kernel, describing the cam-era motion during exposure, convolved with the image intensities.Removing the unknown camera shake is thus a form of blind imagedeconvolution, which is a problem with a long history in the im-age and signal processing literature. In the most basic formulation,the problem is underconstrained: there are simply more unknowns(the original image and the blur kernel) than measurements (theobserved image). Hence, all practical solutions must make strongprior assumptions about the blur kernel, about the image to be re-covered, or both. Traditional signal processing formulations of theproblem usually make only very general assumptions in the formof frequency-domain power laws; the resulting algorithms can typi-cally handle only very small blurs and not the complicated blur ker-nels often associated with camera shake. Furthermore, algorithmsexploiting image priors specified in the frequency domain may notpreserve important spatial-domain structures such as edges.This paper introduces a new technique for removing the effects ofunknown camera shake from an image. This advance results fromtwo key improvements over previous work. First, we exploit recentresearch in natural image statistics, which shows that photographsof natural scenes typically obey very specific distributions of im-age gradients. Second, we build on work by Miskin and MacKay[2000], adopting a Bayesian approach that takes into account uncer-tainties in the unknowns, allowing us to find the blur kernel impliedby a distribution of probable images. Given this kernel, the imageis then reconstructed using a standard deconvolution algorithm, al-though we believe there is room for substantial improvement in thisreconstruction phase.We assume that all image blur can be described as a single convolu-tion; i.e., there is no significant parallax, any image-plane rotationof the camera is small, and no parts of the scene are moving rel-ative to one another during the exposure. Our approach currentlyrequires a small amount of user input.Our reconstructions do contain artifacts, particularly when the787above assumptions are violated; however, they may be acceptable toconsumers in some cases, and a professional designer could touch-up the results. In contrast, the original images are typically unus-able, beyond touching-up — in effect our method can help “rescue”shots that would have otherwise been completely lost.2 Related WorkThe task of deblurring an image is image deconvolution; if the blurkernel is not known, then the problem is said to be “blind”. Fora survey on the extensive literature in this area, see [Kundur andHatzinakos 1996]. Existing blind deconvolution methods typicallyassume that the blur kernel has a simple parametric form, such asa Gaussian or low-frequency Fourier components. However, as il-lustrated by our examples, the blur kernels induced during camerashake do not have simple forms, and often contain very sharp edges.Similar low-frequency assumptions are typically made for the inputimage, e.g., applying a quadratic regularization. Such assumptionscan prevent high frequencies (such as edges) from appearing in thereconstruction. Caron et al. [2002] assume a power-law distributionon the