Computational photography MIT 6.098, 6.882 Bill Freeman, Fredo Durand Analyzing multiple imagesMultiple-exposure images by MareyOther photographs by Doc EdgertonWhat hardware was needed to make these photographs?Computational photographyComputational photographyTypical frameAverage over 50 framesMedian filter over timeVector median filter (20x20 patchs)Shapetime photographyZ-cam, made by 3DV3DV camera operation3DV camera operation3DV camera operationAnalyzing and synthesizing motionAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowOptical flow constraint equationAperture Problem and Normal FlowCombining Local ConstraintsLucas-Kanade (a good, generic motion analysis method): Integrate gradients over a patchMotion without movementA linear combination of quadrature-phase filters can advance the local phaseConvolved with an image, the image data now modulates the local amplitude. People mis-attribute the phase advance to translatMotion without movement videoMotion MagnificationComputational photographyMIT 6.098, 6.882Bill Freeman, Fredo Durand• Finish digital forensics• Analyzing multiple images– Shapetime photography– Image stacks• Analysing and synthesizing motion sequences– Motion without movement– Motion magnificationApril 20, 2006Analyzing multiple imagesBill FreemanFredo DurandMIT Computational Photography, 6.882April 20, 2006Multiple-exposure images by MareyStrobe photograph by EdgertonOther photographs by Doc EdgertonWhat hardware was needed to make these photographs?Strobe light, capacitor, thyristor…Computational photographySurely we can update those photographic techniques, adding the generality and flexibility of digital methods. Analyze and re-render the images.Computational photographyFredo and Bill describing computational photography:• Fredo: using computation to make better quality photographs—to enhance.• Bill: using computation to reveal things about the world that we otherwise couldn’t see—to reveal.How display a single-frame summary of multiple frames?Typical frameAverage over 50 framesMedian filter over timeVector median filter (20x20 patchs)2x2 vector median 2x2 vector least medianShapetime photographyJoint work with Hao Zhang, U.C. Berkeley2002Video framesMultiple-exposureLayer-By-TimeShape-TimeFrame index of each displayed pixelResulting composite imageFrame index of each displayed pixelResulting composite imageWith edge-preserving regularization“how to sew”Shape-Time composite“inside-out”Input sequenceZ-cam, made by 3DVhttp://www.3dvsystems.com3DV camera operationhttp://www.3dvsystems.com3DV camera operationhttp://www.3dvsystems.comv3DV camera operationhttp://www.3dvsystems.comRGB imageZ imageshapetime video imageZitnick et al, Siggraph 2004Show Michael Cohen slides, a selection from:http://research.microsoft.com/~cohen/FindingMagicInAnImageStack.pdfDemonstrate MSR group shot program, downloadable from http://research.microsoft.com/~cohen/orhttp://research.microsoft.com/projects/GroupShot/Analyzing and synthesizing motionBill FreemanFredo DurandMIT Computational Photography, 6.882April 20, 2006Aperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowAperture Problem and Normal FlowOptical flow constraint equation),,(),,( tyxItttvytuxI=+++δδδBrightness should stay constant as you track motion),,(),,( tyxItItIvtIutyxItyx=+++δδδ1storder Taylor series, valid for small tδ0=++tyxIvIuIConstraint equation “BCCE” - Brightness Change Constraint EquationAperture Problem and Normal Flow00=•∇=++UIIvIuItyxrThe gradient constraint:Defines a line in the (u,v) spaceuvIIIIut∇∇∇−=⊥Normal Flow:Combining Local Constraintsuv11tIUI −=•∇22tIUI −=•∇33tIUI −=•∇etc.Lucas-Kanade(a good, generic motion analysis method):Integrate gradients over a patch()∑Ω∈++=yxtyxIvyxIuyxIvuE,2),(),(),(Setting derivative w.r.t. (u, v) equal to zero gives:Assume a single velocity, u, v, for all pixels within an image patch. Find the (u, v) that minimizes the BCCE squared residual over the patch:⎟⎟⎠⎞⎜⎜⎝⎛−=⎟⎟⎠⎞⎜⎜⎝⎛⎥⎥⎦⎤⎢⎢⎣⎡∑∑∑∑∑∑tytxyyxyxxIIIIvuIIIIII22Note similarity of LHS matrix to Harris corner detector. When full-rank (corner-like), specifies a unique (u, v).Motion without movementJoint work with Ted Adelson and David Heeger, MIT1991A linear combination of quadrature-phase filters can advance the local phaseConvolved with an image, the image data now modulates the local amplitude. People mis-attribute the phase advance to translation.(Steerable filters allow synthesizing motion in arbitrary directions.)Motion without movement videohttp://www.cs.yorku.ca/~kosta/Motion_Without_Movement/Motion_Without_Movement.htmlKonstantinos G. Derpanishttp://www.cs.yorku.ca/~kosta/Motion_Without_Movement/Motion_Without_Movement.htmlMotion Magnification(go to other