6.098 Digital and Computational Photography 6.882 Advanced Computational PhotographyMatting & CompositingBill FreemanFrédo DurandMIT - EECSHow does Superman fly?• Super-human powers?• OR• Image Matting and Compositing?Slide from Alyosha EfrosMotivation: compositingCombining multiple images. Typically, paste a foreground object onto a new backgroundMotivation: compositingCombining multiple images. Typically, paste a foreground object onto a new background• Movie special effect• Multi-pass CG• Combining CG & film• Photo retouching– Change background– Fake depth of field– Page layout: extract objects, magazine coversFrom Porter & Duff1984From CinefexFrom the Art & Science of Digital CompositingFrom the Art & Science of Digital CompositingSlide from Alyosha EfrosPage layout, magazine coversPhoto editing• Edit the background independently from foregroundPhoto editing• Edit the background independently from foregroundTechnical Issues• Compositing– How exactly do we handle transparency?• Smart selection – Facilitate the selection of an object• Matte extraction– Resolve sub-pixel accuracy, estimate transparency• Smart pasting– Don't be smart with copy, be smart with paste– See homework (pyramid splining)– See also in a couple weeks (gradient manipulation)• Extension to video– Where life is always harderAlpha• α: 1 means opaque, 0 means transparent• 32-bit images: R, G, B, αFrom the Art & Science of Digital CompositingWhy fractional alpha?• Motion blur, small features (hair) cause partial occlusionFrom Digital DomainWith binary alphaFrom Digital DomainWith fractional alphaFrom Digital DomainPhotoshop layer masksWhat does R, G, B, α represent?– α: 1 means opaque, 0 means transparent• But what about R, G, and B?Two possible answers:• Premultipliedthe color of the object is R/α, G/α, B/α• or notthe color of the object is R, G, B, and these values need to be multiplied by α for compositingPre-multiplied alpha• (R, G, B, α) means that the real object color is (R/α , G/α, B/α) and transparency is α.• Motivated by supersampling for antialiasing in CG• If I combine multiple subpixels, the same operations apply to the four channels– In particular if I transform the image for scale/rotatesupersampled pixelresampled (averaged value)1/n Σ Ri, 1/n Σ Gi, 1/n Σ Bi, 1/n Σαi{Ri,Gi, Bi, αi }The compositing equationPorter & Duff Siggraph 1984• Given Foreground FAand Background FBimages• For premultiplied alpha:Output = FA+(1-αA) FB• For non-premultiplied:Output = α FA+(1-αA) FBComposing Two ElementsBackgroundForeground Traveling MatteHoldout Matte**++==Slide from Pat HanrahanOptical PrintingFrom: “Special Optical Effects,”Zoran PerisicFrom: “Industrial Light and Magic,”Thomas Smith (p. 181)Slide from Pat HanrahanFrom: “Industrial Light and Magic,”Thomas SmithLimitations of alpha• Hard to represent stainglasses– It focuses on subpixel occlusion (0 or 1)• Does not model more complex optical effects – e.g. magnifying glassQuestions?From CinefexCompositing• Non premultiplied version:Given the foreground color F=(RF, GF, BF), the background color (RB, GB, BB) and α for each pixel• The over operation is: C=α F+(1-α)B– (in the premultiplied case, omit the first α)αFCBMatting problem• Inverse problem: Assume an image is the over composite of a foreground and a background• Given an image color C, find F, B and α so thatC=α F+(1-α)Bα?F?CB?Matting ambiguity• C=α F+(1-α)B• How many unknowns, how many equations?α?F?CB?Matting ambiguity• C=α F+(1-α)B• 7 unknowns: α and triplets for F and B• 3 equations, one per color channelCMatting ambiguity• C=α F+(1-α)B• 7 unknowns: α and triplets for F and B• 3 equations, one per color channel• With known background (e.g. blue/green screen): 4 unknowns, 3 equationsCBFQuestions?From CinefexTraditional blue screen matting• Invented by Petro Vlahos(Technical Academy Award 1995)• Recently formalized by Smith & Blinn• Initially for film, then video, then digital• Assume that the foreground has no blue• Note that computation of α has to be analog, needs to be simple enoughFrom CinefexTraditional blue screen matting• Assume that blue b and green g channels of the foreground respect b· a2 g for a2typically between 0.5 and 1.5 • α = 1 - a1(b - a2g)– clamped to 0 and 1–a1and a2are user parameters–Note that α = 1 where assumption holdsb-a2g=0b-a2g=1/a1Traditional blue screen matting• Assume that blue and green channels of the foreground respect b· a2 g for a2typically between 0.5 and 1.5 • α = 1 - a1(b - a2g)– clamped to 0 and 1–where a1and a2are user parameters–Note that α = 1 where assumption holds• Lots of refinements (see Smith & Blinn's paper)Blue/Green screen matting issues• Color limitation– Annoying for blue-eyed peopleÎ adapt screen color (in particular green)• Blue/Green spilling– The background illuminates the foreground, blue/green at silhouettes– Modify blue/green channel, e.g. set to min (b, a2g)• Shadows– How to extract shadows cast on backgroundBlue/Green screen matting issuesFrom the Art & Science of Digital Compositing• http://www.digitalgreenscreen.com/figure3.htmlExtension: Chroma key• Blue/Green screen matting exploits color channels• Chroma key can use an arbitrary background color• See e.g. – http://www.cs.utah.edu/~michael/chroma/– Keith Jack, "Video Demystified", Independent Pub Group (Computer), 1996Questions?Hint: PSet 2 solution is in next slides• Hint 2: start problem set 2 early!Recall: Matting ambiguity• C=α F+(1-α)B• 7 unknowns: α and triplets for F and B• 3 equations, one per color channelCNatural matting [Ruzon & Tomasi 2000, Chuang et al. 2001]• Given an input image with arbitrary background• The user specifies a coarse Trimap(known Foreground, known background and unknown region)• Goal: Estimate F, B, alpha in the unknown region– We don’t care about B, but it’s a byproduct/unkownNow, what tool do we know to estimate something, taking into account all sorts of known probabilities?images from Chuang et alWho's afraid of Bayes?Bayes theoremP(x|y) = P(y|x) P(x) / P(y)The parameters you want to estimateWhat you observe Prior probabilityLikelihood functionConstant w.r.t. parameters x.Matting and Bayes• What do we observe?P(x|y) = P(y|x) P(x) / P(y)The parameters you want to estimateWhat you observe Prior