Background Subtraction and Matting“Smoke” (1996), the “photo album scene”Moving in TimeImage StackExampleGetting the right pixelsWebcamsInput VideoAverage ImageFigure-centric RepresentationContext-based Image CorrectionAverage/Median ImageBackground SubtractionCrowd Synthesis (with Pooja Nath)Slide 15How does Superman fly?Image CompositingCompositing ProcedureCompositing: Two IssuesSolution: alpha channelMultiple Alpha Blending“Pulling a Matte”Blue ScreenBlue Screen mattingThe UltimatteBlue screen for superman?Semi-transparent mattesMatting Problem: Mathematical DefinitionWhy is general matting hard?Solution #1: No Blue!Solution #2: Gray or FleshTriangulation Matting (Smith & Blinn)The AlgorithmTriangulation Matting ExamplesMore ExamplesMore examplesRemoving Shadows (Weiss, 2001)Averaging DerivativesRecovering ShadowsCompositing with ShadowsBackground Subtraction and Matting15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005© Yuri Bonder“Smoke” (1996), the “photo album scene”Moving in TimeMoving only in time, while not moving in space, has many advantages•No need to find correspondences•Can look at how each ray changes over time•In science, always good to change just one variable at a timeThis approach has always interested artists (e.g. Monet)Modern surveillance video camera is a great source of information•There are now many such WebCams now, some running for several years!Image StackAs can look at video data as a spatio-temporal volume•If camera is stationary, each line through time corresponds to a single ray in space•We can look at how each ray behaves •What are interesting things to ask?t0255timeExampleGetting the right pixelsAverage imageMedian ImageWebcamsLots of cool potential projects •PCA, weather morphing, weather extrapolation, etc.Input VideoAverage ImageWhat is happening?Figure-centric RepresentationContext-based Image CorrectionInput sequence3 closestframesmedian imagesAverage/Median ImageWhat can we do with this?Background Subtraction--==Crowd Synthesis (with Pooja Nath)1. Do background subtraction in each frame2. Find and record “blobs”3. For synthesis, randomly sample the blobs, taking care not to overlap themBackground SubtractionA largely unsolved problem…EstimatedbackgroundDifference Image ThresholdedForeground on blueOne videoframeHow does Superman fly?Super-human powers?ORImage Matting and Compositing?Image CompositingCompositing Procedure1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)Composite by David Dewey2. Blend them into the composite (in the right order)Compositing: Two IssuesSemi-transparent objectsPixels too largeSolution: alpha channelAdd one more channel:•Image(R,G,B,alpha) Sprite!Encodes transparency (or pixel coverage):•Alpha = 1: opaque object (complete coverage)•Alpha = 0: transparent object (no coverage)•0<Alpha<1: semi-transparent (partial coverage)Example: alpha = 0.7Partial coverage or semi-transparencyMultiple Alpha BlendingSo far we assumed that one image (background) is opaque. If blending semi-transparent sprites (the “A over B” operation):Icomp = aIa + (1-a)bIbcomp = a + (1-a)bNote: sometimes alpha is premultiplied: im(R,G,B,):Icomp = Ia + (1-a)Ib(same for alpha!)“Pulling a Matte”Problem Definition:•The separation of an image C into–A foreground object image Co, –a background image Cb, –and an alpha matte •Co and can then be used to composite the foreground object into a different imageHard problem•Even if alpha is binary, this is hard to do automatically (background subtraction problem)•For movies/TV, manual segmentation of each frame is infeasible •Need to make a simplifying assumption…Blue ScreenBlue Screen mattingMost common form of matting in TV studios & moviesPetros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement.A form of background subtraction:•Need a known background•Compute alpha as SSD(C,Cb) > threshold–Or use Vlahos’ formula: = 1-p1(B-p2G)•Hope that foreground object doesn’t look like background –no blue ties!•Why blue?•Why uniform?The Ultimattep1 and p2Blue screen for superman?Semi-transparent mattesWhat we really want is to obtain a true alpha matte, which involves semi-transparency•Alpha between 0 and 1Matting Problem: Mathematical DefinitionWhy is general matting hard?Solution #1: No Blue!Solution #2: Gray or FleshTriangulation Matting (Smith & Blinn)How many equations?How many unknowns?Does the background need to constant color?The AlgorithmTriangulation Matting ExamplesMore ExamplesMore examplesRemoving Shadows (Weiss, 2001)How does one detect (subtract away) shadows?Averaging DerivativesRecovering ShadowsCompositing with
View Full Document
Unlocking...