This preview shows page 1-2-3-4-5-6-7-8-9-65-66-67-68-69-70-71-72-73-74-131-132-133-134-135-136-137-138-139 out of 139 pages.
Data-driven Methods: Texture15-463: Computational PhotographyAlexei Efros, CMU, Fall 2007© Darren Green (www.darrensworld.com)Texture• Texture depicts spatially repeating patterns• Many natural phenomena are texturesradishes rocks yogurtTexture Synthesis• Goal of Texture Synthesis: create new samples of a given texture• Many applications: virtual environments, hole-filling, texturing surfacesThe Challenge• Need to model the whole spectrum: from repeated to stochastic texturerepeatedstochasticBoth?Efros & Leung Algorithm• Assuming Markov property, compute P(p|N(p))– Building explicit probability tables infeasible ppSynthesizing a pixelnon-parametricsamplingInput image– Instead, we search the input image for all similar neighborhoods — that’s our pdf for p– To sample from this pdf, just pick one match at randomSome Details• Growing is in “onion skin” order– Within each “layer”, pixels with most neighbors are synthesized first– If no close match can be found, the pixel is not synthesized until the end• Using Gaussian-weighted SSD is very important– to make sure the new pixel agrees with its closest neighbors– Approximates reduction to a smaller neighborhood window if data is too sparseNeighborhood WindowinputVarying Window SizeIncreasing window sizeSynthesis Resultsfrench canvas rafia weaveMore Resultswhite bread brick wallHomage to ShannonHole FillingExtrapolationSummary• The Efros & Leung algorithm– Very simple– Surprisingly good results– Synthesis is easier than analysis!– …but very slowppImage Quilting [Efros & Freeman]• Observation: neighbor pixels are highly correlatedInput imagenon-parametricsamplingBBIdea:Idea:unit of synthesis = blockunit of synthesis = block• Exactly the same but now we want P(B|N(B))• Much faster: synthesize all pixels in a block at once• Not the same as multi-scale!Synthesizing a blockInput textureB1 B2Random placement of blocks blockB1B2Neighboring blocksconstrained by overlapB1 B2Minimal errorboundary cutmin. error boundaryMinimal error boundaryoverlapping blocks vertical boundary__==22overlap errorOur Philosophy• The “Corrupt Professor’s Algorithm”:– Plagiarize as much of the source image as you can– Then try to cover up the evidence• Rationale: – Texture blocks are by definition correct samples of texture so problem only connecting them togetherFailures(ChernobylHarvest)input imagePortilla & SimoncelliWei & Levoy Our algorithmXu, Guo & ShumPortilla & SimoncelliWei & Levoy Our algorithmXu, Guo & Shuminput imagePortilla & SimoncelliWei & Levoy Our algorithminput imageXu, Guo & ShumPolitical Texture Synthesis!Fill Order• In what order should we fill the pixels?Fill Order• In what order should we fill the pixels?– choose pixels that have more neighbors filled– choose pixels that are continuations of lines/curves/edgesCriminisi, Perez, and Toyama. “Object Removal by Exemplar-based Inpainting,” Proc. CVPR, 2003.Exemplar-based Inpainting demohttp://research.microsoft.com/vision/cambridge/i3l/patchworks.htm++==Application: Texture Transfer• Try to explain one object with bits and pieces of another object:Texture Transfer ConstraintTexture sample• Take the texture from one image and “paint” it onto another objectTexture TransferSame as texture synthesis, except an additional constraint:1. Consistency of texture 2. Similarity to the image being “explained”==++Image AnalogiesAaron Hertzmann1,2Chuck Jacobs2Nuria Oliver2Brian Curless3David Salesin2,31New York University2Microsoft Research3University of WashingtonImage AnalogiesAA’BB’Blur FilterEdge FilterAA’BB’Artistic FiltersColorizationTexture-by-numbersAA’BB’Super-resolutionAA’Super-resolution (result!)BB’Scene Completion Using Millions of PhotographsScene Completion Using Millions of PhotographsJames Hays and Alexei A. EfrosCarnegie Mellon UniversityEfros and Leung resultCriminisi et al. resultCriminisi et al. resultScene Matching for Image CompletionScene Matching for Image CompletionScene Completion ResultThe AlgorithmThe AlgorithmInput image Scene DescriptorImage Collection200 matches20 completionsContext matching+ blending……DataDataWe downloaded 2.3 Million unique images from Flickr groups and keyword searches.Scene MatchingScene MatchingScene DescriptorScene DescriptorScene DescriptorScene DescriptorGist scene descriptor (Oliva and Torralba 2001)Scene DescriptorScene DescriptorGist scene descriptor (Oliva and Torralba 2001)Scene DescriptorScene Descriptor+Gist scene descriptor (Oliva and Torralba 2001)… 200 totalContext MatchingContext MatchingResult RankingResult RankingWe assign each of the 200 results a score which is the sum of:The scene matching distanceThe context matching distance (color + texture)The graph cut costTop 20 ResultsTop 20 Results… 200 scene matches… 200 scene matches… 200 scene matches… 200 scene matches… 200 scene matches… 200 scene matches… 200 scene matchesFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresFailuresEvaluationEvaluationCriminisi et al. Scene CompletionSingle result Each result selected from 20Original ImagesSingle result Each result selected from 20Original Images Criminisi et al. Scene CompletionReal Image. This image has not been manipulatedorFake Image. This image has been manipulatedUser Study Results - 20 ParticipantsUser Study Results - 20 ParticipantsWhy does it work?Why does it work?10 nearest neighbors from acollection of 20,000 images10 nearest neighbors from acollection of 2 million imagesTorralba, Fergus, and Freeman. Tiny Images. MIT-CSAIL-TR-2007-024. 2007.Database of 70 Million 32x32 imagesThe Small PictureThe Small PicturePixelsImage CollectionPixels + SemanticsWhat Next? Small PictureWhat Next? Small Picture• Add outline of what we just presented• We presented a very different hole-filling technique• Sometimes works better than old stuff, but not always.• Value in reusing original material. Just need semantics.• Hybrid solution.Hybrid Solution?Hybrid Solution?PixelsImage CollectionSemanticsThe Big PictureThe Big PictureSky, Water, Hills, Beach, Sunny, mid-dayBrute-force Image
View Full Document
Unlocking...