CS664 Lecture #9: Binary labeling problems and their applicationsAnnouncementsRecapGraph constructions-t Graph Cuts for General Binary LabelingWhy should anyone care about binary labeling problems?Application 1: shape reconstructionVoxel algorithm for volume intersectionVoxel algorithm for volume intersectionSilhouette intersectionProblem formalizationEnergy functionWhat smoothness term?Examples on real dataVisualizationAnother exampleFly-throughsApplication 2: medical image segmentations-t Graph Cut for General Binary Labelings-t Graph Cut for General Binary LabelingInteractive Object ExtractionInteractive Object ExtractionApplication 3: object “cut and paste” Photomontage ProblemFramework Graph Cuts Boykov and Jolly (2001) Iterated Graph Cut Iterated Graph Cuts Colour Model Coherence Model Moderately straightforward examples Difficult ExamplesEvaluation – Labelled DatabaseCS664 Lecture #9: Binary labeling problems and their applicationsSome material taken from:Carsten Rother et al, Microsoft Researchhttp://research.microsoft.com/vision/cambridge/i3l/segmentation/GrabCut.htmOlga Veksler, University of Western Ontariohttp://www.csd.uwo.ca/faculty/olga/2Announcements PS1 is on the web, and debugged (?)– Due on 10/6 Quiz 2 on Tuesday 9/27– Coverage through last lecture3 We want to minimize the energy E(f) Binary labeling problem – Also known as Ising model (1924)Recap4Graph construction Binary labeling via graph cuts example: •λ = 3• DP(0) = 1 and DP(1) = 4 whenever I(p) = 0• DP(0) = 3 and DP(1) = 2 whenever I(p) = 101p rqsblackwhite333324114235s-t Graph Cuts forGeneral Binary Labeling Binary Labeling problem:– We have 2 labels, let’s call them s and t (associated with the source and sink terminals, respectively)– We have a set of sites (image pixels, volume voxels, so on) to which we wish to assign these labels– Each site p prefers one label more than another– We wish to find a labeling L (assignment of labels s and t to sites) such that• Each site is assigned a label it prefers, as much as possible• Nearby pixels tend to have similar labels6Why should anyone care about binary labeling problems? Remarkably powerful and common– Now that we can solve them fast!– You can use graph cuts to evaluate simulated annealing, and also to determine if the minimum is actually what you want• Pretty good, but not quite (why?) 3 applications today:– 3D shape reconstruction (CVPR 2000)– Medical image segmentation (ICCV 2001)– Object “cut and paste” (SIGGRAPH 2004)7Application 1: shape reconstruction Snow, Viola & Zabih, CVPR00– Create a 3D shape from multiple cameras Generalization of “image differencing”– Take two consecutive images– Subtract and threshold Easy to build a simple motion detector– Common in “security webcams”– Where does this fail? Consider an object surrounded by cameras– Or, on a turntable8Voxel algorithm for volume intersection Color voxel gray if on silhouette in every image9Voxel algorithm for volume intersectionhole Color voxel gray if on silhouette in every image10Silhouette intersection Early hard decisions are a problem– Errors in silhouette identification– Can lead to “holes” Can we add spatial smoothness with energy minimization? More importantly, can we do it fast?11Problem formalization We are given a set of voxels p∈P which must be assigned binary labels fp– 1 if filled, 0 if empty A pixel in an image corresponds to a set of voxels in a line– If there is a large change in intensity (versus background image), these voxels “prefer” to be 1’s. Similarly for 0’s. Note what we don’t represent:– Surfaces, photoconsistency, etc.12Energy function The cost for the voxel p to have the label fpwill be written Dp(fp) We seek the labeling f that minimizesfor some smoothness term Esmooth(f))()()( fEfDfEsmoothppp∑+=13What smoothness term? It is easy to imagine lots of clever smoothness terms– That we have no idea how to minimize! Here is a natural one where we can efficiently find the global minimum!][),(),()(,qpqpNqpqpsmoothffffVffVfE≠==∑∈T14Examples on real data15Visualization16Another exampleConventional Energy minimization17Fly-throughs Synthetic example (cylinder)– silhouette intersection reconstruction– graph cuts reconstruction Sara– graph cuts reconstruction Paul– silhouette intersection reconstruction– graph cuts reconstruction18Application 2: medical image segmentation Boykov & Jolly, ICCV01– Interactive tool to segment objects19s-t Graph Cut forGeneral Binary Labeling For the fax image problem, we minimized:()()()∑∑∈≠+=NpqqpppPLLLDLETλ More general energy function: ()()()∑∑∈≠+=NpqqppqppPLLwLDLET¾ The weight of discontinuity is wpqand it depends on a particular pair of pixels p and q.In grayscale (or color) images, discontinuities between similarly colored pixels cost more than discontinuities between differently colored pixels; object boundaries tend to coincide with intensity boundaries20s-t Graph Cut forGeneral Binary Labeling How does our graph construction change?¾ Cost of t-links does not change¾ Instead of λ, make the weight of n-link between pixels p and qto be wpqstp rquDp(t)Dp(s)Du(s)Du(t)wqrwruwpq21sthard constrainthard constraintInteractive Object Extraction 2 labels: background (blue), object (red) User clicks on a few pixels with blue and red paintbrushes Data term¾ DP(blue) = infinity (that is a very large number) if the user clicked on pixel p with a red paintbrush9 Hard constraint: pixel p must be in the background ¾ DP(red) = infinity (that is a very large number) if the user clicked on pixel p with a blue paintbrush9 Hard constraint: pixel p must be in the object¾ Otherwise DP(red) = DP(blue) = 022n-linkssta cuthard constrainthard constraint Prior term¾ Object boundaries tend to have sharp intensity edge¾ Encourage cutting around sharp intensity edge¾ Make wpqcheaper for larger intensity difference between pand qInteractive Object Extraction2324Application 3: object “cut and paste” Rother, Kolmogorov, Blake SIGGRAPH04PhotomontagePhotomontagePhotomontageGrabCut – Interactive Foreground Extraction 1ProblemProblemProblemGrabCut – Interactive Foreground Extraction 2Fast & Accurate ?FrameworkFrameworkFrameworkInput: Image Output: Segmentation Parameters:
View Full Document