Lecture 10: Stereo and Graph CutsAnnouncementsReadingsWhat we’ve learned so farFeature detection and matchingPanoramasMagic: ghost removal Magic: ghost removal Optical flowYour basic stereo algorithmStereo resultsResults with window searchCan we do better?Stereo as energy minimizationStereo as energy minimizationStereo as energy minimizationStereo as energy minimizationStereo as energy minimizationSmoothness costDynamic programmingDynamic programmingDynamic ProgrammingDynamic programmingStereo as a minimization problemInterlude: binary segmentationSlide Number 26Binary segmentation as energy minimizationSlide Number 28Slide Number 29Binary segmentation as energy minimizationGraph min cut problemSegmentation by min cutSlide Number 33GrabCutBack to stereoEnergy minimization via graph cutsEnergy minimization via graph cutsComputing a multiway cutMove examplesThe swap move algorithmResults with window correlationResults with graph cutsOther energy functionsQuestions?Real-time stereoStereo reconstruction pipelineActive stereo with structured lightLaser scanningLaser scanned modelsLaser scanned modelsLaser scanned modelsSlide Number 52Spacetime StereoLecture 10: Stereo and Graph CutsCS6670: Computer VisionNoah SnavelyAnnouncements• Project 2 out, due Wednesday, October 14– Artifact due Friday, October 16• Questions?Re adings• Szeliski, Chapter 11.2 – 11.5What we’v e learned so far=*Image filteringEdge detectionCamerasImage transformationsFeature detection and matchingPanoramasMagic: ghost removal M. Uyttendaele, A. Eden, and R. Szeliski. Eliminating ghosting and exposure artifacts in image mosaics. In Proceedings of the Interational Conference on Computer Vision and Pa ttern Recognition, volume 2, pages 509‐‐516, Kauai, Hawaii, December 2001.Magic: ghost removal M. Uyttendaele, A. Eden, and R. Szeliski. Eliminating ghosting and exposure artifacts in image mosaics. In Proceedings of the Interational Conference on Computer Vision and Pattern Recognition, volume 2, pages 509‐‐516, Kauai, Hawaii, December 2001.Optical flowYour basic stereo algorithmFor each epipolar lineFor each pixel in the left image• compare with every pixel on same epipolar line in right image• pick pixel with minimum match costImprovement: match windowsStereo resultsGround truthSceneResults with window searchWindow‐based matching(best window size)Ground truthproblems in areas of uniform textureCan we do better?• What defines a good stereo correspondence?1. Match quality• Want each pixel to find a good match in the other image2. Smoothness• two adjacent pixels should (usually) move about the same amount Stereo as energy minimization• Find disparities d that minimize an energy function • Simple pixel / window matchingSSD distance between windows I(x, y) and J(x, y + d(x,y))=Stereo as energy minimizationI(x, y) J(x, y) y = 141C(x, y, d); the disparity space image (DSI)xdStereo as energy minimizationy = 141xdSimple pixel / window matching: choose the minimum of each column in the DSI independently:Stereo as energy minimization• Better objective functionmatch costsmoothness costWant each pixel to find a good match in the other imageAdjacent pixels should (usually) move about the same amountStereo as energy minimizationmatch cost:smoothness cost:4‐connected neighborhood8‐connected neighborhood: set of neighboring pixelsSmoothness cost“Potts model”L1distancelast time: looked at quadratic and truncated quadratic models for VDynamic programming• Can minimize this independently per scanlineusing dynamic programming (DP): minimum cost of solution such that d(x,y) = dDynamic programming• Finds “smooth” path through DPI from left to righty = 141xdDynamic ProgrammingDynamic programming• Can we apply this trick in 2D as well?• No: dx,y‐1and dx‐1,ymay depend on different values of dx‐1,y‐1Slide credit: D. HuttenlocherStereo as a minimization problem• The 2D problem has many local minima– Gradient descent doesn’t work well– Simulated annealing works a little better• And a large search space– n x m image w/ k disparities has knmpossible solutions– Finding the global minimum is NP‐hard• Good approximations exist…Interlude: binar y segmentation• Suppose we want to segment an image into foreground and backgroundInterlude: binar y segmentation• Suppose we want to segment an image into foreground and backgroundUser sketches out a few strokes on foreground and background…How do we classify the rest of the pixels?Binary segmentation as energy minimization• Define a labeling L as an assignment of each pixel with a 0‐1 label (background or foreground)• Problem statement: find the labeling L that minimizesmatch costsmoothness cost(“how similar is each labeled pixel to the foreground / background?”)• Neighboring pixels should generally have the same labels– Unless the pixels have very different intensities: similarity in intensity of p and q= 10.0= 0.1(can use the same trick for stereo)Binary segmentation as energy minimization• For this problem, we can easily find the global minimum!• Use max flow / min cut algorithmGraph min cut problem• Given a weighted graph G with source and sink nodes (s and t), partition the nodes into two sets, S and T such that the sum of edge weights spanning the partition is minimized – and sS and t TSegmentation by min cut• Graph– node for each pixel, link between adjacent pixels– specify a few pixels as foreground and background• create an infinite cost link from each bg pixel to the t node• create an infinite cost link from each fg pixel to the s node• create finite cost links from s and t to each other node– compute min cut that separates s from t• The min‐cut max‐flow theorem [Ford and Fulkerson 1956]ts (“foreground”)min cut(“background”)Segmentation by min cuttsmin cut• The partitions S and T formed by the min cut give the optimal foreground and background segmentation• I.e., the resulting labels minimizeGrabCutGrabcut [Rother et al., SIGGRAPH 2004]Back to stereo• Can formulate as a (no longer binar y) labeling problem– with one label per disparity• Can create similar setup as with segmentation, but with k source/sink nodes– k = number of disparities– Using the Potts model, the setup is straightforwardEnergy minimization via graph cutsLabels (disparities)d1d2d3edge weightedge weightd1d2d3• Graph Cut– Delet e enough edges so that•