1Layered Image RepresentationChuck DyerBased on the paper “Representing moving images with layers,”J. Wang and E. Adelson, IEEE Trans. Image Processing 3(5), 1994Motivation+ Standard flow assumes optical flow is smooth+ Bad things happens at occlusion boundaries+ Instead, decompose image sequence into a set of overlapping layers + Each layer is smooth in its own motion2Problem DefinitionExample Input Video34Algorithm5Motion Vectors vs Motions Hypothesis• There are a number of different motion hypotheses available–In theory, each of these hypothesis corresponds to a distinct motion in the video• Each pixel is assigned to the motion hypothesis that most closely approximates its motion vector• This segments the frame into distinct regions, one for each motion hypothesis6Motion Hypothesis Generation• For each region want motion hypothesis that best represents all pixel motions in that region• Least squares fit to find best affine motion parameter in a region• First iteration initialized with small blocksMotion Hypothesis Refinement• K-Means used to cluster motion hypotheses• K unknown• Empty clusters removed• Large clusters split to maintain minimum kvalue7Region Segmentation• For each pixel compare hypotheses to dense motion vectors• Find closest hypothesis• Group all pixels represented by a motion hypothesis into a region• Pixels with large error unassigned• Hypotheses without membership removedRegion Adjustment• Region Splitter• Assumes areas with same motion are connected• Disconnected areas within a region are split into separate regions• Increases number of hypotheses for k-means• Region Filter• Small regions give poor motion estimates• Remove all regions with area below threshold• Disconnected objects with same motion will be merged at next segmentation step8Algorithm Summary• Dense motion estimation, region segmentation, and motion estimation performed for all pairs of consecutive frames• For first pair, segmentation initialized to blocks and k-Means initialized to lattice in 6D affine space• Subsequent frame pairs initialized with final segmentation and motion hypotheses from previous frame pairLayer Synthesis• Motion estimates relate each frame only to the previous frame•Frames are projected onto first video frame•Cumulative projection kept in 3x3 transformation matrix• Layers are not necessarily ordered similarly between frames•Assume largest layer is background • Median taken of all values projected to each pixel in final image9Affine Motion SegmentationVideo Mosaic of Each Layer• Flower Bed regions in all images aligned10Motion Compensation• Aligned regionsFlower BedTree House113 Major Layers12Application: Video Synthesis• Layered decomposition captures spatial coherence of object motion and temporal coherence of object shape and texture in a few semantically-meaningful layers• Synthesize new sequences from the
View Full Document
Unlocking...