Video Alignment Literature Survey Spring 2005 Prof. Brian Evans Multidimensional Digital Signal Processing Project The University of Texas at Austin Omer Shakil Abstract This literature survey compares various methods to align two videos. The idea can be extended for alignment of multiple video sequences. The sequences are recorded by un-calibrated video cameras with fixed (but unknown) internal parameters. It will be shown that by using a combination of spatial information, temporal changes and the inbuilt frame-to-frame transformation of the video sequences, efficient video alignment can be achieved. The last method described in the survey demonstrates a novel approach to recover alignment of non-overlapping videos. Video Alignment gives rise to a wide variety of new applications that are not possible when only image-to-image alignment is used.Literature Survey Video Alignment -1- 1. Introduction The problem of image alignment has been extensively studied, and successful solutions have been suggested [4, 6]. The issue here is to estimate point correspondences between two images, i.e. given any pixel (x, y) in one image, find its corresponding pixel (x’, y’) in the other image, where x’ = x + u, y’ = y + v, and (u, v) is the calculated spatial displacement vector. The image alignment techniques can be divided into two broad categories: the first category includes feature-based approaches [7], in which common features are detected and matched across the two images; the second category comprises of methods that directly match image intensities [4]. As both of these categories rely on the spatial coherence of the given images, there must be a significant overlap between the two images for these techniques to work. This literature survey evaluates methods to align two videos. Here, the problem is that the sequences may not be synchronized; hence alignment in time is required as well. Given points (x, y, t) from video sequence V and (x’, y’, t’) from video sequences V’, the task is to recover the transformation between them i.e. find (u, v, w) such that (x’, y’, t’) = (x+u, y+v, t+w). The temporal “redundancy” in successive frames of a video can be used to move a step beyond from the traditional image alignment techniques that exploit spatial coherence between the given images. Therefore, by using the temporal behavior of the videos (frame-to-frame transformations), alignment can be achieved even when the corresponding frames from each sequence have no spatial overlap between them [3].Literature Survey Video Alignment -2- A variety of applications, esp. in surveillance and security systems, can benefit heavily from alignment of multiple video sources. A primary application is monitoring multiple video inputs (high security areas) and the output being displayed in one composite video sequence, instead of various monitors dedicated for each individual input source. Other applications include generation of wide screen movies and super-resolution in time and space. 2. Overview The problem of video alignment can be simplified to aligning corresponding frames using image alignment techniques, but there are cases when using only common spatial information is not enough to determine the transformation. One such example is illustrated below: Figure 1: Spatial ambiguities in image-to-image alignment. (a) and (b) Show two corresponding frames in time from two different video sequences viewing the same moving ball. There are infinitely many valid image alignments between the two frames, some of them shown in (c). (d) and (e) Display the two sequences of the moving ball. There is only one valid alignment of the two trajectories of the ball. This uniquely defines the alignment both in time and in space between the two video sequences (f). [1] This literature survey describes various approaches with different constraints that attempt to solve the problem of Video Alignment. As explained above, the simple image alignment techniques would not produce desirable results. In some of the successful methods, only the temporal variations (moving objects) between the sequences are used [1, 7, 9]. Other methods try to link both spatial and temporal information together forLiterature Survey Video Alignment -3- better results [1, 2, 3]. All of these methods impose some constraint on the movement of the video sequences. A table summarizing the advantages and disadvantages of all these techniques is also provided after discussing them in detail. 3. Feature-Based Alignment Feature-based image alignment can be generalized to video alignment by extending the notion of feature points into feature trajectories. The alignment between the given sequences can be recovered by establishing correspondences between trajectories. The advantage of using this method over simple image alignment is illustrated in figure 2. It shows two sequences with several small moving objects. Figure 2: Point versus trajectory correspondences. (a) and (b) Display two frames out of two sequences recording five small moving objects (marked by A, B, C, D, and E). (c) and (d) Display the trajectories of these moving objects over time. When analyzing only single frames, it is difficult to determine the correct point correspondences across images. However, point trajectories have additional properties that simplify the correspondence problem across two sequences (both in space and time). [1] Actually, a single pair of non-trivial (moving in a straight line can lead to ambiguity) corresponding trajectories is sufficient to find the alignment between the two image sequences. The main idea of the algorithm [1] is described below: 1. Construct feature trajectories by tracking the centroids of moving objects. 2. For each possible pair of corresponding trajectories, find the transformation matrix. 3. Choose the transformation matrix that gives the least MSE after alignment.Literature Survey Video Alignment -4- 4. Repeat steps 2 and 3 N times, and choose the best transformation matrix. Stein [7, 9] has proposed similar technique. However, he has treated the features as unordered collection of points and not
View Full Document
Unlocking...