Multiple Camera Tracking Helmy Eltoukhy and Khaled Salama Stanford image sensors group Electrical Engineering Department, Stanford University Tracking of humans or objects within a scene has been studied extensively. We present a multiple camera system for object tracking. The system employs uncalibrated cameras and depends on the motion-tracking algorithm to achieve both point correspondence and image registration. The system is capable of switching between different cameras to achieve the best tracking conditions of the object. A simple communication scheme is established between the cameras in order to achieve both robust tracking and occlusion prevention. 1. Introduction Recently multiple sensor environment gained interest within academia. The problem of having multiple sensors covering a relatively small environment and interacting with each other is of huge interest due to great number of problems that arise which spans different areas of research. Those sensor networks provide redundant information due to their large number and their overlap that can be used as a means to implement robust algorithms or to achieve better performance. Those sensors can be biological, chemical, temperature, pressure, audio, image or video sensors. They should be able to communicate with each other wirelessly so as to minimized the infra structure needed for their deployment. Object tracking in a video sequence has been studied extensively for surveillance purposes. The ability to detect an object and then later tracking is of great interest in many applications like security, missile tracking and rescue operations. The traditional tracking algorithms are designed either to track a single object within the field of view of the camera or to detect multiple objects within the same scene and both cases depends on using a single camera. Recently there has been some work in multiple camera environments, in which an array of cameras are used to image the same scene. The main application for such a system is for 3D modeling of objects like in light fields. It was alsoused to track multiple objects in the scene such that each camera is dedicated to an object. Only recently people started to consider using multiple cameras to track a single object in the scene. This has the advantage of having more information about the object being tracked. In this project we present a system in which two cameras at 180 degrees are used to track an object in the scene. This provides more robust tracking since the camera system is capable of switching between the 2 camera view for better tracking of the object if it is outside the field of view of one of them or if it is occluded by an object. Ideally the system consists of three main parts: a) Object tracking: In which each camera is tracking the object independently and producing an estimate of its position with some kind of an error measure b) Real time communication: each camera should be able to send the position and error information to a central node to be processed c) Point correspondence: a central node should be able to achieve point correspondence between the two cameras so as to confirm the position of the object and based on the make some decision on what action to take. This is done by building a model and using calibrated points to estimate the parameters In reality we had to simplify the system so as to be easily done with the class project period. The system currently looks like the following: a) Object tracking: a two-camera system is used to capture 2 sequences of the scene that are later are processed independently to track the object. The output of this stage is a matrix which includes the estimated object position and an error measurement for each scene within the sequence b) Communication: The matrices obtained from the previous stage are operated in sequence in order to simulate the communication part between cameras and central node c) Point correspondence: After trying different models for the scene it was apparent that the accuracy of this registration problem is really hard and it can not be trusted to give accurate results. So we switched to a dynamic real-time point correspondence that depends on the tracking algorithm. And based on that we can access the exact location of the object and determine which view is better to see.2. Optical Flow and Feature Tracking The brightness constancy assumption is vital to the successful implementation of correlation or gradient-based optical flow estimation algorithms, i.e., ψ(xk+1, t+∆t) = ψ(xk, t), where ψ(x, t) is the frame intensity at location x and time t. Accordingly, all methods discussed herein make ample use of this assumption. . First of all, any thriving feature tracking algorithm must be predicated upon a reliable optical flow estimation algorithm which is simple enough to be implemented at 30 frames per second or higher. Such stringent requirements preclude the use of computationally intensive methods such as that of Fleet and Jepson. Furthermore, since it is well-known that gradient-based methods, such as Lucas-Kanade, are fairly accurate when applied to subpixel optical flow estimation, as well as computationally tractable, a logical first step is to explore the feature tracking scheme proposed by Shi and Tomasi. 2.1 Shi and Tomasi Feature Tracking This algorithm employed the use of Lucas-Kanade on carefully chosen “corner” points. Intuitively, it is clear that good features constitute those with large spatial gradients in two orthogonal directions. Since Lucas-Kanade involves solving the optical flow equation iteratively assuming the displacement is characterized by constant velocity as given by, ∂∂⋅∂∂∂∂⋅∂∂=⋅∂∂∂∂⋅∂∂∂∂⋅∂∂∂∂ystsxstsvvysysxsxsysxsyx22, the two by two spatial gradient matrix can be used to determine the quality of each possible corner point, where the gradients are summed across an n x n block. Tomasi, et al. suggests that a reasonable criterion for feature selection is for the minimum eigenvalue of the spatial gradient matrix to be no less than some λ. This ensures that the matrix is well conditioned and above the noise level of the image so that its inverse does not unreasonably amplify possible noise in a certain critical direction, i.e., there