HERA: 360° Image Warping Surveillance System Final Report Team Members Bryan Gwin [[email protected]]David Lariviere [[email protected]] DJ Park [[email protected]] Michael Verbalis [[email protected]]Table of Contents 1.0 Introduction 1.1 System Overview 1.2 Block Diagram 2.0 Components 2.1 Camera System 2.2 Video Sub-system 2.2.1 Overview 2.2.2 Block Diagram 2.2.3 Memory Details 2.2.4 Video Input 2.2.5 Video Output 2.3 Warping Module 2.3.1 Overview 2.3.2 Block Diagram 2.3.3 Parameter Generation 2.3.4 BRAM Block 2.3.5 Warper 3.0 Miscellaneous 3.1 Team Structure 3.2 Lessons learned 4.0 Appendix 4.1 System Environment Set-Up 4.2 Video Sub-system Code 4.3 Warping Module Code1.0 Introduction 1.1 System Overview The goal of this project is to take a real-time video stream containing a 360° image into the video input of the XESS XSB-300E board and to warp a portion of it in order to display a properly scaled image onto a monitor. 1.2 Block Diagram The following block diagram describes the current flow of data in this system: Video Input Video DecoderBRAM Block MicroblazeSRAM Warping Module VGA Out 2.0 Components 2.1 Camera System In order to capture the initial 360° video stream, we will be using a standard parabolic 360° lens mounted onto a PC165C Color ExView CMount Video Camera. This camera is in the family of high performance security cameras and has extremely sharp resolution (480 x 480) for a camera its size. 2.2 Video System 2.2.1 Overview Video is captured by a video input module that digitizes each pixel of the input video stream into an 8 bit luminance signal and stores it in the SRAM (16 bits at a time). A video output module generates the appropriate memory address in the SRAM, fetches this data, converts it into data suitable for the DAC, and then it is processed into output.2.2.2 Block Diagram2.2.3 Memory Details 2.2.4 BRAM We will be using the on-chip, dual-ported BRAMs on the FPGA. Each of these BRAMs have adjustable widths for each port and are .5K in size. The following chart summarizes the use of BRAMs in this system: Component # of BRAMs used Size of BRAMs Video Decoder 4 512 words x 8 bits Warper 3 256 words x 16 bits The Video Decoder component uses the BRAMs to buffer the image data before sending it off to the SRAM. The Warper component uses the BRAMs to store the warping parameters that are generated by software.SRAM We will be using the Toshiba TC55V16256J SRAM, which features 512Kb (256K x 16bits) of memory, to store each frame of video. 2.2.5 Video Input We will be using the on-board Phillips SAA7114-H video decoder chip to handle the analog video signal obtained from the camera. The chip is capable of digitizing any NTSC, SECAM, or PAL video signal and scaling it to an appropriate frame size. The chip also allows for precise control over the luminance, chrominance, hue, saturation, etc. of the video through numerous controls. Each pixel of the input video is digitized into an 8-bit luminance value and sent to a block ram module that takes care of merging the pixel data together so that it can be sent to the SRAM. This system uses a software module to perform actual transfer of the pixel data from the BRAM to the SRAM by passing the information through the OPB Bus. Currently, the SRAM stores 16 bits of pixel data (2 pixels) at each memory address. 2.2.6 Video Output We will be using the on-board Texas Instruments THS8133B Digital-To-Analog Converter that features a triple 10-bit Digital to Analog Converter (DAC) to translate and deliver the desired image to a VGA display. The warping module (explained in section 2.3) identifies the pixels and their corresponding memory addresses in SRAM that are needed to generate the properly warped image. Once the proper pixel data is grabbed from the SRAM, this information is then converted into a data format recognizable by the DAC and then processed into output. 2.3 Warping Module 2.3.1 Overview The input signal consists of a video stream from a 360 degree catadioptric lens (see image on the next page). The purpose of warping is to take a portion of the 360 degree image and unwarp it into a standard perspective-view image representing the image that would have been captured from a standard camera taken in the corresponding direction. The computational cost of mapping input pixels to corresponding output pixels exactly is rather large and is therefore not possible in real-time. However, a series of simplifications and approximations were used to make it feasible.360° Input Image Unwarped Perspective View Please see Appendix for an example C program that generates unwarped images. 2.3.2 Theory 2.3.3 The original formulation was in polar coordinates utilizing transcendental functions: Consider a set of coplanar points in 3D space. Given a specific point representing a on the image plane, (xp, yp, zp): Trace a ray between the optical center of the camera and the point to be displayed. Calculate the intersection of the ray with the paraboloid that defines the mirror, and its projection onto the (x,y) plane which yields the corresponding input pixel.Given: Ө = cos-1( (zp) / sqrt(xp2 + yp2 + zp2) ) Φ = tan-1(yp/xp) Xi = p * sin Ө * cos φ Yi = p * sin Ө * sin φ By restricting possible camera planes to a single frame that is orthogonal to the (x,y) plane and sitting atop the point (r,0,0), it is possible to derive a clean euclidian formulation: X = (h * xp) / (rp + zp) Y = (h * yp) / (rp + zp). By parameterizing the arc traced out upon the (x,y) plane in terms of a rotation about a point on the plane, for each scanline, it becomes possible to iteratively calculate every point on a scanline, provided only with the first point on the scanline and the angle of increment. To calculate this parameterization, simply calculate the first and last points along a scanline, and then the angle between them. Lastly divide the angle by the number of pixels per scanline, which results in the “angle of increment.” This is the amount Ө, by which one must rotate about the center of the particular circle. Xi = x cos Ө + y sin Ө Yi = - x sin Ө + y cos Ө Finally, through a first order linear approximation, we can accurately derive: X = x + y * sin Ө Y = x * (-sin Ө) + yNote that this formulation has one additional benefit as well. While the initial points calculated
View Full Document