ECE 532 design project 2009 Group Report Real Time Color Replacement Chang Liu 995416250 Andrei Bajenov 995478763 Jeffery Sham 995384310 April 9 2009 Table of Contents PAGE 1 Overview 3 1 1 Goals 3 1 2 Project Background 3 1 3 Invisibility Effect 3 1 4 System Block Diagram 5 1 5 Brief Description of IP 5 1 6 Clock Domain 7 2 Project Outcome 7 2 1 Review of Original Plan 7 2 2 Our Final Product 8 2 3 Suggestions For the Future 8 3 Description of Blocks 9 3 1 MicroBlaze 9 3 2 PLB Bus 10 3 3 IIC Interface 11 3 4 TFT Controller 11 3 5 MPMC Controller 12 3 6 Decoder Logic 12 3 7 1 IP to Mem Custom Logic 12 3 7 2 Color Detection Algorithm 15 3 8 Mem to IP Custom Logic 16 3 9 Connection between IP to Mem and Mem to IP 18 4 Description of Our Design Tree 19 5 Reference 20 6 Appendix 21 6 1 PLB Burst Write Timing Diagram 21 6 2 PLB Burst Read Timing Diagram 22 6 3 PLB Single Beat Read Timing Diagram 23 2 Page OVERVIEW 1 1 Goals The goal of our project was to develop a system which performs real time color replacement of a video stream In particular our system will detect a certain colored pixel in the video frames and alter it with a pixel from a different image in the video output Using this concept we are not only able to implement a simple commercial Chroma Key using only hardware but to also develop a device that can make you completely invisible on camera 1 2 Project Background Chroma keying more commonly known as green screening is a technique used in movie production to film scenes that are too difficult to achieve with props set pieces and other special effects In Chroma keying the actor performs a scene in front of a coloured screen and later the specified colour is filtered out and replaced with the desired background There are a number of subtleties involved in perfecting Chroma keying that makes it a tedious process This includes correctly defining the colour to key out in order to avoid colour bleeding and applying even lighting on the backdrop and subject to achieve the best contrast However for the purposes of our application we are more interested in demonstrating that the process can be done in real time and will accept the slightly lower quality result given the limitations of our lighting and backdrop equipment Thus our goal is to be able to take in video frames from the camera detect the pixels with a backdrop colour and replace them with an image that is saved in memory In order to satisfy the real time criteria the processing should be fast enough that when the foreground subject changes there is no detectable glitch in the video output 1 3 Invisibility Effect The trick behind the invisibility effect is taking a still shot of the background before the subject enters the set and replace the desired colour with the saved image to create the illusion that the subject has disappeared This requires that the camera does not move and that the background is not changed Failure to do so will cause discrepancies between the saved background and actual background thus ruining the effect In terms of the technical aspect we first need to be able to save the background as a static frame somewhere in our memory This will allow to know exactly what the background is at all times during the live stream even when something or someone is blocking it There are key similarities in the implementation of Chroma key and the 3 Page invisibility effect that allowed us to achieve both with very little addition In the original Chroma keying concept the static image comes from non volatile memory such as compact flash In the invisibility effect the static frame comes from the initial frame input from the camera In both cases we would need to use memory for the frame and we must create our hardware so that it will read in lines of data for the pixel replacement The slight extension in implementing the invisibility effect is the extra logic required in our hardware to save the initial frame into DDR memory However it should be noted that if our goal was only to replace one colour with another colour we can simply modify the video stream as it streams to VGA and bypass one of our bigger challenges of utilizing memory in our design The diagram below shows a high level layout of our system 4 Page 1 4 System Block Diagram 1 5 Brief Description of IP Block Functionality Origin IP block inside the FPGA Decoder logic Coverts each pixel of the video frame coming from the vdec composite into data in RGB format IIC interface Sends instructions to the Vdec 5 Page Modified from video capture 0 from digilentinc com Xilinx IP replaces the I2C module used in VideoCapture IP To Mem Mem To IP MicroBlaze 0 dlmb ilmb MPMC controller PLB bus TFT controller Detect the target color from the incoming frame replace it with pixels from the Mem to IP module and then stores altered frame into memory Bus Master uses burst write Read a frame from the memory and sends the data to the Ip to mem module Bus Master uses burst read 1 send commands and configs to the IIC 2 used by XMD to write configuration data to the memory It allows us to access the same piece of memory from multiple ports Custom Logic Four PLB bus were used in this project 1 Used for the microblaze to talk to the DDR and iic 2 Used between IP To Mem and memory 3 Used between Mem To IP and memory 4 Used by the TFT to get data from the DDR Xilinx IP Reads a video frame from the memory and sends it to the VGA monitor to be displayed Xilinx IP Custom Logic Xilinx IP Xilinx IP Software block inside the MicroBlaze ddr test c This configures the iic hardware module Modified from video capture 0 from digilentinc com Devices outside the FPGA DDR memory VGA monitor Video Camera Video Decode Chip vdec 6 Page The DDR stores the static image of the background the incoming frame of the live video and the 32 bit information that controls the color detection algorithm configuration Display the output Capture the video and streams it to vdec in composite format Reads the video from the camera through composite and sents the data to the decoder logic Given Given Self supplied Given 1 6 Clock Domain Component Video In Custom Hardware IP to mem mem to IP other system components DDR Memory VGA Clock Speed 27MHz 100MHz 133MHz 27 MHz The video in is in a clock domain of 27MHz which includes the Vdec and decoder logic The custom hardware runs at 100 MHz and this domain crossing is handled by the asynchronous double line buffer The DDR memory runs at 133 MHz therefore the MPMC …