Lecture notes on Image Compression and Video Compression 7 Video Compression Topics z Introduction to Image Compression z Transform Coding z Subband Coding Filter Banks z Introduction to Wavelet Transform z Wavelet Image Compression z Perceptual Audio Coding z Video Compression 2 Contents z Introduction to Video Compression z MPEG 1 Video Compression z Motion Compensation z Video Compression Standards 3 Digital Video z A series of frames i e digital images z Can be sampled From raster analog scan z Directly with a CCD camera z z 3 color components z Usually color components downsampled 4 Color Down sampling 5 The Need for Compression z z Image sequences must be significantly compressed for efficient storage and transmission as well as for efficient data transfer among various components of a video system Examples Motion Picture z z One frame of a Super 35 format motion picture may be digitized via Telecine equipment to a 3112 lines by 4096 pixels color 10 bits color image As a result 1 sec of the movie takes 1 Gbytes 6 The Need for Compression Examples z HDTV A typical progressive scan non interlaced HDTV sequence may have 720 lines and 1280 pixels with 8 bits per luminance and chroma channels z The data rate corresponding to a frame rate of 60 frames sec is 720 x 1280 x 3 x 60 165 Mbytes sec z 7 Applications of Digital Video Compression z Teleconference or video phone z z Live Broadcast Video z z z Modest delay is tolerable seconds is normal Error tolerance is needed Video in a can DVD Video on Demand z z z Very low delay 1 10 second is a standard Random access to compressed data is desired Encoding can take a lot of time Decoding must always be at least the frame rate 8 Approaches to Video Compression z Intraframe compression treats each frame of an image sequence as a still image z z Intraframe compression when applied to image sequences reduces only the spatial redundancies present in an image sequence Interframe compression employs temporal predictions and thus aims to reduce temporal as well as spatial redundancies increasing the efficiency of data compression z Example Temporal motion compensated predictive compression 9 Why can Images be Compressed z z z Image compression can be achieved primarily because image data are highly redundant The degree of redundancy determines how much compression can be achieved Four types of redundancy can be identified z Spatial Redundancy z z Temporal Redundancy z z Correlation between different frames in an image Spectral Redundancy z z z Correlation between adjacent data points Correlation between different color planes or sensors Limitation of Low level Human Vision System Psycho visual Redundancy z Limitation of high level Human Vision System 10 Human Perception of Video z z 30 frames per second seems to allow the visual system to integrate the discrete frames into continuous perception If distorted nearby frames in the same scene should have only small details wrong z z A difference in average intensity is noticeable Compression choice when reducing bit rate z z skipped frames cause stop action lower fidelity frames may be better 11 High Compression Ratios Possible z Nearby frames are highly correlated Use the previous frame to predict the current one z Need to take advantage of the fact that usually objects move very little in 1 30th of a second z Video coders use motion compensation as part of prediction 12 Video Compression z Main addition over image compression z z z Exploit the temporal redundancy Predict current frame based on previously coded frames Three types of coded frames z z z I frame Intra coded frame coded independently of all other frames P frame Predictively coded frame coded based on previously coded frame B frame Bi directionally predicted frame coded based on both previous and future coded frames 13 MPEG 1 Structure z MPEG codes video in a hierarchy of layers 14 MPEG 1 Group of Pictures GOP Structure z z z Composed of I P and B frames Periodic I frames enable random access into the coded bit stream Parameters 1 Spacing between I frames 2 number of B frames between I and P frames 15 Example Use of I P B frames MPEG 1 Group of Pictures GOP z Arrows show prediction dependencies between frames 16 Definition of P Frames z P Pictures are composed of macroblocks that are either z z z z forward predictive non intra coded Or intra coded using the same quantization and VLC as macroblocks of the I pictures The encoder is allowed to make an Intra Nonlntra decision depending on the accuracy of the prediction This decision can be made in many different ways and the choice is up to the encoder 17 P Frames z A possible simple decision mechanism compares the variance of the luminance component of the original macroblock with that of the prediction error macroblock z If the variance of the prediction error macroblock is higher then the macroblock is intra coded 18 Definition of B Pictures z B Pictures are composed of macroblocks that are z z z z z z bi directional predictive coded or backward predictive coded or forward predictive coded or intra coded A possible decision mechanism is picking the mode that results in the least macroblock luminance component variance The macroblocks in the B pictures are not used as references 19 MPEG 1 z Relative number of I P and B pictures can be arbitrary z z z It depends on the nature of the application For instance it depends on fast access and compression ratio requirements relatively smaller amount of compression is expected to be achieved at I pictures compared to P and B pictures The B pictures are expected to provide relatively the largest amount of compression under favorable predictability conditions 20 MPEG 1 Frame Size z Size of I P B frames 21 MPEG 1 Bit Stream Hierarchy z MPEG 1 Bit Stream Hierarchy 22 Temporal Redundancy z Adjacent frames are highly correlated Frame 7 No motion compensation Frame 8 With motion compensation 23 Motion Compensation z In general we speak of motion of objects in 3 D real world z z z z Here we are concerned with the projected motion of 3 D objects onto the 2 D plane of an imaging sensor By motion estimation we mean the estimation of the displacement or velocity of image structures from one frame to another in a time sequence of 2 D images In the literature this projected motion is referred to as apparent motion 2 D image motion or optical flow The detail algorithms on optical flow is beyond the content of this course 24 Motion Compensation z Predict the