Wavelet Video Coding –Principles, Applications and StandardizationMihaela van der SchaarElectrical and Computer Engineering DepartmentUniversity of California Davis2Outline Introduction Scalable coding – principles (review) Basic principles of wavelets (review) Motion Compensated Wavelet Coding – basic principles and classification  Motion Compensation Temporal Filtering (MCTF) Overcomplete Motion Compensated Wavelet Coding Encoding of spatio-temporal wavelet coefficients Scalable coding of motion information Error resilience aspects Current status in MPEG standardization Comparisons with state-of-the-art non-scalable coding techniquesIntroduction4Challenges for ubiquitous multimedia communicationEncoder+ServerIP-based< 64 k< 512 k< 11 M802.11b6 M802.11a< 2 M802.1164 k -2 M3G/4GInternet/Internet25Sample of concrete problems/questions Signal processing compression efficiency versus quality of signal reproduction (rate-distortion tradeoffs) compression efficiency versus robustness to losses Networking realistic channel models for effective joint source/channel coding source-channel interface control strategies for efficient network resource usage and high quality signal reproduction Computer Architecture compression efficiency versus computational complexity6Possible solution: compression meets the network Do not require the transport mechanism to be flawless (modulation, channel coding, transmission protocol etc.), just design the coding system and transmission jointly Do not design for worst-case scenario - just adapt on the fly based on the network and device characteristicsHence:  A. Scalable Coding  B. Adaptive Streaming7Principles of Scalable Coding Encoding of video signal with different resolutionscales Downscaling of video signal by Coding noise insertion – SNR Scalability Spatial subsampling – Spatial Scalability Sharpness reduction – Frequency Scalability Temporal subsampling – Temporal Scalability Selection of content – Content related ScalabilityScaleConversion& EncodinglowmediumhighRate / ResolutionVideoInput8The Simple Way – Advance Scaling Requires feedback about channel / decoder status Only point-to-point connection supported Example : Stream switchingCoderNetworkDecoderScaleConverter9The Parallel Way - Simulcast Run independent encoders in parallel Requires a priori knowledge about network and decoder capabilities to select optimum scalinglevels Point-to-multipoint connections possibleLow Scale CoderMed. Scale CoderHigh Scale CoderMultiplex10Simulcast Multiplexed transmission of streams Loss in efficiency due to multiple streams Can cause network overload Restricted number of scalesMultiplex StreamMedium rate streamLow rate streamHigh rate stream11The Embedded Way – Layered Coding "Chain of layers" - information from low resolutionutilized to encode next-higher resolutionΣCoder Layer 1...Layer 1...ΣΣ...−+Σ−+y1y2yT(Base layer)(Enhancement layers)x...Σ++++++Q1Q2QTQ1Q2QTPreprocessing 1Midprocessing 1Preprocessing 2Midprocessing 2Midprocessing 1Midprocessing 2Coder Layer 2Coder Layer TDecoder Layer 1Decoder Layer 2Layer 2Layer TDecoder Layer 1Decoder Layer 2Decoder Layer T12Layered Coding Layered coding supports embedded streams Re-configuration of bit stream for reconstruction withdifferent spatial/temporal/quality resolution Possible loss in efficiency depends on coding scheme In theory, arbitrary number of scales could be achievedFull multiplex = high rate streamPartial multiplex = medium rate streamLow rate stream13SNR Scalability – Re-quantisation Example : 2-stage quantizerQ1Q2Σ-+BaseEnhancementLarge stepsSmall steps (≤Q1/2)Reconstruction valueDecision (threshold) valueQ1Q214SNR Scalability – Bit-plane Coding Quantization related to bit planes No zero reconstruction,unsignedZero reconstruction,sign/magnitudeZero reconstruction,sign/magnitude, dead zone... ... ...Bit 1 Bit 2 Bit 3 Bit 1Bit 2Bit 3MINMAX0MAXReconstruction valueDecision (threshold) valueBit 1 Bit 2 Bit 30MAX... ... ...... ... ... ...... ...15SNR Scalability – Bit-plane Coding Magnitude of MSB encoded by run-length orbinary entropy coding Sign and remaining bits encoded binary, conditional on MSBSample1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Bit 5Bit 4Bit 3Bit 2Bit 1Sign000011000000010110000010110011001100000101000000000010000100000000010110001000000001000111Run-length code4,92,103,5,23,4,10,1,1,0,2binary coded16Spatial Scalability Base-to enhancement predictionQ1Q2Σ-+BaseEnhancementLow passFilterN:1Low passFilter1:NDecimationInterpolation17Temporal Scalability Temporal downsampling with temporal anti-alias filter or by frame skipping Temporal upsampling by MC predictionQ1Q2Σ-+BaseEnhancementLow passFilterN:1Low passFilter (MC)1:NTemporalsubsampling(optional)18Frequency Scalability / "Data Partitioning" Popular in context of Transform Coding Allocation of coefficients to different layersdepending on frequency Very low complexityDataPartitioningSingle-layerEncoderOriginalVideoOutputstreamBase-layerstreamEnhancement-layer streamMUXPriorityBreakPointBase-layercoefficientsEnhancement-layercoefficients19Multiresolution Concepts Generate different resolution levels by successivedown/upsampling operations Resolution pyramids example : Spatial resolutionreduction by factors of 2Full resolutionLowest resolution...c0c1c2cU-120Multiresolution Concepts – Pyramids Gaussian Pyramid Each layer is self-contained Corresponds to Simulcast concept More samples to be encoded4:1......H(z1,z2)4:1H(z1,z2)cU-2c0c1cU-1x(m,n)21Multiresolution Concepts – Pyramids Laplacian Pyramid (Differential Pyramid) All lower-resolution layers required to reconstruct high-resolution layers Corresponds to Layered Coding concept Not critically sampled – more samples than originalΣΣ...+-+-cU-1c0c14:1H(z1,z2)...G(z1,z2)1:44:1H(z1,z2)G(z1,z2)1:4x(m,n)22Multiresolution Concepts – Pyramids Advantages : Pyramids can be combined with any coding scheme forthe different resolution levels Downsampling can be made alias-free Disadvantages : Number of pixels higher than in original signal Higher data rate than one-layer coding Possible solution : Critically sampled pyramids (Wavelets) Disadvantage : Downsampled signals bear aliasBasic Principles of Wavelets24Filter Pairs Critically