New Techniques forNew Techniques forImproved Video CodingImproved Video CodingThomas WiegandFraunhofer Institute for TelecommunicationsHeinrich Hertz InstituteBerlin, [email protected]. Wiegand: New Techniques for Improved Video Coding 1Outline Outline § Inter-frame Encoder Optimization§ Texture Analysis and Synthesis§ Model-Aided CodingT. Wiegand: New Techniques for Improved Video Coding 2ScopeScopeof Video Coding Standardizationof Video Coding StandardizationOnly Restrictions on the Bitstream, Syntax, and Decoderare standardized:• Permits optimization beyond the obvious• Permits complexity reduction for implementability• Provides no guarantees of qualityPre-ProcessingEncodingSourceSinkPost-Processing& Error RecoveryDecodingScope of StandardT. Wiegand: New Techniques for Improved Video Coding 3EntropyCodingScaling & Inv. TransformMotion-CompensationControlDataQuant.Transf. coeffsMotionDataIntra/InterCoderControlDecoderMotionEstimationTransform/Scal./Quant.-InputVideoSignalSplit intoMacroblocks16x16 pixelsIntra-frame PredictionDe-blockingFilterOutputVideoSignalHybrid Video EncoderHybrid Video EncoderT. Wiegand: New Techniques for Improved Video Coding 4InterInter--frame Optimization of Video Encodingframe Optimization of Video Encoding§ Most video encoders select locally optimal transform coefficients as they ignore temporal dependencies§ New approach that takes inter-frame dependencies into accountTimeMotion CompensationT. Wiegand: New Techniques for Improved Video Coding 5Linear Signal Model for DecoderLinear Signal Model for DecoderpTcMss ++=reconstructed sample valuesmotion datainverse transform & scalingstatic predictortransform coefficients§ Very large size of matrices§ Numbers when optimizing 4 QCIF pictures jointly: M, T: 100,000 x 100,000 entriesT. Wiegand: New Techniques for Improved Video Coding 6Quadratic ProgramQuadratic Program§ Use Mean Squared Error distortion§ Select transform coefficient levels by solving:(s − v)T(s − v) +λabs(ci)i∑subject to:§Equivalent to a quadratic program§Can be solved by standard solution methods! minimizepTcMss ++=original video signalT. Wiegand: New Techniques for Improved Video Coding 7ResultsResults§ Used H.264/MPEG4-AVC§ Compare looking at 2 or 3 frames at a time§ Compare to H.264/MPEG4-AVC Test Model§ Constant QP§ Motion estimation on original frames§ Only optimized inter-framesT. Wiegand: New Techniques for Improved Video Coding 8FlowerGardenFlowerGarden30Hz IPPP…..30Hz IPPP…..16% Bit-rate improvement29303132333435363738390 100 200 300 400 500Bit-Rate [kbits/sec]Average Y PSNR [dB]TM22 Frames Jointly3 Frames JointlyT. Wiegand: New Techniques for Improved Video Coding 9FlowerGardenFlowerGarden30Hz 30Hz IbPbPbPIbPbPbP…..…..15% Bit-rate improvement30313233343536370 50 100 150 200 250 300Bit-Rate [kbits/sec]TM23 Frames JointlyAverage Y PSNR [dB]T. Wiegand: New Techniques for Improved Video Coding 10Summarizing InterSummarizing Inter--frame Optimizationframe Optimization§Very complex: has only recently become possible§Problems with H.264/MPEG4-AVC: rounding§Adjust design to support efficient encoding methods§Heavy incorporation of encoding methods has been used in H.264/MPEG4-AVC development with Lagrangian encoder optimization§Do not standardize the encoder!T. Wiegand: New Techniques for Improved Video Coding 11Texture Analysis and Synthesis Texture Analysis and Synthesis §Textures with large amount of visible details are difficult to code e.g., grass, sand, clouds, water ...§Viewer often does not perceive large differences between different versions of grass, sand, clouds, water ...Where is the original texture ?T. Wiegand: New Techniques for Improved Video Coding 12Video Analysis and SynthesisVideo Analysis and Synthesis§Approach: Exact reproduction of details is irrelevant if• Textures are shown with limited spatial accuracy• Viewer does not know the original video§Consequence: Mean Squared Error (MSE) distortion criterion is not suitable for efficient coding of detail-irrelevant textures§Issue: Could we use similarity criteria (color histograms, etc) instead of MSE as coding distortion ?§Scheme: If information needed for approximate reproduction of detail-irrelevant textures requires less bit-rate than using MSE Ł bit-rate savingsT. Wiegand: New Techniques for Improved Video Coding 13Video Video CodecCodecArchitectureArchitectureSide InfoVideo InEncoderTATSDecoderBitsVideo OutVideo coding using a texture analyzer (TA) and a texture synthesizer (TS)T. Wiegand: New Techniques for Improved Video Coding 14Texture Analyzer PrincipleTexture Analyzer PrincipleSplit & MergeSegmentationMotion CompensationTexture AnalyzerCurrent frameRef. maskRef.frameCurrent maskT. Wiegand: New Techniques for Improved Video Coding 15Texture SynthesizerTexture SynthesizerStuffing procedureT. Wiegand: New Techniques for Improved Video Coding 16Texture SynthesizerTexture Synthesizer§Match surrounding samples between current picture and warped pictures to find synthesis sample for the current sample§Decoder processing required!§Volume matching for water and temporal consistenceT. Wiegand: New Techniques for Improved Video Coding 17Flowergarden, CIF, 30 Hz, QP=16, 41 framesBit-rate savings: 19.4%T. Wiegand: New Techniques for Improved Video Coding 18Canoe, CIF, 30 Hz, QP=16, 73 framesBit-rate savings: 8.8 %T. Wiegand: New Techniques for Improved Video Coding 19Summarizing Texture Analysis and Synthesis Summarizing Texture Analysis and Synthesis § Textures with large amount of visible details are difficult to code e.g., grass, sand, clouds, water ...§ Errors in these textures are barely visible§ Texture analysis and synthesis can help to represent these textures efficiently§ Method requires processing at the decoder side § Other methods require processing at the decoder side to achieve coding efficiency improvements through backward adaptation (CABAC)T. Wiegand: New Techniques for Improved Video Coding 20ModelModel--Aided CodingAided Coding§ Model-based coding (MBC):• Extreme low bit-rates possible• Missing generality§ Hybrid video coding (H.263/4, MPEG-2/4,...)• Robust against scene changes• Higher bit-rates§ Combination of MBC with hybrid video coding ⇒ Model-aided codec• High coding efficiency• No restriction to scene contentT. Wiegand: New Techniques for Improved Video Coding 21Architecture of the ModelArchitecture of the Model--Aided CodecAided CodecIntraframeDCT
View Full Document
Unlocking...