PhilipsResearch Fine Granular Scalability - A new framework for real-timestreaming of video over the InternetMihaela van der SchaarPhilipsResearchOverview■Why a new coding paradigm for Internet Video?■“Fine-Granular-Scalability” (FGS) framework■FGS features■How does FGS compare with other coding schemes?■New extensions■FGS applications beyond video streaming■ConclusionsPhilipsResearchWhy a new coding solution?Internet characteristics!■Wide range of available bandwidths & packet-loss rates=> No QoS guaranteesPhilipsResearchBandwidth variations0.000.501.001.502.002.503.003.504.00kbits/sPercentBandwidth probability densityNew York - L.A.1.1 5.4 9.7 14.0 18.3 22.6 26.1 31.2PhilipsResearchBandwidth variationsPhilipsResearchBandwidth variations■ “Broadband” Internet access has wider variation:◆◆Cable modem:Cable modem:from < 100 from < 100 to > 1000to > 1000 Kbit Kbit/sec/sec◆◆DSL:DSL: from < 600 to > 6000 from < 600 to > 6000 KbitKbit/sec/secPhilipsResearchPacket-lossesTime02468101214Percent19:4823:483:497:3110:5014:2717:5922:322:336:3410:0013:35Packet Loss RatioNew York - L.A.PhilipsResearchWhy a new coding paradigm?Networks with time-varying bandwidthcharacteristics (e.g. Internet):■ No guarantees of QoS - wide range ofavailable bandwidths and data loss rateTherefore required■ Easy adaptability to changing bandwidth■ Resilience to packet-lossesSolution: scalable codingPhilipsResearchScalable Video in current standardsEnhancement Layer(s)Base LayerI P P P PP B B B BP B B B BRBLREnh2REnh1PhilipsResearchScalable Video in current standards■ Alternative solution to simulcast■ Operate at a discrete set of bit-rates■ Supported bit-rates determined at Encoding-Time■ Overhead increases with the number of layers■ Not applicable for Packet NetworksPhilipsResearchWhat is important for a good solution?An adaptive solution for Internet video■Easy bandwidth adaptability■Encoding & transmission processes should beseparated (transmission bit-rates do not need tobe specified at encoding-time)■Resilience to packet-losses (Inter-picturepredictions within EL complicates loss recovery)■Low server complexity -> unicast■Support both unicast & multicast■Low/scalable decoder complexitySolution: Fine-Granularity-ScalabilityPhilipsResearchFine-Granularity Scalability (FGS) RminRMaxPhilipsResearchFine-Granularity Scalability (FGS)Base Layer (e.g. MPEG-4 non-scalable codec)I B P B PRBL < RminA Single Enhancement LayerRMaxPhilipsResearchBase LayerI B P B PFGS Enhancement LayerFGS EL-portiontransmittedin real-timeR(t)Internet Video Streaming with FGS -Server sidePhilipsResearchInternet Video Streaming with FGS -Decoder sideBase LayerI B P B PFGS Enhancement LayerPhilipsResearchAdvantages of the FGS structure■ Fine granular scalable enhancement layer■ Encoding & Transmission processes - separated■ Easy rate-control on multiple-streams for VOD■ Resilience to packet-losses■ Efficient for both unicast & multicastPhilipsResearchFGS features◆◆Adaptive Adaptive quantizationquantization◆◆Frequency weightingFrequency weighting◆◆Hybrid temporal-SNR scalabilityHybrid temporal-SNR scalability◆◆Error-resilience markers for wireless appsError-resilience markers for wireless appsPhilipsResearchBlock-diagram of FGS-scalabilityVLC Q DCTΣΣΣΣMCBase-layer (BL)encoderIDCTΣΣΣΣMEMemoryIQOriginalVideoMUXBLMVsBL streamPhilipsResearchBlock-diagram of FGS-scalabilityVLC Q DCTΣΣΣΣMCFGS Bit-plane DCTscanning and entropycodingSNR FGS streamBase-layer (BL)encoderFGS EL EncoderΣΣΣΣIDCTΣΣΣΣMEMemoryIQOriginalVideoMUXBLMVsBL streamPhilipsResearchFGS bitplane codingBP( 1) = msbBP(N) = lsbDCT bitplanesPhilipsResearchFGS AQ - Selective enhancementBP(1)BP(N+1)DCT bitplanesMacroblock k –shifted up by one bitplanePhilipsResearchHow to select visually important regions?Possible solution:Possible solution:use real-time face-detection/tracking algorithmsuse real-time face-detection/tracking algorithmsPhilipsResearchFGS - Selective enhancement resultPhilipsResearchHybrid FGS temporal-SNR scalability -Motivation■For optimal quality: trade-offs between individual imagequality (SNR) and temporal resolution (higher frame-rates)■Transmission frame-rate ~ transmission bit-rate => Hybrid SNR-temporal FGS scalabilityPhilipsResearchImproved scalability■ A single FGS enhancement-layer■ Total flexibility in supporting◆◆SNR scalability with same frame-rateSNR scalability with same frame-rate◆◆temporal scalability by increasing ONLY thetemporal scalability by increasing ONLY theframe-rateframe-rate◆◆both FGS & temporal scalabilityboth FGS & temporal scalability■ Low added complexityPhilipsResearchAn all FGS temporal-SNR scalabilityI P P PBase LayerFGS Enhancement LayerSNRSNR SNR SNRRMax RBLPhilipsResearchAn all FGS temporal-SNR scalabilityI P P PBase LayerFGS Enhancement LayerSNRSNR SNR SNRTEMP TEMP TEMP TEMPRMax RBLPhilipsResearchReal-time trade-off SNR-temporalPortion of the enhancement layertransmitted in real-timeI P P PFGS Enhancement LayerBase LayerPhilipsResearchReal-time trade-off SNR-temporalI P P PBase LayerFGS Enhancement LayerPortion of the enhancement layertransmitted in real-timePhilipsResearchReal-time trade-off SNR-temporalI P P PFGS Enhancement LayerBase LayerPortion of the enhancement layertransmitted in real-timePhilipsResearchRate-control SNR/TemporalI P PFGS Enhancement LayerBase LayerScenario BI P PFGS Enhancement LayerBase LayerScenario APhilipsResearchRate-control SNR/TemporalSNR versus temporal trade-off based onsequence characteristics (static):◆◆motion-info (e.g. motion-info (e.g. MVsMVs size) size)◆◆texture characteristics (e.g. texture characteristics (e.g. Xi Xi computed bycomputed bybase-layer RC)base-layer RC)However, also dependent on RtPhilipsResearchPerformance of FGS and FGS+FGST atsame bit-rate1520253035400 20 40 60 80FramenumberFOREMAN at 500 kbit/sPSNR (dB)Scenario AScenario BPhilipsResearchSNR versus motion-smoothnessA18202224262830323436PSNR (dB)0 10 20 30 40 50 60 70 80 90 100FramenumberScenariochosen:BBAFOREMAN at 500 kbit/sPhilipsResearchRmin, Rmax calculationVideosourceVariable-bandwidthnetworkInternet video streaming - EncodingPhilipsResearchRmin, Rmax calculationBL EncoderVideosourceRBLVariable-bandwidthnetworkBase-layer stream of rate RBLInternet video streaming - EncodingPhilipsResearchRmin, Rmax calculationBL EncoderVideosourceSNR residualcomputation(SNR-frames)RBL