MPEG-7 Visual DescriptorsAcknowledgementsWhat you may (may not) expect..Audio-Visual Content Description and the MPEG-7 StandardLow level Visual Information DescriptionMotivationMPEG StandardsObjective of MPEG-7Type of applicationsExample of application areasExample of queriesRelation content / descriptionType of descriptionWhy do we need a standard ?Scope of MPEG-7MPEG-7: The WorkplanInformation FlowParts of the MPEG-7 StandardVisual DescriptorsPowerPoint PresentationColor Datasets and Evaluation CriteriaPerformance evaluationAverage Retrieval Rate (AVR) and ANMRRColor DescriptorsColor spacesRGB color spaceHSV color spaceRGB to YCrCbRGB to HSVRGB to HMMDHMMDHMMD QuantizationScalable color histogramScalable color descriptorExtraction & matchingSlide 36GoP/GoF descriptorColor structureColor StructureStructuring element: scalabilityInteroperabilityCSD: Experimental resultsDominant colorDescriptor DefinitionDominant color (contd.)Similarity Distance MeasureEnhancementsANMRR Results for DCWith Spatial Coherence fieldMeritsColor Layout Descriptor (CLD)Experimental ResultsApplication to video segment retrievalSlide 54Texture descriptorsTexture Browsing DescriptorProposed Texture DescriptorExamplesSlide 59Slide 60Slide 61Similarity Retrieval: Homogeneous Texture DescriptorSRCDescriptor computation: Frequency LayoutE.g.: Browsing large aerial photographsExamples of tile-based searchApplications - Web Image SearchWeb image searchSlide 69More detailsEdge histogramEdge masksLocal edge histogramGlobal histogramANMRR resultsLocal vs Global histogramsSome example resultsSlide 78Shape DescriptorsTypes of Shape DescriptorCSS DescriptorSlide 82CSS image formationART DescriptorThe Definition of ARTRotation Invariance of the ARTExperimental Dataset & ProcedureSlide 88Slide 89Retrieval ExampleSlide 91Ex: Trademark Registration ApplicationSlide 93Motion DescriptorsCamera MotionMotion Activity: motivationPROPOSED MOTION ACTIVITY DESCRIPTORINTENSITYSpatial Distribution: Using run-lengths to describe moving regionsSPATIAL DISTRIBUTIONTEMPORAL DISTRIBUTIONDIRECTIONAPPLICATION TO VIDEO BROWSINGImprove Search by Combining Intensity and Spatial AttributesAPPLICATIONSMotion activity: conclusionsMotion TrajectoryTrajectory (contd.)ConclusionsDs to DSFurther informationMPEG-7 Visual DescriptorsMPEG-7 Visual DescriptorsIEEE International Conference on Consumer ElectronicsIEEE International Conference on Consumer ElectronicsLos Angeles, CA, June 2001Los Angeles, CA, June 2001B. S. ManjunathUniversity of California, Santa Barbara, USA [email protected]Prof. P. Salembier (for permitting me to use some of his slides fr our tutorial at ICIP’2000).Editors of the MPEG-7 XM and WD. Dr. L. Cieplensky, Mitsubishi ElectricDr. A. Divakaran, MitsubishiDr. S. Jeannin, Philips research lab Prof. W.Y. Kim, Hanyang UniversityDr. M. Bober, Mitsubishi ElectricDr. H. J. Kim, LG ElectronicsDr. S. Park, ETRIWhat you may (may not) expect..Overview of the visual descriptorsTheir capabilities and limitationsSome application examples Pointers to publicly available documentsDo not expectProgramming and implementation details – these are available in the MPEG-7 eXperimental Model (XM) document.Binary bit stream syntax – see the MPEG-7 Committee Draft(s).Audio-Visual Content Description and the MPEG-7 StandardObjective, goals, requirements and applicationsBasic component of the MPEG-7 standardDescription Definition LanguageDescription SchemesA/V DescriptorsLow level Visual Information DescriptionColorShapeTextureMotionFaceMotivationThe multimedia context:More information is in digital form and is on-line.AV content covers: still pictures, audio, speech, video, graphics, 3D models, etc.AV content is available at all bitrates and on all networks.Increasing number of multimedia applications, services.Necessity of describing content:Increasing amount of information.More needs to have “information about the content”.Difficult to manage (find, select, filter, organize, etc) content.User: human or computational systems.©SalembierMPEG StandardsMPEG-1: Storage of moving picture and audio on storage media (CD-ROM)11 / 1992MPEG-2: Digital television 11 / 1994MPEG-4: Coding of natural and synthetic media objects for multimedia applications v1: 09 / 1998 v2: 11 / 1999MPEG-7: Multimedia content description for AV material 08 / 2001MPEG-21: Digital audiovisual framework: Integration of multimedia technologies (identification, copyright, protection, etc.) 11 / 2001©SalembierObjective of MPEG-7Objective of MPEG-7Objective of MPEG-7Objective of MPEG-7Standardize content-based description for various types of audiovisual information Enable fast and efficient content searching, filtering and identificationDescribe several aspects of the content (low-level features, structure, semantic, models, collections, creation, etc.)Address a large range of applications ( user preferences)Types of audiovisual information: Audio, speechMoving video, still pictures, graphics, 3D modelsInformation on how objects are combined in scenesDescriptions independent of the data support Existing solutions for textual content or description ©SalembierType of applicationsPull Applications: “Search and Browsing”Example: Internet search engines and databasesAdvantage: Queries based on standardized descriptionsPush Applications: “Filtering”Example: Broadcast video, Interactive televisionAdvantage: Intelligent agents filter on the basis of standardized descriptionsUniversal Multimedia Access:Adapt delivery to network and terminal characteristics (QoS)Specialized Professional and Control Applications©SalembierExample of application areasStorage and retrieval of audiovisual databases (image, film, radio archives)Broadcast media selection (radio, TV programs)Surveillance(traffic control, surface transportation, production chains)E-commerce and Tele-shopping (searching for clothes / patterns)Remote sensing(cartography, ecology, natural resources management)Entertainment (searching for a game, for a karaoke)Cultural services (museums, art galleries)Journalism (searching for events, persons)Personalized news service on Internet (push media filtering)Intelligent multimedia presentationsEducational applications Bio-medical