Department of Computer Science & EngineeringContent-Based Multimedia Information RetrievalIshwar K. SethiIntelligent Information Engineering LaboratoryDepartment of Computer Science & EngineeringOakland UniversityRochester, MI 48309Email: [email protected]: www.cse.secs.oakland.edu/isethiDepartment of Computer Science & EngineeringDescriptive contentSubjective contentBehavioral reaction of a viewer to the imageContent?Department of Computer Science & EngineeringContent-Based Information Retrieval (CBIR)An inherently difficult problem because “what is actually in a document” is a function of both the document and the user. The ideal situation for perfect retrieval occurs when the document representation of the retrieval system and document representation of the user are in complete match.Department of Computer Science & EngineeringTypes of CBIR Queries• Level 1– Find pictures with round red objects in the top left-hand corner• Level 2 (Descriptive queries)– Find images containing multistory buildings• Level 3– Find images showing tranquilityDepartment of Computer Science & EngineeringCurrent Content-Based Retrieval MethodsKeyword-based retrieval (KBR)Similarity-based retrieval (SBR)Department of Computer Science & EngineeringKeyword-Based RetrievalGood for finding images containing instances of desired objects (descriptive queries)Manual catalogingHigh expressive powerCan be used to describe any aspect of image content at various levels of complexitySubject to user differencesTwo people choose the same main keyword for a single well-known object only about once in five timesDepartment of Computer Science & EngineeringSimilarity-Based RetrievalAvoids issues related to manual catalogingSuitable for computerized indexingAble to capture the compositional aspects to a limited extentGood for Level 1 queriesDepartment of Computer Science & EngineeringSimilarity-Based RetrievalImagesQueryFeature ExtractionFeature ExtractionBest Match SearchRetrieved ImagesDepartment of Computer Science & EngineeringAn Example of SBRDepartment of Computer Science & EngineeringMajor Limitation of the SBR ApproachSignal versus descriptive/semantic content similarity (Semantic gap)Department of Computer Science & EngineeringHow to Reduce the Semantic Gap? • Stuff detectors• Image category detectors / feature associations• Exploiting other information sources– Surrounding text / image captions– Associated audio– Cross-modal associationDepartment of Computer Science & EngineeringStuff DetectorsStuff detectors are object detectors. Current computer vision methods allow to build a small set of special detectors, each designed to detect the presence of a particular type of “stuff.” Examples of some stuff detectors include- faces- traffic signs- treesDepartment of Computer Science & EngineeringFace DetectorDepartment of Computer Science & EngineeringTraffic Sign DetectorDepartment of Computer Science & EngineeringImage Category DetectorsThese detectors try to determine the broad category of image content by building image classifiers. These detectors are different from stuff detectors which locate specific types of objects within an image. Here, the image as a whole is assigned a descriptive keyword.Department of Computer Science & EngineeringImage Category LearningSemi-Automatic generation of Semantic ConceptsPixel DataLearned visual Concept: ‘sunset’Feature ExtractionLow-level features: Color Semi-Automatic generation of Semantic ConceptsPixel DataLearned visual Concept: ‘sunset’Feature ExtractionLow-level features: Color The relationship between image data, low-level features, and high-level concepts (image categories) can be visualized using the triangle relationship between data, information, and knowledge: low-level features (information) are extracted purely from pixel data, and knowledge (learned visual concepts) is discovered from the most important low-level features and image contexts.Department of Computer Science & EngineeringAn Example of Image Category ClassificationSample images classified as ‘sunset’ by a rule-based image classifier, eID systemDepartment of Computer Science & EngineeringCodebook Based Image Category Detection• Good for mass noun entities, for example grass, water, sand etc.• Entity specific codebook designed through vector quantization• A confidence value is attached to each codeword in the entity specific codebook• Image category is decided by encoding a given image through different entity specific codebooks and integrating the resulting confidence valuesDepartment of Computer Science & EngineeringVector Quantization Based Image Category Classifier Smoke AgentGrass AgentFire AgentSky AgentWater AgentDepartment of Computer Science & EngineeringExploiting Text Surrounding Images• Keywords extracted from text surrounding images can provide a way of reducing semantic gap• The image search engine Google, for example, has cataloged over 450 million images using the surrounding text to extract keywordsDepartment of Computer Science & EngineeringGoogle Example for “Taj Mahal”Keyword = Taj Mahal, Source = Google Image FinderDepartment of Computer Science & EngineeringGoogle Result for “Prayer”Department of Computer Science & EngineeringInformation Sources in a Multimedia StreamDepartment of Computer Science & EngineeringVideo Analysis for CBIRWhat should be the analysis level?A frame? A shot? A scene?Scene componentsObjects (who), action or event (what), and place or context (where)Compositional componentsCamera shot, angle, and movementSubjective componentsEmotion and moodDepartment of Computer Science & EngineeringIntegrated Analysis Approach for VideoVideo and image analysisface detection, tracking, and recognitionAudio analysisaudio segmentation and classificationspeech/speaker recognitiontext understandingClosed caption text analysisTranscript understandingDepartment of Computer Science & EngineeringInputVideoDataAudioClassificationCutDetectionKeyword Spotting Speaker IdentificationFace Detection & TrackingDigitalVideoDatabaseGUIPartial Block Diagram of the Integrated SystemDepartment of Computer Science & EngineeringAudio Analysis for Video IndexingAudio segmentation and classificationSpeaker identificationKeyword spottingSpeech recognitionText understandingDepartment of Computer Science & EngineeringCross-Modal RetrievalLocate or retrieve documents of all modalities in response to a query in any