View Full Document

Content-Based Multimedia Information Retrieval



View the full content.
View Full Document
View Full Document

43 views

Unformatted text preview:

Content Based Multimedia Information Retrieval Department of Computer Science Engineering Ishwar K Sethi Intelligent Information Engineering Laboratory Department of Computer Science Engineering Oakland University Rochester MI 48309 Email isethi oakland edu URL www cse secs oakland edu isethi Content Department of Computer Science Engineering Descriptive content Subjective content Behavioral reaction of a viewer to the image Content Based Information Retrieval CBIR Department of Computer Science Engineering An inherently difficult problem because what is actually in a document is a function of both the document and the user The ideal situation for perfect retrieval occurs when the document representation of the retrieval system and document representation of the user are in complete match Types of CBIR Queries Level 1 Department of Computer Science Engineering Find pictures with round red objects in the top left hand corner Level 2 Descriptive queries Find images containing multistory buildings Level 3 Find images showing tranquility Current Content Based Retrieval Methods Keyword based retrieval KBR Department of Computer Science Engineering Similarity based retrieval SBR Keyword Based Retrieval Department of Computer Science Engineering Good for finding images containing instances of desired objects descriptive queries Manual cataloging High expressive power Can be used to describe any aspect of image content at various levels of complexity Subject to user differences Two people choose the same main keyword for a single well known object only about once in five times Similarity Based Retrieval Department of Computer Science Engineering Avoids issues related to manual cataloging Suitable for computerized indexing Able to capture the compositional aspects to a limited extent Good for Level 1 queries Similarity Based Retrieval Department of Computer Science Engineering Images Feature Extraction Best Match Search Query Feature Extraction Retrieved Images An Example of SBR Department of Computer Science Engineering Major Limitation of the SBR Approach Department of Computer Science Engineering Signal versus descriptive semantic content similarity Semantic gap How to Reduce the Semantic Gap Department of Computer Science Engineering Stuff detectors Image category detectors feature associations Exploiting other information sources Surrounding text image captions Associated audio Cross modal association Stuff Detectors Department of Computer Science Engineering Stuff detectors are object detectors Current computer vision methods allow to build a small set of special detectors each designed to detect the presence of a particular type of stuff Examples of some stuff detectors include faces traffic signs trees Face Detector Department of Computer Science Engineering Traffic Sign Detector Department of Computer Science Engineering Image Category Detectors Department of Computer Science Engineering These detectors try to determine the broad category of image content by building image classifiers These detectors are different from stuff detectors which locate specific types of objects within an image Here the image as a whole is assigned a descriptive keyword Image Category Learning Department of Computer Science Engineering Low level features Color Semi Automatic generation of Semantic Concepts Learned visual Concept sunset Feature Extraction Pixel Data The relationship between image data low level features and high level concepts image categories can be visualized using the triangle relationship between data information and knowledge low level features information are extracted purely from pixel data and knowledge learned visual concepts is discovered from the most important low level features and image contexts An Example of Image Category Classification Department of Computer Science Engineering Sample images classified as sunset by a rule based image classifier eID system Codebook Based Image Category Detection Department of Computer Science Engineering Good for mass noun entities for example grass water sand etc Entity specific codebook designed through vector quantization A confidence value is attached to each codeword in the entity specific codebook Image category is decided by encoding a given image through different entity specific codebooks and integrating the resulting confidence values Vector Quantization Based Image Category Classifier Smoke Agent Fire Agent Department of Computer Science Engineering Grass Agent Sky Agent Water Agent Exploiting Text Surrounding Images Department of Computer Science Engineering Keywords extracted from text surrounding images can provide a way of reducing semantic gap The image search engine Google for example has cataloged over 450 million images using the surrounding text to extract keywords Google Example for Taj Mahal Department of Computer Science Engineering Keyword Taj Mahal Source Google Image Finder Google Result for Prayer Department of Computer Science Engineering Information Sources in a Multimedia Stream Department of Computer Science Engineering Video Analysis for CBIR What should be the analysis level A frame A shot A scene Department of Computer Science Engineering Scene components Objects who action or event what and place or context where Compositional components Camera shot angle and movement Subjective components Emotion and mood Integrated Analysis Approach for Video Video and image analysis Department of Computer Science Engineering face detection tracking and recognition Audio analysis audio segmentation and classification speech speaker recognition text understanding Closed caption text analysis Transcript understanding Partial Block Diagram of the Integrated System Audio Classification Input Department of Computer Science Engineering Keyword Spotting Speaker Identification Digital G Video Video U Data Database Cut Detection Face Detection Tracking I Audio Analysis for Video Indexing Department of Computer Science Engineering Audio segmentation and classification Speaker identification Keyword spotting Speech recognition Text understanding Cross Modal Retrieval Department of Computer Science Engineering Locate or retrieve documents of all modalities in response to a query in any modality Opportunistic Vs Cross Modal Integration Opportunistic Approach Department of Computer Science Engineering The data from different modalities is processed independently and the results are used merged on a need basis Cross Modal


Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view Content-Based Multimedia Information Retrieval and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Content-Based Multimedia Information Retrieval and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?