1SVCLContent based Image Retrieval(at SVCL)Nikhil Rasiwasia, Nuno VasconcelosStatistical Visual Computing LaboratoryUniversity of California, San DiegoECE271A – Fall 20072SVCLImage retrieval•Metadata based retrieval systems– text, click-rates, etc.– Google Images– Clearly not sufficient•what if computers understood images? –Content basedimage retrieval (early 90’s)–search based on the image contentTop 12 retrieval results for the query ‘Mountain’Metadata based retrieval systems3SVCLContent based image retrieval -1•Query by Visual Example(QBVE) –user provides query image– system extracts image features (texture, color, shape)–returns nearest neighbors using suitable similarity measureTexturesimilarityColorsimilarityShapesimilarity4SVCLQuery by visual exampleBag of DCT vectors+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++GMMQuery ImageCandidate ImagesProbability under various modelsRanking5SVCLQuery by visual example (QBVE)QUERY TOP MATCHES6SVCL• visual similarity does not always correlate with “semantic” similarityQuery by visual example (QBVE)Both have visually dissimilar skyDisagreement of the semantic notions of train with the visual notions of arch.7SVCLContent based image retrieval -2• Semantic Retrieval (SR)– User provided a query text (keywords)– find images that contains the associated semantic concept.– around the year 2000,– model semantic classes, learn to annotate images– Provides higher level of abstraction, and supports natural language queriesabcquery: “people, beach”8SVCLSemantic Class ModelingBag of DCT vectors+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++GMMwi= mountainmountain()mountainxPWX||Semantic Class ModelEfficient Hierarchical Estimation•“Formulating Semantics Image Annotation as a Supervised Learning Problem”[G. Carneiro, IEEE Trans. PAMI, 2007]9SVCL10SVCL11SVCLFirst Five Ranked Results• Query: mountain• Query: pool• Query: tiger12SVCLFirst Five Ranked Results• Query: horses•Query: plants• Query: blooms13SVCLFirst Five Ranked Results•Query: clouds•Query: field•Query: flowers14SVCLFirst Five Ranked Results•Query: jet•Query: leaf•Query: sea15SVCLSemantic Retrieval (SR)•Problemof lexical ambiguity– multiple meaning of the same word•Anchor - TV anchor or for Ship?•Bank - Financial Institution or River bank?•Multiple semantic interpretations of an image•Boating or Fishing or People?•Limited by Vocabulary size – What if the system was not trained for ‘Fishing’– In other words, it is outside the space of trained semantic conceptsLake? Fishing? Boating? People?Fishing! what if not in the vocabulary?abc16SVCLIn Summary•SR Higher level of abstraction – Better generalization inside the space of trained semantic concepts– But problem of • Lexical ambiguity • Multiple semantic interpretations• Vocabulary size •QBVEis unrestricted by language. – Better Generalization outside the space of trained semantic concepts• a query image of ‘Fishing’ would retrieve visually similar images.– But weakly correlated with human notion of similarityVSabcBoth have visually dissimilar skyFishing! what if not in the vocabulary?Lake? Fishing? Boating? People?The two systems in many respects are complementary!17SVCLQuery by Semantic Example (QBSE)•Suggests an alternate query by exampleparadigm. – The user provides an image.– The image is mapped to vector of weights of all the semantic concepts in the vocabulary, using a semantic labeling system.– Can be thought as an projection to an abstract space, called as the semantic space– To retrieve an image, this weight vector is matched to database, using a suitable similarity functionLakeWaterPeopleSky…Boat.2 .3 .2 .1 … …Semantic SpaceMapping to an abstract space of semantic conceptsSemantic multinomialvector of weights or18SVCLQuery by Semantic Example (QBSE)•As an extension of SR– Query specification not as set of few words.– But a vector of weights of all the semantic concept in the vocabulary.– Eliminates•Problem of lexical ambiguity-Bank+’more’• Multiple semantic interpretation–Boating, People•Outside the ‘semantic space’–Fishing.•As an enrichment of QBVE–The query is still by an example paradigm.–But feature space is Semantic. •A mapping of the image to an abstract space.– Similarity measure at a higher level of abstraction. .1 .2 .1 .3 … ….2LakeWaterPeopleBoating…Boat0.5 0.5…… 0(SR) query: water, boating=≠(QBVE) query: imageLakeWaterPeopleBoating…BoatSemantic SpaceBoatingWaterLake19SVCLQBSE SystemConcept 1Query ImageAny Semantic Labeling SystemConcept 2Concept 3Concept L. . . DatabaseWeight Vector 1Weight Vector 2Weight Vector 3Weight Vector 4Weight Vector 5Weight Vector N. . . Suitable Similarity Measure. . . Ranked RetrievalPosterior probability Weight Vector1π2π3πLπ20SVCLQBSE SystemConcept 1Query ImageAny Semantic Labeling SystemConcept 2Concept 3Concept L. . . DatabaseWeight Vector 1Weight Vector 2Weight Vector 3Weight Vector 4Weight Vector 5Weight Vector N. . . Suitable Similarity Measure. . . Ranked Retrieval1π2π3πLπPosterior probability Weight Vector21SVCLSemantic Class ModelingBag of DCT vectors+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++GaussianMixture Modelwi=