1Image and Image and VideoVideoCollectionsCollectionsAili LiCSE 591:Visual AnalyticsProf. Klaus Mueller2BackgroundBackgroundz Digital photograph + internet sharing!z Lacked methods for interactive browsing and searching!3ChallengesChallengesz How to organize unstructured collection of photographs?– By annotated with caption and keywords, the photos can be indexed and searched as textual documents– By measuring the similarities between images– Most annotation have been done by manually, automatic annotation is not used in image browsing– Lack methods to monitor and evaluate the automatic image analysis algorithmz Image browsing is different from image querying4Structure of This TalkStructure of This Talk1. Image Browsing by Similarity2. Semantic Image Browser3. MorieGraph4. Photo Tourism43215Image Browsing by SimilarityImage Browsing by Similarityz CHI 2001 [[RoddenRoddenet al, CHI01]et al, CHI01]Does Organization by Similarity Assist Image Browsing?6OverviewOverviewz Construct arrangement of images based on:– Caption similarity– Visual similarityz Performed two experiments– if arrangement of images benefits users– Users preferences[[RoddenRoddenet al, CHI01]et al, CHI01]7Interface of the Experiment SoftwareInterface of the Experiment Softwarez 10x10 caption-based arrangement of 100 images of NY[[RoddenRoddenet al, CHI01]et al, CHI01]8Image ArrangementImage Arrangementz Multi-Dimensional Scaling (MDS)– treats inter-object dissimilarities as distances in some high dimensional space, and then attempts to approximate them in a low dimensional output.1. Create a similarity matrix2. Find 2D configuration of points using MDS3. Place thumbnail images at these points[[RoddenRoddenet al, CHI01]et al, CHI01]9Remove OverlapsRemove Overlapsz Arrangement of 100 images of Kenya, based on visual similarity. – Left: MDS arrangement, Right: 12x12 grid (remove overlap)[[RoddenRoddenet al, CHI01]et al, CHI01]10Change Grid SizeChange Grid Sizez Arrangement of 100 images of Kenya, based on visual similarity. – Left: MDS arrangement, Right: 10x10 grid (maximize thumbnail size)[[RoddenRoddenet al, CHI01]et al, CHI01]11The TaskThe Taskz The task was the same for both experimentsz Participants were given the following written description of it:You have been asked to choose photographs to illustrate a set of“destination guide” articles for a new “independent travel” World Wide Web site. Each article will be an overview of a different location, and is to appear on a separate page. The articles have not yet been written, so all you have are short summaries to indicate the general impression that each will convey. You also have 100 photographs of each location, and your task is to choose 3 of the photos (to be used together) for each article. It is entirely up to you to decide on the criteria you use to make your selections—there are no “right”answers, and you are not bound by the given summaries.12Experiment IExperiment Iz Goal:– If users find either of the similarity-based arrangement useful– If it was helpful to have both arrangements availablez Participants– 18 participants were all attendees of “infodesign 99”z Apparatus– For each of the four places: New York, Paris, Kenya and Alaska– Created two 12x12 grid arrangements of 100 images, (visual similarity and caption similarity).[[RoddenRoddenet al, CHI01]et al, CHI01]13ResultsResults4014First chose to search87 (ct > 85%)3 (ct<22%)Heavily favored in63%Caption 37%Average spent timenoVisual14Experiment IIExperiment IIz Goal:– If users would prefer a similarity-base arrangement to a random arrangement– If a similarity-based arrangement would help users to carry out the given task more quicklyz Participants– 10 students of graphic design from Anglia Polytechnic Univ.z Apparatus– Created two 10x10 grid arrangements of 100 images, (visual similarity and randomly) for each of 9 places.[[RoddenRoddenet al, CHI01]et al, CHI01]15ArrangementsArrangementsz Two 10x10 grids of 100 images of Brazil[[RoddenRoddenet al, CHI01]et al, CHI01]Left: randomly arranged Right:visual similarity16ProcedureProcedurez Part one: – Group X {Denmark, Jamaica and Nepal}– Group Y {Death Valley, Ireland and Kenya}z Part two:– Group Z {Brazil, Canada and Yellowstone National Park}– The students had to choose one of them to use initially, and could then switch between the two views as they wished......X rY v9thY vX r5thY vX vX rY vY rY rY rY vX rX v10th4th3rd2nd1st[[RoddenRoddenet al, CHI01]et al, CHI01]17ResultsResults[[RoddenRoddenet al, CHI01]et al, CHI01]921First chose to search32 (vt<22%)5 (vt>73%)Heavily favored in134%Random54Preference66%Average spent timebothVisual18DiscussionDiscussionz Arranging by similarity does seem to be usefulz A caption-based arrangement helps to break down the set– Its usefulness is affected by the level of details of the captionsz A visual-based arrangement helps to divide the set into simple genres– Cause adjacent images to appear to “merge”z Preferences may simply is due to individual differences[[RoddenRoddenet al, CHI01]et al, CHI01]19Semantic Image BrowserSemantic Image Browserz VAST 2006[Yang et al, VAST06][Yang et al, VAST06]Semantic Image Browser: Bridging Information Visualization with Automated Intelligent Image Analysis20OverviewOverview[Yang et al, VAST06][Yang et al, VAST06]Semantic Image AnalysisInformation Visualization(MDS, VaR)Semantic Image Browser21Automatic Annotation EngineAutomatic Annotation Enginez Concept-sensitive image content analysis techniquez Pre-defined salient object detect functions– Low-level automatic image segmentation– Classification by using Support Vector MachineFigure:The semantic image classification results for the concept “sea world”with the salient objects, such as “sand field”[Yang et al, VAST06][Yang et al, VAST06]22Image OverviewImage Overviewz MDS layoutz Interactions– Reordering– Dynamic scaling– Relocation– Distortion– Showing original image– Zooming and panning[Yang et al, VAST06][Yang et al, VAST06]Figure:An MDS image overview of Corel collection (1100 images)23Content OverviewContent Overviewz Generated by visualizing the content dataset in VaR display[Yang et al, VAST06][Yang et al, VAST06]24Interactions In Content Overview Interactions In Content Overview z VaR display provides a rich set of interaction tools– Clutter reduction– Reordering– Detection of correlationsz Combine search for images