Geotagging ImagesA Short SurveyBen TribelhornApril 8, 2010PapersIM2GPS: estimating geographic information from a single image (Hays & Efros, CVPR 2008)Image Sequence Geolocation with Human Travel Priors (Kalogerakis, et al., ICCV 2009)http://www.dgp.toronto.edu/~kalo/papers/images2gps/http://graphics.cs.cmu.edu/projects/im2gps/Problem StatementInput: Any ImageOutput: GPS coordinates or P(L | I)“Easy” “Moderate”“Hard”DublinVatican CitySimilar WorkVisual localization using geometric reasoning (Thompson et. el., 1999)Association of video with satellite weather (Jacobs et. el., 2007)Urban (Zhang & Kosecka 2006, Schindler 2008)Geometric matching with famous landmarks (Snavely 2006, Crandall 2009, Zheng 2009)Regional (Cristani 2008)OverviewMotivation: Large amounts of emerging geographically-calibrated image data.Steps:Build a large databasePrecompute features for DBMatchingContributionsShows potential of cluster computingSecondary geographic tasks:Estimating population densityLand cover estimates“Which images were from my India trip?”Urban/rural classification& more!Building Geo-tag DBFlickr - Provided script, uses API-keyMatlab script downloads images separatelyMinimum dimension is 500 pixels, max 1024DB was 1TB (vs. 2.5GB when redone)NOT SUPERVISEDDB ProblemsDespite prohibited keywords, faces are in a huge portion of images (over 50% for Corvallis).Doesn’t do face recognitionLabels are often vague, always in English, ambiguous (eg Washington, Georgia), etc.Style (eg person vs. architecture focus)DB ProblemsDB isn’t geographically uniform“Pretty” Landscapes are very similar (eg. sunsets), not geo-specificIndoor photos are potentially less geo-specificSlow/simplistic constructionSingle photographerLocations chosen to be in DBHand picked 72 imagesAll full size (max dim 1024)Test SetScene MatchingTiny Images - 16x16 (Torralba)Line Features - Canny edges (Kosecka & Zhang)Geometric Context - Class probs (Hoiem)Color histograms*Texton Histograms*Gist Descriptor*3.08 CPU Years for O(6 million) images(2008 computers)Color HistogramsColor is very descriptiveBut, it can be very misleadingCIE L*a*b - 4, 14, 14 bins = 784 dimensionsχ2 DistanceOpenCVTexton HistogramsFilter responses512 entry universal texton dictionary. I chose to use a provided set of textons.χ2 DistanceModified Matlab code from Martin et. al. (Very slow calculations)http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/Gist DescriptorEssentially PCAMultiple orientations and multiple scales produces a “histogram” of 960 filter responses. (Different than im2gps).Their approach included a 5x5 color image, which seems redundant?Modified code from Torralbahttp://people.csail.mit.edu/torralba/code/spatialenvelope/Matching?1st Nearest Neighbork-NN (They used 120)Overall 1-NN is comparable to Mean Shift clusteringError CombinationHow did they “add” together the errors from each feature detection method? Not specified.I reduced the χ2‘s to MSE by histogram size accounted for average image size.Then the approximately [0,1] values were summed for an overall “match.”Feature weight learning would increase performance.Results @ CMUBest Features1-NN Better than MS for region res.Continent cut-off:~40% correctCity 16 miRegion 125Country 470Cont. 1560My ResultsCity Identification 10%Region Identification 15% (same as previous)Input: Pembroke, Wales DB: NYCCity IdentificationDB InputDBInputInputInputInputDBSame LocationsSame CityDBCity IdentificationDB InputInputInputDBDBPompeiiCatastrophic FailuresNYC?Rome?Bottom LineWas this method reproducible? YES.Is it reasonable?Failures: DB is terrible. eg. nyc tag had pictures of the inside of an airplaneImprovements: Face detection, fewer features, stricter keyword search, supervisionPerfect application for EM learning: features and weights.FIXEDFIXEDFIXED2009 SEQApproachImage Sequence Geolocation with Human Travel PriorsEvangelos Kalogerakis, Olga Vesselova, James Hays, Alexei A. Efros, Aaron HertzmannRevised Problem StatementAdditional input: Time elapsedStillImage SequencesMore information, virtually guaranteed by all cameras (unlike GPS tags)Use HMM to relate & extractSecondary geographic tasks:Epidemic forecastingUrban PlanningNeed to find: Bayes Rule:P (Lk= i | Ik)P (Lk= i)p(Ik| L = i)=P (Lk= i | Ik)p(Ik)P (Lk= i)1)2)HMM Setup3)P (Lk= j | Lk−1, ∆Tk= τ )Human Travel DistributionsLévy flight model (power law)Many short trips, some longInvariant to start/end locationswheresgeorge.com (Random walks)Mobile phone tracesP (Lk= j | Lk−1, ∆Tk= τ )Spatially-varying DistributionPijτ ~ # pairs (i, j) / # pairs that end at jSpatially-varying DistributionAlso fills in under sampled long-distance bins from short trips:ΣaPia(τ −1)Paj(τ −1)P’ijτ = Pijτ +Single-image priorNormalized counts over image database.P (Lk= i)Image LikelihoodmatchesdistanceNormalize by best M=60Image LikelihoodDistance D(I, Im) between images is L2 distance of:Gist descriptorColor histogramsTexton histogramsLine histogramP (Lk= i | Ik)Reduced to most effective descriptors from prior work.Informative?SynergyUsing the HMMForward-Backward belief propagationLandmark recognitionCombined ReasoningResultsCorrect within 400km (regional) for test set:London always3%IM2GPS10%Sequence58%Landmark only41%Landmark-less19%Uniform prob. for “non-distinctive” imagesMost common in DBUniform prob. for “distinctive” imagesImprovementsBetter binningBetter image matchingMore general models for meta-data, Flickr tags, user types, image types, weather,