Lecture 3: Feature detection and matchingAdministriviaReadingWhy do we flip the kernel?Feature extraction: Corners and blobsMotivation: Automatic panoramasMotivation: Automatic panoramasWhy extract features?Why extract features?Why extract features?Image matchingHarder caseHarder still?Answer below (look for tiny colored squares…)Feature MatchingFeature MatchingInvariant local featuresAdvantages of local featuresMore motivation… What makes a good feature?Want uniquenessLocal measures of uniquenessLocal measure of feature uniquenessHarris corner detection: the mathSmall motion assumptionCorner detection: the mathCorner detection: the mathThe second moment matrixSlide Number 29Slide Number 30General caseCorner detection: the mathCorner detection: the mathCorner detection: the mathInterpreting the eigenvaluesCorner detection summaryCorner detection summaryThe Harris operatorThe Harris operatorHarris detector examplef value (red high, blue low)Threshold (f > value) Find local maxima of fHarris features (in red)Weighting the derivativesQuestions?Image transformationsHarris Detector: Invariance PropertiesHarris Detector: Invariance PropertiesHarris Detector: Invariance PropertiesScale invariant detectionSlide Number 52Slide Number 53Slide Number 54Slide Number 55Slide Number 56Slide Number 57Slide Number 58Slide Number 59ImplementationAnother common definition of fLaplacian of GaussianScale selectionCharacteristic scaleScale-space blob detector: ExampleScale-space blob detector: ExampleScale-space blob detector: ExampleQuestions?Feature descriptorsFeature descriptorsInvariance vs. discriminabilityImage transformationsInvarianceHow to achieve invarianceRotation invariance for feature descriptorsMultiscale Oriented PatcheS descriptorDetections at multiple scalesScale Invariant Feature TransformSIFT descriptorProperties of SIFTMaximally Stable Extremal RegionsFeature matchingFeature distanceSlide Number 84Evaluating the resultsTrue/false positivesEvaluating the resultsEvaluating the resultsMore on feature detection/descriptionLots of applicationsObject recognition (David Lowe)3D ReconstructionSlide Number 93Questions?Assignment 1: Feature detection and matchingLecture 3: Feature detection and matchingCS6670: Computer VisionNoah SnavelyAdministrivia• New location: please sit in the front rows• Assignment 1 (feature detection and matching) will be released right after class, due Thursday, September 24 by 11:59pm– More details at the end of lectureReading• Szeliski: 4.1Why do we flip the kernel?• Convolution is commutative• Cross‐correlation is noncommutativeFeature extraction: Corners and blobsMotivation: Automatic panoramasCredit: Matt Brownhttp://research.microsoft.com/en‐us/um/redmond/groups/ivm/HDView/HDGigapixel.htmHD ViewAlso see GigaPan: http://gigapan.org/Motivation: Automatic panoramasWhy extract features?• Motivation: panorama stitching– We have two images –how do we combine them?Why extract features?• Motivation: panorama stitching– We have two images –how do we combine them?Step 1: extract featuresStep 2: match featuresWhy extract features?• Motivation: panorama stitching– We have two images –how do we combine them?Step 1: extract featuresStep 2: match featuresStep 3: align imagesImage matchingby Diva Sianby swashfordHarder caseby Diva Sian by scgbtHarder still?NASA Mars Rover imagesNASA Mars Rover imageswith SIFT feature matchesAnsw er below (look for tiny color ed squares…)Feature MatchingFeature MatchingInvariant local featuresFind features that are invariant to transformations– geometric invariance: translation, rotation, scale– photometric invariance: brightness, exposure, …Feature DescriptorsAdvantage s of local featuresLoca lity – features are local, so robust to occlusion and clutterQuantity– hundreds or thousands in a single imageDistinctiveness: – can differentiate a large database of objectsEfficiency– real‐time performance achievableMore motivation… Feature points are used for:– Image alignment (e.g., mosaics)– 3D reconstruction– Motion tracking– Object recognition– Indexing and database retrieval– Robot navigation– … otherSnoop demoWhat makes a good feature?Want uniquenessLook for image regions that are unusual– Lead to unambiguous matches in other imagesHow to define “unusual”?Local measures of uniquenessSuppose we only consider a small window of pixels– What defines whether a feature is a good or bad candidate?Credit: S. Seitz, D. Frolova, D. SimakovLocal measure of feature uniqueness“flat” region:no change in all directions“edge”: no change along the edge direction“corner”:significant change in all directions• How does the window change when you shift it?• Shifting the window in any direction causes a big changeCredit: S. Seitz, D. Frolova, D. SimakovConsider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after bysumming up the squared differences (SSD)• this defines an SSD “error” E(u,v):Harris corner detection: the mathWTaylor Series expansion of I:If the motion (u,v) is small, then first order approximation is goodPlugging this into the formula on the previous slide…Small motion assumptionCorner detection: the mathConsider shifting the window W by (u,v)• define an SSD “error” E(u,v):WCorner detection: the mathConsider shifting the window W by (u,v)• define an SSD “error” E(u,v):W• Thus, E(u,v) is locally approximated as a quadratic error functionThe surface E(u,v) is locally approximated by a quadratic form. The second moment matrixLet’s try to understand its shape.Horizontal edge: uvE(u,v)Vertical edge: uvE(u,v)General caseWe can visualize H as an ellipse with axis lengths determined by the eigenvalues of H and orientation determined by the eigenvectors of Hdirection of the slowest changedirection of the fastest change(λmax)-1/2(λmin)-1/2const][ =⎥⎦⎤⎢⎣⎡vuHvuEllipse equation:λmax, λmin: eigenvalues of HCorner detection: the mathEigenvalues and eigenvectors of H• Define shift directions with the smallest and largest change in error• xmax= direction of largest increase in E• λmax= amount of increase in direction xmax• xmin= direction of smallest increase in E• λmin= amount of increase in direction xminxminxmaxCorner detection: the mathHow are λmax,xmax, λmin,and xminrelevant for feature detection?• What’s our feature scoring