1CS 188: Artificial IntelligenceFall 2007Lecture 25: Kernels11/27/2007Dan Klein – UC BerkeleyFeature Extractors A feature extractor maps inputs to feature vectors Many classifiers take feature vectors as inputs Feature vectors usually very sparse, use sparse encodings (i.e. only represent non-zero keys)Dear Sir.First, I must solicit your confidence in this transaction, this is by virture of its nature as being utterly confidencial and top secret. …W=dear : 1W=sir : 1W=this : 2...W=wish : 0...MISSPELLED : 2NAMELESS : 1ALL_CAPS : 0NUM_URLS : 0...2The Binary Perceptron Inputs are features Each feature has a weight Sum is the activation If the activation is: Positive, output 1 Negative, output 0Σf1f2f3w1w2w3>0?Example: Spam Imagine 4 features: Free (number of occurrences of “free”) Money (occurrences of “money”) BIAS (always has value 1)BIAS : -3free : 4money : 2the : 0 ...BIAS : 1 free : 1money : 1the : 0...“free money”3Binary Decision Rule In the space of feature vectors Any weight vector is a hyperplane One side will be class 1 Other will be class 0BIAS : -3free : 4money : 2the : 0 ...0 1012freemoney1 = SPAM0 = HAMThe Multiclass Perceptron If we have more than two classes: Have a weight vector for each class Calculate an activation for each class Highest activation wins4ExampleBIAS : -2win : 4game : 4vote : 0the : 0 ...BIAS : 1win : 2game : 0vote : 4the : 0 ...BIAS : 2win : 0game : 2vote : 0the : 0 ...“win the vote”BIAS : 1win : 1game : 0vote : 1the : 1...The Perceptron Update Rule Start with zero weights Pick up training instances one by one Try to classify If correct, no change! If wrong: lower score of wrong answer, raise score of right answer5ExampleBIAS :win : game : vote : the : ...BIAS : win : game : vote : the : ...BIAS : win : game : vote : the : ...“win the vote”“win the election”“win the game”Examples: Perceptron Separable Case6Mistake-Driven Classification In naïve Bayes, parameters: From data statistics Have a causal interpretation One pass through the data For the perceptron parameters: From reactions to mistakes Have a discriminative interpretation Go through the data until held-out accuracy maxes outTrainingDataHeld-OutDataTestDataProperties of Perceptrons Separability: some parameters get the training set perfectly correct Convergence: if the training is separable, perceptron will eventually converge (binary case) Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separabilitySeparableNon-Separable7Examples: Perceptron Non-Separable CaseIssues with Perceptrons Overtraining: test / held-out accuracy usually rises, then falls Overtraining isn’t quite as bad as overfitting, but is similar Regularization: if the data isn’t separable, weights might thrash around Averaging weight vectors over time can help (averaged perceptron) Mediocre generalization: finds a “barely” separating solution8Linear Separators Which of these linear separators is optimal? Support Vector Machines Maximizing the margin: good according to intuition and PAC theory. Only support vectors matter; other training examples are ignorable. Support vector machines (SVMs) find the separator with max margin Mathematically, gives a quadratic program to solve Basically, SVMs are perceptrons with smarter update counts!9Summary Naïve Bayes Build classifiers using model of training data Smoothing estimates is important in real systems Classifier confidences are useful, when you can get them Perceptrons: Make less assumptions about data Mistake-driven learning Multiple passes through dataSimilarity Functions Similarity functions are very important in machine learning Topic for next class: kernels Similarity functions with special properties The basis for a lot of advance machine learning (e.g. SVMs)10Case-Based Reasoning Similarity for classification Case-based reasoning Predict an instance’s label using similar instances Nearest-neighbor classification 1-NN: copy the label of the most similar data point K-NN: let the k nearest neighbors vote (have to devise a weighting scheme) Key issue: how to define similarity Trade-off: Small k gives relevant neighbors Large k gives smoother functions Sound familiar? [DEMO]http://www.cs.cmu.edu/~zhuxj/courseproject/knndemo/KNN.htmlParametric / Non-parametric Parametric models: Fixed set of parameters More data means better settings Non-parametric models: Complexity of the classifier increases with data Better in the limit, often worse in the non-limit (K)NN is non-parametricTruth2 Examples10 Examples 100 Examples 10000 Examples11Nearest-Neighbor Classification Nearest neighbor for digits: Take new image Compare to all training images Assign based on closest example Encoding: image is vector of intensities: What’s the similarity function? Dot product of two images vectors? Usually normalize vectors so ||x|| = 1 min = 0 (when?), max = 1 (when?)Basic Similarity Many similarities based on feature dot products: If features are just the pixels: Note: not all similarities are of this form12Invariant MetricsThis and next few slides adapted from Xiao Hu, UIUC Better distances use knowledge about vision Invariant metrics: Similarities are invariant under certain transformations Rotation, scaling, translation, stroke-thickness… E.g: 16 x 16 = 256 pixels; a point in 256-dim space Small similarity in R256 (why?) How to incorporate invariance into similarities?Rotation Invariant Metrics Each example is now a curve in R256 Rotation invariant similarity:s’=max s( r( ), r( )) E.g. highest similarity between images’ rotation lines13Tangent Families Problems with s’: Hard to compute Allows large transformations (6 → 9) Tangent distance: 1st order approximation at original points. Easy to compute Models small rotations Template Deformation Deformable templates: An “ideal” version of each category Best-fit to image using min variance Cost for high distortion of template Cost for image points
View Full Document