1CS 188: Artificial IntelligenceFall 2007Lecture 26: Kernels11/29/2007Dan Klein – UC BerkeleyFeature Extractors A feature extractor maps inputs to feature vectors Many classifiers take feature vectors as inputs Feature vectors usually very sparse, use sparse encodings (i.e. only represent non-zero keys)Dear Sir.First, I must solicit your confidence in this transaction, this is by virture of its nature as being utterly confidencial and top secret. …W=dear : 1W=sir : 1W=this : 2...W=wish : 0...MISSPELLED : 2NAMELESS : 1ALL_CAPS : 0NUM_URLS : 0...2The Perceptron Update Rule Start with zero weights Pick up training instances one by one Try to classify If correct, no change! If wrong: lower score of wrong answer, raise score of right answerNearest-Neighbor Classification Nearest neighbor for digits: Take new image Compare to all training images Assign based on closest example Encoding: image is vector of intensities: What’s the similarity function? Dot product of two images vectors? Usually normalize vectors so ||x|| = 1 min = 0 (when?), max = 1 (when?)3Basic Similarity Many similarities based on feature dot products: If features are just the pixels: Note: not all similarities are of this formInvariant MetricsThis and next few slides adapted from Xiao Hu, UIUC Better distances use knowledge about vision Invariant metrics: Similarities are invariant under certain transformations Rotation, scaling, translation, stroke-thickness… E.g: 16 x 16 = 256 pixels; a point in 256-dim space Small similarity in R256 (why?) How to incorporate invariance into similarities?4Rotation Invariant Metrics Each example is now a curve in R256 Rotation invariant similarity:s’=max s( r( ), r( )) E.g. highest similarity between images’ rotation linesTemplate Deformation Deformable templates: An “ideal” version of each category Best-fit to image using min variance Cost for high distortion of template Cost for image points being far from distorted template Used in many commercial digit recognizersExamples from [Hastie 94]5A Tale of Two Approaches… Nearest neighbor-like approaches Can use fancy kernels (similarity functions) Don’t actually get to do explicit learning Perceptron-like approaches Explicit training to reduce empirical error Can’t use fancy kernels (why not?) Or can you? Let’s find out!The Perceptron, Again Start with zero weights Pick up training instances one by one Try to classify If correct, no change! If wrong: lower score of wrong answer, raise score of right answer6Perceptron Weights What is the final value of a weight wc? Can it be any real vector? No! It’s built by adding up inputs. Can reconstruct weight vectors (the primal representation) from update counts (the dual representation)Dual Perceptron How to classify a new example x? If someone tells us the value of K for each pair of examples, never need to build the weight vectors!7Dual Perceptron Start with zero counts (alpha) Pick up training instances one by one Try to classify xn, If correct, no change! If wrong: lower count of wrong class (for this instance), raise score of right class (for this instance)Kernelized Perceptron If we had a black box (kernel) which told us the dot product of two examples x and y: Could work entirely with the dual representation No need to ever take dot products (“kernel trick”) Like nearest neighbor – work with black-box similarities Downside: slow if many examples get nonzero alpha8Kernelized Perceptron StructureKernels: Who Cares? So far: a very strange way of doing a very simple calculation “Kernel trick”: we can substitute any* similarity function in place of the dot product Lets us learn new kinds of hypothesis* Fine print: if your kernel doesn’t satisfy certain technical requirements, lots of proofs break. E.g. convergence, mistake bounds. In practice, illegal kernels sometimes work (but not always).9Properties of Perceptrons Separability: some parameters get the training set perfectly correct Convergence: if the training is separable, perceptron will eventually converge (binary case) Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separabilitySeparableNon-SeparableNon-Linear Separators Data that is linearly separable (with some noise) works out great: But what are we going to do if the dataset is just too hard? How about… mapping data to a higher-dimensional space:000x2xxxThis and next few slides adapted from Ray Mooney, UT10Non-Linear Separators General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable:Φ: x → φ(x)Some Kernels Kernels implicitly map original vectors to higher dimensional spaces, take the dot product there, and hand the result back Linear kernel: Quadratic kernel: RBF: infinite dimensional representation Discrete kernels: e.g. string kernels11Recap: Classification Classification systems: Supervised learning Make a rational prediction given evidence We’ve seen several methods for this Useful when you have labeled data (or can get it)Clustering Clustering systems: Unsupervised learning Detect patterns in unlabeled data E.g. group emails or search results E.g. find categories of customers E.g. detect anomalous program executions Useful when don’t know what you’re looking for Requires data, but no labels Often get gibberish12Clustering Basic idea: group together similar instances Example: 2D point patterns What could “similar” mean? One option: small (squared) Euclidean distanceK-Means An iterative clustering algorithm Pick K random points as cluster centers (means) Alternate: Assign data instances to closest mean Assign each mean to the average of its assigned points Stop when no points’ assignments change13K-Means ExampleK-Means as Optimization Consider the total distance to the means: Each iteration reduces phi Two stages each iteration: Update assignments: fix means c,change assignments a Update means: fix assignments a,change means cpointsassignmentsmeans14Phase I: Update Assignments For each point, re-assign to closest
View Full Document
Unlocking...