CS 188: Artificial Intelligence Fall 2007Feature ExtractorsThe Binary PerceptronExample: SpamBinary Decision RuleThe Multiclass PerceptronExampleThe Perceptron Update RuleSlide 9Examples: PerceptronMistake-Driven ClassificationProperties of PerceptronsSlide 14Issues with PerceptronsLinear SeparatorsSupport Vector MachinesSummarySimilarity FunctionsCase-Based ReasoningParametric / Non-parametricNearest-Neighbor ClassificationBasic SimilarityInvariant MetricsRotation Invariant MetricsTangent FamiliesTemplate DeformationA Tale of Two Approaches…The Perceptron, AgainPerceptron WeightsDual PerceptronSlide 34Kernelized PerceptronKernelized Perceptron StructureKernels: Who Cares?Slide 38Non-Linear SeparatorsSlide 40Some KernelsCS 188: Artificial IntelligenceFall 2007Lecture 25: Kernels11/27/2007Dan Klein – UC BerkeleyFeature ExtractorsA feature extractor maps inputs to feature vectorsMany classifiers take feature vectors as inputsFeature vectors usually very sparse, use sparse encodings (i.e. only represent non-zero keys)Dear Sir.First, I must solicit your confidence in this transaction, this is by virture of its nature as being utterly confidencial and top secret. …W=dear : 1W=sir : 1W=this : 2...W=wish : 0...MISSPELLED : 2NAMELESS : 1ALL_CAPS : 0NUM_URLS : 0...The Binary PerceptronInputs are featuresEach feature has a weightSum is the activationIf the activation is:Positive, output 1Negative, output 0f1f2f3w1w2w3>0?Example: SpamImagine 4 features:Free (number of occurrences of “free”)Money (occurrences of “money”)BIAS (always has value 1)BIAS : -3free : 4money : 2the : 0 ...BIAS : 1 free : 1money : 1the : 0...“free money”Binary Decision RuleIn the space of feature vectorsAny weight vector is a hyperplaneOne side will be class 1Other will be class 0BIAS : -3free : 4money : 2the : 0 ...0 1012freemoney1 = SPAM0 = HAMThe Multiclass PerceptronIf we have more than two classes:Have a weight vector for each classCalculate an activation for each classHighest activation winsExampleBIAS : -2win : 4game : 4vote : 0the : 0 ...BIAS : 1win : 2game : 0vote : 4the : 0 ...BIAS : 2win : 0game : 2vote : 0the : 0 ...“win the vote”BIAS : 1win : 1game : 0vote : 1the : 1...The Perceptron Update RuleStart with zero weightsPick up training instances one by oneTry to classifyIf correct, no change!If wrong: lower score of wrong answer, raise score of right answerExampleBIAS :win : game : vote : the : ...BIAS : win : game : vote : the : ...BIAS : win : game : vote : the : ...“win the vote”“win the election”“win the game”Examples: PerceptronSeparable CaseMistake-Driven ClassificationIn naïve Bayes, parameters:From data statisticsHave a causal interpretationOne pass through the dataFor the perceptron parameters:From reactions to mistakesHave a discriminative interpretationGo through the data until held-out accuracy maxes outTrainingDataHeld-OutDataTestDataProperties of PerceptronsSeparability: some parameters get the training set perfectly correctConvergence: if the training is separable, perceptron will eventually converge (binary case)Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separabilitySeparableNon-SeparableExamples: PerceptronNon-Separable CaseIssues with PerceptronsOvertraining: test / held-out accuracy usually rises, then fallsOvertraining isn’t quite as bad as overfitting, but is similarRegularization: if the data isn’t separable, weights might thrash aroundAveraging weight vectors over time can help (averaged perceptron)Mediocre generalization: finds a “barely” separating solutionLinear SeparatorsWhich of these linear separators is optimal?Support Vector MachinesMaximizing the margin: good according to intuition and PAC theory.Only support vectors matter; other training examples are ignorable. Support vector machines (SVMs) find the separator with max marginMathematically, gives a quadratic program to solveBasically, SVMs are perceptrons with smarter update counts!SummaryNaïve BayesBuild classifiers using model of training dataSmoothing estimates is important in real systemsClassifier confidences are useful, when you can get themPerceptrons:Make less assumptions about dataMistake-driven learningMultiple passes through dataSimilarity FunctionsSimilarity functions are very important in machine learningTopic for next class: kernelsSimilarity functions with special propertiesThe basis for a lot of advance machine learning (e.g. SVMs)Case-Based ReasoningSimilarity for classificationCase-based reasoningPredict an instance’s label using similar instancesNearest-neighbor classification1-NN: copy the label of the most similar data pointK-NN: let the k nearest neighbors vote (have to devise a weighting scheme)Key issue: how to define similarityTrade-off:Small k gives relevant neighborsLarge k gives smoother functionsSound familiar?[DEMO]http://www.cs.cmu.edu/~zhuxj/courseproject/knndemo/KNN.htmlParametric / Non-parametricParametric models:Fixed set of parametersMore data means better settingsNon-parametric models:Complexity of the classifier increases with dataBetter in the limit, often worse in the non-limit(K)NN is non-parametricTruth2 Examples10 Examples 100 Examples 10000 ExamplesNearest-Neighbor ClassificationNearest neighbor for digits:Take new imageCompare to all training imagesAssign based on closest exampleEncoding: image is vector of intensities:What’s the similarity function?Dot product of two images vectors?Usually normalize vectors so ||x|| = 1min = 0 (when?), max = 1 (when?)Basic SimilarityMany similarities based on feature dot products:If features are just the pixels:Note: not all similarities are of this formInvariant MetricsThis and next few slides adapted from Xiao Hu, UIUCBetter distances use knowledge about visionInvariant metrics:Similarities are invariant under certain transformationsRotation, scaling, translation, stroke-thickness…E.g: 16 x 16 = 256 pixels; a point in 256-dim spaceSmall similarity in R256 (why?)How to incorporate invariance into
View Full Document
Unlocking...