1Lecture note for Stat 231: Pattern Recognition and Machine LearningLecture 8K-Nearest Neighbor (K-nn) method1. K-nn density estimation --- estimate the prior and class models2. K-nn classification --- estimate the posterior directly3. K-nn decision rule 4. k-nn error analysisLecture note for Stat 231: Pattern Recognition and Machine LearningSummaryThe Parzen window method is an unbiased estimate of the underlying density p(x)2Lecture note for Stat 231: Pattern Recognition and Machine LearningParzen window vs K-NN--- a comparisonGiven n samples, in a non-parametric method, we estimate the density at x bynnnVnkxp/)( =Design I: Parzen window. Fix the window size and kn=kn(x) is a function of x.nVxkVnkxpnnnn1)(/)( ==nVVn1=Design II: K-nn method. Fix the number of sample inside window and Vn=Vn(x) --- a function of x.nxVVnkxpnnnn)(1/)( ==nKn=Lecture note for Stat 231: Pattern Recognition and Machine LearningParzen window vs K-NN—a comparisonParzen windownVxkVnkxpnnnn1)(/)( ==K-nnnKn=nVVn1=∑=−=nininhxxxK1)()(ϕ1. Here the number of samples falling a window can be counted explicitly2. We can prove the convergence1. The window size does not have an explicitly form. Very irregular plot2. The integration of pn(x) is infinitynxVVnkxpnnnn)(1/)( ==∞=∫dxxpn)(},...,{|,|2)(1**nKKnxxxxxxV ∈−=3Lecture note for Stat 231: Pattern Recognition and Machine LearningExamples of K-nn densitynxVVnkxpnnnn)(1/)( ==||21)(11xxxp−=Lecture note for Stat 231: Pattern Recognition and Machine LearningNon-parametric Classification4Lecture note for Stat 231: Pattern Recognition and Machine LearningNon-parametric ClassificationIt turns out that the K-nn classification is also a special case of Bayes decision ! Lecture note for Stat 231: Pattern Recognition and Machine LearningNearest Neighbor Decision Rule5Lecture note for Stat 231: Pattern Recognition and Machine LearningNearest Neighbor Decision RuleLecture note for Stat 231: Pattern Recognition and Machine LearningNearest Neighbor Decision Rule6Lecture note for Stat 231: Pattern Recognition and Machine LearningRandomized decisionIt is randomized not by our decision,but by the random nature of the samples !Lecture note for Stat 231: Pattern Recognition and Machine LearningTwo Special Cases7Lecture note for Stat 231: Pattern Recognition and Machine LearningNN error analysisLecture note for Stat 231: Pattern Recognition and Machine LearningNN error analysis8Lecture note for Stat 231: Pattern Recognition and Machine LearningNN error analysisLecture note for Stat 231: Pattern Recognition and Machine LearningNN error analysis9Lecture note for Stat 231: Pattern Recognition and Machine LearningKnn-Decision
View Full Document