Estimating Misclassification ProbabilityEstimating and Comparing ClassifiersConfidence Intervals95% Confidence IntervalsComparison of Classifier AccuracyProblem Average Error Rate:Estimation of parameters in small sample size casesEstimating Misclassification ProbabilityEstimating and Comparing ClassifiersCross Validation: Divide training set into m disjoint sets of equal sizeClassifier is trained m timesEstimated performance is mean on m errorsConfidence Intervals• If true but unknown error rate of classifier is p• k of the n independent randomly drawn samples are misclassified then k has the binomial distribution• Maximum likelihood estimate for p isknkppknkP−−⎟⎟⎠⎞⎜⎜⎝⎛= )1()(nkp =∧95% Confidence IntervalsComparison of Classifier Accuracy• Jackknife estimates of classification accuracy are 80% and 85%• Full widths (2*std dev) are 12% and 15%• Standard Hypothesis Testing can show that it is not statistically significantProblem Average Error Rate:Estimation of parameters in small sample size
View Full Document
Unlocking...