DOC PREVIEW
Purdue CS 59000 - Midterm Review

This preview shows page 1-2-3-4-5-6-7-8-59-60-61-62-63-64-65-66-67-120-121-122-123-124-125-126-127 out of 127 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 127 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

CS 59000 Statistical Machine learning Lecture 18: Midterm ReviewOverview Polynomial Curve Fitting 1st Order Polynomial3rd Order Polynomial9th Order PolynomialOver-fittingPolynomial Coefficients RegularizationRegularization: Regularization: Polynomial Coefficients Probability TheoryThe Rules of ProbabilityBayes’ TheoremProbability DensitiesTransformed DensitiesExpectationsThe Gaussian DistributionThe Multivariate GaussianMaximum (Log) LikelihoodProperties of and Maximum LikelihoodPredictive DistributionMAP: A Step towards BayesBayesian Curve FittingBayesian Predictive DistributionCurse of DimensionalityDecision TheoryMinimum Misclassification RateWhy Separate Inference and Decision?Decision Theory for RegressionThe Squared Loss FunctionEntropyEntropyEntropyDifferential EntropyThe Kullback-Leibler DivergenceMutual InformationProbabilistic DistributionsParameter EstimationThe Exponential Family (1)The Exponential Family (2.1)The Exponential Family (2.2)The Exponential Family (3.1)The Exponential Family (3.2)The Exponential Family (3.3)The Exponential Family (4)ML for the Exponential Family (1)ML for the Exponential Family (2)Conjugate priorsPosterior of Gaussian mean parameter Choice of Priors Nonparametric Methods (1)Nonparametric Methods (2)Nonparametric Methods (3)Nonparametric Methods (4)Nonparametric Methods (5)Nonparametric Methods (6)K-Nearest-Neighbours for Classification (1)Linear RegressionBasis FunctionsExamples of Basis Functions (1)Maximum Likelihood Estimation (1)Maximum Likelihood Estimation (2)Maximum Likelihood Estimation (3)Maximum Likelihood Estimation (4)Maximum Likelihood Estimation (2)Sequential EstimationRegularized Least SquaresMore Regularizers Bayesian Linear RegressionPosterior Distributions of ParametersPredictive Posterior DistributionBayesian Model Comparison Likelihood, Parameter Posterior & EvidenceCrude Evidence ApproximationEvidence penalizes over-complex modelsEvidence Approximation & Empirical BayesLinear Classification Three ApproachesGenerative vs DiscriminativeDistance from to decision surfaceFisher’s Linear Discriminant Fisher Linear DiscriminantCost FunctionWithin Class and Between Class Scatter MatricesGenerative eigenvalue problemFisher’s Linear DiscriminantPerceptron Stochastic Gradient Descent Probabilistic Generative ModelsGaussian Class-Conditional DensitiesMaximum Likelihood EstimationMaximum Likelihood EstimationDiscrete featuresLogistic RegressionMaximum Likelihood EstimationNewton-Raphson Optimization for Logistic Regression Newton-Raphson Optimization for Logistic RegressionProbit RegressionLabeling Noise ModelLaplace Approximation for PosteriorEvidence ApproximationBayesian Information CriterionBayesian Logistic RegressionKernel MethodsKernel TrickDual Representation for Ridge RegressionKernel Ridge RegressionCombining Generative & Discriminative Models by KernelsMeasure Probability Similarity by Kernels Principle Component Analysis (PCA)Feature MappingDual VariablesEigen-problem in Feature Space (1)Eigen-problem in Feature Space (2)General Case for Non-zero Mean CaseGaussian ProcessesLinear Regression RevisitedFrom Prior on Parameter to Prior on FunctionStochastic ProcessGaussian ProcessesGaussian Process for RegressionSamples of GP Prior over FunctionsPredictive DistributionPredictive MeanCS 59000 Statistical Machine learningLecture 18: Midterm ReviewYuan (Alan) QiPurduce CS590 ‐‐ Yuan Qi 2011OverviewOverfitting, probabilities, decision theory, entropy and KL divergence, ML and Bayesian estimation of Gaussian and multinomial distributions, exponential family, conjugate priorsNonparametric methods: Parzen window and k‐nearest neighbor classifiersRegression: ML and Bayesian approaches for linear regression, Lasso, evidenceClassification: Fisher linear discriminant, Perceptron, generative models, naïve Bayes, conditional models, logistic regression, Laplace approximationKernel methods: kernel ridge regression, kernel PCAGaussian processes (GP): GP regression and classificationPurduce CS590 ‐‐ Yuan Qi 2011Polynomial Curve FittingPurduce CS590 ‐‐ Yuan Qi 20111stOrder PolynomialPurduce CS590 ‐‐ Yuan Qi 20113rdOrder PolynomialPurduce CS590 ‐‐ Yuan Qi 20119thOrder PolynomialPurduce CS590 ‐‐ Yuan Qi 2011Over‐fittingRoot‐Mean‐Square (RMS) Error:Purduce CS590 ‐‐ Yuan Qi 2011Polynomial Coefficients Purduce CS590 ‐‐ Yuan Qi 2011RegularizationPenalize large coefficient valuesPurduce CS590 ‐‐ Yuan Qi 2011Regularization: Purduce CS590 ‐‐ Yuan Qi 2011Regularization: Purduce CS590 ‐‐ Yuan Qi 2011Polynomial Coefficients Purduce CS590 ‐‐ Yuan Qi 2011Probability TheorySum RuleProduct RulePurduce CS590 ‐‐ Yuan Qi 2011The Rules of ProbabilitySum RuleProduct RulePurduce CS590 ‐‐ Yuan Qi 2011Bayes’ Theoremposterior ∝ likelihood × priorPurduce CS590 ‐‐ Yuan Qi 2011Probability DensitiesPurduce CS590 ‐‐ Yuan Qi 2011Transformed DensitiesPurduce CS590 ‐‐ Yuan Qi 2011ExpectationsConditional Expectation(discrete)Approximate Expectation(discrete and continuous)Purduce CS590 ‐‐ Yuan Qi 2011The Gaussian DistributionPurduce CS590 ‐‐ Yuan Qi 2011The Multivariate GaussianPurduce CS590 ‐‐ Yuan Qi 2011Maximum (Log) LikelihoodPurduce CS590 ‐‐ Yuan Qi 2011Properties of and Purduce CS590 ‐‐ Yuan Qi 2011Maximum LikelihoodDetermine by minimizing sum‐of‐squares error, .Purduce CS590 ‐‐ Yuan Qi 2011Predictive DistributionPurduce CS590 ‐‐ Yuan Qi 2011MAP: A Step towards BayesDetermine by minimizing regularized sum‐of‐squares error, .Purduce CS590 ‐‐ Yuan Qi 2011Bayesian Curve FittingPurduce CS590 ‐‐ Yuan Qi 2011Bayesian Predictive DistributionPurduce CS590 ‐‐ Yuan Qi 2011Curse of DimensionalityPolynomial curve fitting, M = 3Gaussian Densities in higher dimensionsPurduce CS590 ‐‐ Yuan Qi 2011Decision TheoryInference stepDetermine either or .Decision stepFor given x, determine optimal t.Purduce CS590 ‐‐ Yuan Qi 2011Minimum Misclassification RatePurduce CS590 ‐‐ Yuan Qi 2011Why Separate Inference and Decision?• Minimizing risk (loss matrix may change over time)• Reject option• Unbalanced class priors• Combining modelsPurduce CS590 ‐‐ Yuan Qi 2011Decision Theory for RegressionInference stepDetermine .Decision stepFor given x, make optimal prediction, y(x), for t.Loss function:Purduce CS590 ‐‐ Yuan Qi


View Full Document

Purdue CS 59000 - Midterm Review

Documents in this Course
Lecture 4

Lecture 4

42 pages

Lecture 6

Lecture 6

38 pages

Load more
Download Midterm Review
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Midterm Review and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Midterm Review 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?