CS 59000 Machine learning Lecture 2Review: Polynomial Curve Fitting Sum-of-Squares Error Function1st Order Polynomial3rd Order Polynomial9th Order PolynomialOver-fittingPolynomial Coefficients RegularizationRegularization: Regularization: Regularization: vs. Polynomial Coefficients Data Set Size: N = 10 Data Set Size: Training Data: the more, the better... Review: Probability TheoryProbability TheoryThe Rules of ProbabilityBayes’ TheoremProbability Density & Cumulative Distribution FunctionsTransformed DensitiesExpectationsVariances and CovariancesThe Gaussian DistributionGaussian Mean and VarianceThe Multivariate GaussianGaussian Parameter EstimationMaximum (Log) LikelihoodProperties of and Curve Fitting Re-visitedMaximum LikelihoodPredictive DistributionMAP: A Step towards BayesBayesian Curve FittingBayesian Predictive DistributionModel Selection via Cross-ValidationCurse of DimensionalityCurse of DimensionalityCS 59000 Machine learningLecture 2Yuan (Alan) Qi ([email protected])Review: Polynomial Curve FittingSum‐of‐Squares Error Function1stOrder Polynomial3rdOrder Polynomial9thOrder PolynomialOver‐fittingRoot‐Mean‐Square (RMS) Error:Polynomial Coefficients RegularizationPenalize large coefficient valuesRegularization: Regularization: Regularization: vs. Polynomial Coefficients Data Set Size: N = 109thOrder PolynomialData Set Size: 9thOrder PolynomialTraining Data: the more, the better... 9thOrder PolynomialReview: Probability TheoryMarginal ProbabilityConditional ProbabilityJoint ProbabilityProbability TheorySum RuleProduct RuleThe Rules of ProbabilitySum RuleProduct RuleBayes’ Theoremposterior ∝ likelihood × priorProbability Density & Cumulative Distribution FunctionsTransformed DensitiesExpectationsConditional Expectation(discrete)Approximate Expectation(discrete and continuous)Variances and CovariancesThe Gaussian DistributionGaussian Mean and VarianceThe Multivariate GaussianGaussian Parameter EstimationLikelihood functionMaximum (Log) LikelihoodProperties of and UnbiasedBiasedCurve Fitting Re‐visitedMaximum LikelihoodDetermine by minimizing sum‐of‐squares error, .Predictive DistributionMAP: A Step towards BayesDetermine by minimizing regularized sum‐of‐squares error, .Bayesian Curve FittingBayesian Predictive DistributionModel Selection via Cross‐ValidationCurse of DimensionalityCurse of DimensionalityPolynomial curve fitting, M = 3Gaussian Densities in higher
View Full Document