1Fitting a Function or DistributionCarolyn Penstein RoséSchool of Computer ScienceCarnegie Mellon UniversityWhat does it mean to fit a distribution to some data points?Slope = Intercept = 0-1 1-10Y = Slope*X + InterceptWhat does it mean to fit a distribution to some data points?0-1 1Here each dot is an instance and we are predicting the class value (assume it is a number) from the X value, which is our one feature.If our class value was nominal, we would be defining the boundary between instances of one class and instances of the other class.Another kind of function we try to fitis a histogram of Y values, which we call a Probability Distribution Function.Let’s say we want a distribution of a height variable of students in this classHeightNumber of students with heightWhat does it mean to fit a distribution to some data points?Arithmetic MeanStandardDeviation… And from this you can compute probabilities of specific x positions!Define a binary variable that asks whether a value is within a particular probability bandVarA is true if the value is within one standard deviationVabB is true if the value is within two standard deviationsDefine a binary variable that asks whether a value is within a particular probability bandVarA is true if the value is within one standard deviationVabB is true if the value is within two standard deviationsVarA =VarB=TrueTrueDefine a binary variable that asks whether a value is within a particular probability bandVarA is true if the value is within one standard deviationVabB is true if the value is within two standard deviationsVarA =VarB=FalseFalseDefine a binary variable that asks whether a value is within a particular probability bandVarA is true if the value is within one standard deviationVabB is true if the value is within two standard deviationsVarA =VarB=FalseTrueConfidence Intervalsn We know that the accuracy measured on each fold of a cross validation will be differentn The average across these is a better estimate of expected performance than any of the separate onesn Confidence intervals allow us to say that the probability of the real performance value being within a certain range from the observed average value is within a confidence level – like 90%0 10 20 30 40( )Confidence Intervalsn Confidence limits come from the normal distributionn Computed in terms of number of standard deviations from the meann If the data is normally distributed, there is a 15% chance of the real value being more than 1 standard deviation above the meanWhat is a significance test?n How likely is it that the difference in average performance you see when you compare from cross-validations of two algorithms occurred by chance?n How could the difference occur by chance?0 10 20 30 40( ( ) )If the mean of one distribution is within theconfidence interval of another, the difference you observe could be by chance.If you want p<.05, you need the 90% confidence intervals. Find the correspondingZ scores from a standard normal distribution table.Computing Confidence Intervalsn 90% confidence interval corresponds to z=1.65¨ 5% chance that a data point will occur to the right of the rightmost edge of the intervaln f = percentage of successesn N = number of trialsn f=75%, N=1000, c=90% ->