Calculus 220, section 7.5 Least SquaresCalculus 220, section 7.5 Least Squares notes prepared by Tim Pilachowski So far, we have dealt with equations, and found both derivatives and integrals of those functions. In the real world, the equations didn’t fall out of the sky but were developed from data—observations made about phenomenon. The curve is found that fits the data. Hopefully you remember how to find the equation of a line given two points: calculate slope (m), substitute to find the y-coordinate of the y-intercept (b), and write the equation (y = mx + b). When there are more than two points, and they don’t conveniently line up for us, we use partial derivatives to minimize the difference (error) between the observed data and the line of best fit. The process is called regression analysis and the method is called least squares. We’ll adopt the statistics convention and use the formula for the line of regression. BAxy +=(9, 8)(6, 7)••(5, 6)••(2, 4) Example A: A manufacturer has collected preliminary data relating number of units produced (x, measured in hundreds) and cost (y, measured in $1000’s). Find the equation that best represents cost as a function of number of units produced. Answer: ŷ = 0.58x + 3.06 The data are pictured in the scatterplot to the right. The line of regression will go “through the middle”, but the question becomes, “Of all the lines that we might draw that seem to fit the data, which is the one that has the least error between the observed (actual) y-value and the regression’s (predicted) ŷ-value = Ax + B?” xiobserved yiregression ŷ error Ei(Ei)22 4 5 6 6 7 9 8 Since some of the points will lie above the regression line, and some will lie below it, some of the errors will be positive and some will be negative. Finding the sum of the errors would result in a “canceling” effect. Instead, to keep the value of each error and retain its effect we’ll find the sum of the squared errors. This is the function for which we want a minimum. We’ll use the techniques of the previous sections: partial derivatives. We’ll need the chain rule.The process above involved only four data points. If we had 400, or 4000, the same process would become very unwieldy very quickly. We can develop a general formula which can be applied to N = any number of points. xiobserved yiregression ŷ error Ei(Ei)21x 1y BAx+1 BAxy−−11 ()211BAxy −− 2x 2y BAx+2 BAxy−−22 ()222BAxy −− M M M M M Nx Ny BAxN+ BAxyNN−− ()2BAxyNN−− Using the symbol Σ to mean “the sum of”, the least-squares error function we want to minimize is sum of squared errors =()()∑−−=2, BAxyBAf Note: I really should write , but chose the version above for simplicity’s sake. () ( )∑=−−=NiBAxyBAfii12,Using the sum rule and chain rule, we get ()()()()()( )02120222=++−=−−−=∂∂=++−=−−−=∂∂∑∑∑∑BAxyBAxyBfBxAxxyxBAxyAf Note that since we have N points and thus have N terms in our sum, we can replace ∑B with NB. Dividing by two to get easier numbers and rearranging gives us ⎪⎩⎪⎨⎧=+=+→⎪⎩⎪⎨⎧=++−=++−∑∑∑∑∑∑∑∑∑∑yNBxAxyxBxANBxAyxBxAxy2200 Solving the second equations for B we get NxAyB∑∑∗−=. Substituting into the first equation and solving for A we get ()22∑∑∑∑∑−∗∗−∗=xxNyxxyNAExample A revisited: Use the formulas above to find the least-squares regression equation. x y xy 2x 2 4 5 6 6 7 9 8 Σ