perimeter = 4perimeter = πperimeter = 4perimeter = πperimeter = 4perimeter = πperimeter = 4perimeter = πperimeter = 4perimeter = ππ = 4Happy Halloween!Armin [email protected]• xˆis a least squares solution of the syste m Ax = bxˆis such that Axˆ− b is as small as possibleGFTLAATAxˆ= ATb(the normal equations)Appl ication: least squares linesExample 1. Find β1, β2such that the line y = β1+ β2x best fits the da ta points (2, 1),(5, 2), (7, 3), (8, 3).0 2 4 6 8024Comment. As usual in practice, we are minimizing the (sum of squares of the) vertical offsets:http://mathworld.wolfram.com/LeastSquaresFitting.htmlSolution. The equations yi= β1+ β2xiin matrix form:Armin [email protected]1 x11 x21 x31 x4design m atrix Xβ1β2=y1y2y3y4observatio nvector yHere, we need to find a least squares solution to1 21 51 71 8β1β2=1233.XTX =1 1 1 12 5 7 81 21 51 71 8=4 2222 142XTy =1 1 1 12 5 7 81233=957Solving4 2222 142βˆ=957, we findβ1β2=2/75/14.Hence, the least squares line isy =27+514x.0 2 4 6 8024How well does the line fit the data (2, 1), ( 5, 2), (7, 3), (8, 3)?How small is the sum of squares of the vertical offsets?Armin [email protected]• residual sum of s quares: SSres=P(yi− (β1+ β2xi)error at (xi,yi))2The choice of β1, β2from l east s quares, makes SSre sas small as possible.• total sum of squares: SStot=P(yi− y¯)2,where y¯ =1nPyiis the mean of the observed data• coefficient of determination: R2= 1 −SSr e sSSt o tGeneral rule: the closer R2is to 1, the bet ter the regression l ine fits the data.Here,y¯ = 9/4 : (2, 1), (5, 2), (7, 3), (8, 3)R2=1 −1 −27+51422+2 −27+51452+3 −27+51 472+3 −27+51 4821 −942+2 −942+3 −942+3 −942= 1 −0.0752.75= 0.974very close to 1good fitOther curvesWe can also fit the experimenta l data(xi, yi) using other curves.Example 2. yi≈ β1+ β2xi+ β3xi2with param eters β1, β2, β3.The equations yi= β1+ β2xi+ β3xi2in matrix form:1 x1x121 x2x221 x3x32 desig n m atrix Xβ1β2β3=y1y2y3observationvector yGiven data (xi, yi), we then find the least square s solution to Xβ = y.Multiple linear regressionIn stat i stics, linear regression is an a pproach for modeling the relation-ship betwee n a scalar dep endent variable and one or more explanatoryvariables.The case of one explanatory variable is called simple linear regression.For more than one e xplanatory variable, the process is called multiplelinear regression.http://en.wikipedia.org/wiki/Linear_regressionThe experimen tal data might be of the form (vi, wi, yi), where now the dependentvariab le yidepends on two explanatory variables vi, wi(instead of just one xi).Armin [email protected] 3. Fitting a linear relationship yi≈ β1+ β2vi+ β3wi, we get:1 v1w11 v2w21 v3w3 design m atrixβ1β2β3=y1y2y3observationvectorAnd we again proceed by finding a least squares solution.Reviewv1v2xˆxx⊥• Suppose v1,, vmis an orthonormal basis of W .The orthogonal projection ofx onto W is:xˆ= hx, v1iv1proj. of x on to v1++ hx, vmivmproj. of x onto vm.(To stay agile, we are writing hx, v1i= x ·v1for the inner product.)Gram–SchmidtExample 4. Find an orthonormal basis for V = span1000,2100,1111.Recipe. (Gram– Sc hmidt orthonormalization)Given a basisa1,, an, produce an orthonormal basis q1,, qn.b1= a1, q1=b1kb1kb2= a2− ha2, q1iq1, q2=b2kb2kb3= a3− ha3, q1iq1− ha3, q2iq2, q3=b3kb3kArmin [email protected] 5. Find an orthonormal basis for V = span1000,2100,1111.Solution.b1=1000, q1=1000b2=2100−h2100, q1iq1=0100, q2=0100b3=1111−h1111, q1iq1−h1111, q2iq2=0011, q3=12√0011We hav e obtained an ort h o normal basis for V :1000,0100,12√0011.Armin
View Full Document
Unlocking...