Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 19: Data fittingLast time we have derived the important formulaP = A(ATA)−1AT.which gives the projection matrix when projecting onto the image of a matrix A.Given a system of linear equations Ax = b, the point x = (ATA)−1ATb is called theleast square solution o f the system.If A has no kernel, then the least square solution exists.Pro of. We know that if A has no kernel then the square matrix ATA has no kernel and is t hereforeinvertible.In applications we do not have to worry about this. In general, A is a n × m matrix where n ismuch larger than m meaning that we have lots of equations but few variables. Such matrices ingeneral have a trivial kernel. For linear regression for example, it only appears if all data pointsare on a vertical axes like (0, 2), (0, 6), (0, 0), (0, 4) and where any line y = mx + 3 is a least squaresolution. If in real life you should get into a situation where A has a kernel, you use the wrongmodel or have not enough few data.If x is the least square solution of Ax = b then Ax is the closest point on the image of A to b. Theleast square solution is the best solution of Ax = b we can find. Since P x = Ax, it is the closestpoint to b on V . Our knowledge about kernel and the image of linear transformations helped usto derive this.1 Finding the best polynomial which passes through a set of points is a data fitting problem.If we wanted to accommodate all data, the degree of the polynomial would become toolarge. The fit would look too wiggly. Taking a smaller degree polynomial will not only bemore convenient but also give a better picture. Especially important is regression, thefitting of da ta with linear polynomials.The above pictures show 30 data points which are fitted best with polynomials of degree 1,6, 11 and 16. The first linear fit maybe tells most about the tr end of the data.2 The simplest fitting pr oblem is fitting by lines. This is called linear regression. Find thebest line y = ax + b which fits the datax y-1 11 22 -1Solution. We will do this in class. The best solution is y = −x/2 + 1.3 Find the best parabola y = ax2+ bx + c which fits the pointsx y-1 80 81 42 16We do this in class. The best solution is f (x) = 3x2− x + 5.4 Find the f unction y = f(x) = a cos(πx) + b sin(πx), which best fits the datax y0 11/2 31 7Solution: We have to find the least square solution to the system of equations1a + 0b = 10a + 1b = 3−1a + 0b = 7which is in matrix for m written as A~x =~b withA =1 00 1−1 0,~b =137.Now AT~b ="−63#and ATA ="2 00 1#and (ATA)−1="1/2 00 1#and (ATA)−1AT~b is"−33#. The best fit is the function f(x) = −3 cos(πx) + 3 sin(πx).5 Find the circle a(x2+ y2) + b(x + y) = 1 which best fits the datax y0 1-1 01 -11 1In other words, find the least square solution for the system of equations for the unknownsa, b which aims to have all 4 data points (xi, yi) on the circle. To get system of linearequations Ax = b, plug in the datalla + b = 1a − b = 12a = 12a + 2b = 1 .This can be written as Ax = b, whereA =1 11 −12 02 2, b =1111.We get the least square solution with the usual formula. First compute(ATA)−1="3 −2−2 5#/22and thenATb ="62#,-2-1012-2-1012Homework due March 23, 20111 Find the f unction y = f(x) = ax2+ bx3, which best fits the datax y-1 11 30 102 A curve of the formy2= x3+ ax + bis called an elliptic curve in Weierstrass form. Elliptic curves are import ant in cryptog-raphy. Use data fitting to find the best parameters (a, b) for an elliptic curve given thefollowing points:(x1, y1) = (1, 2)(x2, y2) = (−1, 0)(x3, y3) = (2, 1)(x4, y4) = (0, 1)3 Find the f unction of the formf(t) = a sin(t) + b cos(t) + cwhich best fits the data points (0, 0), (π, 1), (π/2, 2), (−π,