Assignment ILinear RegressionGenerating Synthetic DataThis assignment shows how we can extend ordinary least squares regression, which uses the hypothesis classof linear regression functions, to non-linear regression functions modeled using polynomial basis functions andradial basis functions. The function we want to fit is . Thisis a univariate function as it has only one input variable. First, we generate synthetic input (data) bysampling points from a uniform distribution on the interval .= (x) = 6(sin(x+ 2) + sin(2x+ 4))ytrueftruexin= 750[−7.5, 7.5]In [ ]:# The true functiondef f_true(x): y = 6.0 * (np.sin(x + 2) + np.sin(2*x + 4)) return y We can generate a synthetic data set, with Gaussian noise.In [ ]:import numpy as np # For all our math needsn = 750 # Number of data pointsX = np.random.uniform(-7.5, 7.5, n) # Training examples, in one dimensione = np.random.normal(0.0, 5.0, n) # Random Gaussian noisey = f_true(X) + e # True labels with noiseNow, we plot the raw data as well as the true function (without noise).In [ ]:import matplotlib.pyplot as plt # For all our plotting needsplt.figure() # Plot the dataplt.scatter(X, y, 12, marker='o') # Plot the true function, which is really "unknown"x_true = np.arange(-7.5, 7.5, 0.05)y_true = f_true(x_true)plt.plot(x_true, y_true, marker='None', color='r')Recall that we want to build a model to generalize well on future data, and in order to generalize well on futuredata, we need to pick a model that trade-off well between fit and complexity (that is, bias and variance). Werandomly split the overall data set ( ) into three subsets:Training set: consists of the actual training examples that will be used to train the model;Validation set: consists of validation examples that will be used to tune model hyperparameters(such as in ridge regression) in order to find the best trade-off between fit and complexity (that is, thevalue of that produces the best model);Test set: consists of test examples to estimate how the model will perform on future data.For this example, let us randomly partition the data into three non-intersecting sets: of , of and of .DDtrnDvalλ> 0λDtst= 60%DtrnD= 10%DvalD= 30%DtstDIn [ ]:# scikit-learn has many tools and utilities for model selectionfrom sklearn.model_selection import train_test_splittst_frac = 0.3 # Fraction of examples to sample for the test setval_frac = 0.1 # Fraction of examples to sample for the validation set # First, we use train_test_split to partition (X, y) into training and test setsX_trn, X_tst, y_trn, y_tst = train_test_split(X, y, test_size=tst_frac, random_state=42) # Next, we use train_test_split to further partition (X_trn, y_trn) into training and validation setsX_trn, X_val, y_trn, y_val = train_test_split(X_trn, y_trn, test_size=val_frac, random_state=42) # Plot the three subsetsplt.figure()plt.scatter(X_trn, y_trn, 12, marker='o', color='orange')plt.scatter(X_val, y_val, 12, marker='o', color='green')plt.scatter(X_tst, y_tst, 12, marker='o', color='blue')1. **Regression with Polynomial Basis Functions**, 30points.This problem extends ordinary least squares regression, which uses the hypothesis class of linear regressionfunctions, to non-linear regression functions modeled using polynomial basis functions. In order to learnnonlinear models using linear regression, we have to explicitly transform the data into a higher-dimensionalspace. The nonlinear hypothesis class we will consider is the set of -degree polynomials of the form or a linear combination of polynomial basis function:.The monomials are called basis functions, and each basis function has acorresponding weight associated with it, for all . We transform each univariate data point into into a multivariate ( -dimensional) data point via . When this transformationis applied to every data point, it produces the Vandermonde matrix:.df(x) = +x+ +. . . +w0w1w2x2wdxdf(x) = [ , , . . . ,w0w1w2wd]T⎡⎣⎢⎢⎢⎢⎢⎢⎢1xx2⋮xd⎤⎦⎥⎥⎥⎥⎥⎥⎥{1,x, , . . . , }x2xdxkwkk= 1, . . . ,dxidϕ( ) → [1, , , . . . , ]xixix2ixdiΦ =⎡⎣⎢⎢⎢⎢⎢11⋮1x1x2⋮xnx21x22⋮x2n. . .. . .⋱⋯xd1xd2⋮xdn⎤⎦⎥⎥⎥⎥⎥a. (10 points)Complete the Python function below that takes univariate data as input and computes a Vandermonde matrix ofdimension . This transforms one-dimensional data into -dimensional data in terms of the polynomial basis andallows us to model regression using a -degree polynomial.d ddIn [ ]:# X float(n, ): univariate data# d int: degree of polynomial def polynomial_transform(X, d): # # # *** Insert your code here *** # #b. (10 points)Complete the Python function below that takes a Vandermonde matrix and the labels as input and learnsweights via ordinary least squares regression. Specifically, given a Vandermonde matrix , implement thecomputation of . Remember that in Python, @ performs matrix multiplication, while *performs element-wise multiplication. Alternately, numpy.dot (https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.dot.html) also performs matrix multiplication.ΦyΦw= ( ΦyΦT)−1ΦTIn [ ]:# Phi float(n, d): transformed data# y float(n, ): labelsdef train_model(Phi, y): # # # *** Insert your code here *** # #c. (5 points)Complete the Python function below that takes a Vandermonde matrix , corresponding labels , and a linearregression model as input and evaluates the model using mean squared error. That is, .Φyw= ( −ϵMSE1n∑ni=1yiwTΦi)2In [ ]:# Phi float(n, d): transformed data# y float(n, ): labels# w float(d, ): linear regression modeldef evaluate_model(Phi, y, w): # # # *** Insert your code here *** # #d. (5 points, Discussion)We can explore the effect of complexity by varying to steadily increase the non-linearity ofthe models. For each model, we train using the transformed training data ( , whose dimension increases) andevaluate its performance on the transformed validation data and estimate what our future accuracy will be usingthe test data.From plot of vs. validation error below, which choice of do you expect will generalize best?d= 3, 6, 9, ⋯ , 24Φd dIn [ ]:w = {} # Dictionary to store all the trained modelsvalidationErr = {} # Validation error of the modelstestErr = {} # Test error of all the models for d in range(3, 25, 3): # Iterate over polynomial degree Phi_trn = polynomial_transform(X_trn,