Singular Value Decomposition (SVD)SVD of a Matrix= * *M = U S VTU and V are orthogonal matrices, and S is a diagonal matrix consistingof singular values.Singular Value Decomposition (SVD)SVD of a Matrix: observationsM = U S VTMultiply both sides by MTMTM = (U S VT)TU S VTMTM = (V S UT) U S VTUTU = IMTM = V S2VTMMT= U S VT(U S VT)TMMT= U S VT(V S UT)VTV = IMMT= U S2UTMultiplying on the left Multiplying on the rightSingular Value Decomposition (SVD)MTM = V S2VTMMT= U S2UTdiagonalizationsDiagonalization of a Matrix: (finding eigenvalues)A = W Λ WTwhere:• A is a square, symmetric matrix• Columns of W are eigenvectors of A• Λ is a diagonal matrix containing the eigenvaluesTherefore, if we know U (or V) and S, we basically have found out the eigenvectors and eigenvalues of MMT(or MTM) !SVD of a Matrix: observationsPrincipal Component Analysis (PCA)What is PCA?• Analysis of n-dimensional data• Observes correspondence between different dimensions• Determines principal dimensions along which the variance of the data is highWhy PCA?• Determines a (lower dimensional) basis to represent the data• Useful compression mechanism• Useful for decreasing dimensionality of high dimensional dataPrincipal Component Analysis (PCA)Steps in PCA: #1 Calculate Adjusted Data Set…ndimsdata samplesData Set: D Mean values: M-…Adjusted Data Set: A=Miis calculated by taking the mean of the values in dimension iPrincipal Component Analysis (PCA)Steps in PCA: #2 Calculate Co-variance matrix, C, from Adjusted Data Set, ACo-variance Matrix: CnnCij= cov(i,j)Note: Since the means of the dimensions in the adjusted data set, A, are 0, the covariance matrix can simply be written as:C = (A AT) / (n-1)Principal Component Analysis (PCA)Steps in PCA: #3 Calculate eigenvectors and eigenvalues of CEigenvectorsEigenvaluesIf some eigenvalues are 0 or very small, we can essentially discard those eigenvalues and the corresponding eigenvectors, hence reducing the dimensionality of the new basis.EigenvectorsEigenvaluesxxMatrix EMatrix EPrincipal Component Analysis (PCA)Steps in PCA: #4 Transforming data set to the new basisF = ETAwhere:• F is the transformed data set• ETis the transpose of the E matrix containing the eigenvectors• A is the adjusted data setNote that the dimensions of the new dataset, F, are less than the data set ATo recover A from F:(ET)-1F = (ET)-1ETA(ET)TF = AEF = A* E is orthogonal, therefore E-1= ETPCA using SVDRecall: In PCA we basically try to find eigenvalues and eigenvectors of the covariance matrix, C. We showed that C = (AAT) / (n-1), and thus finding the eigenvalues and eigenvectors of C is the same as finding the eigenvalues and eigenvectors of AATRecall: In SVD, we decomposed a matrix A as follows:A = U S VTand we showed that:AAT= U S2UTwhere the columns of U contain the eigenvectors of AATand the eigenvalues of AATare the squares of the singular values in SThus SVD gives us the eigenvectors and eigenvalues that we need for