Slide 1Principal Components AnalysisPCAPCA propertiesSlide 5Principal Components Analysis(PCA)273A Intro Machine LearningPrincipal Components Analysis• We search for those directions in space that have the highest variance.• We then project the data onto the subspace of highest variance.• This structure is encoded in the sample co-variance of the data:• Note that PCA is a unsupervised learning method (why?)1111( )( )NiiNTi iixNC x xNmm m==== - -��PCA• We want to find the eigenvectors and eigenvalues of this covariance:TC U U= L1l2ldl001ur2urdureigenvalue = variancein direction eigenvector( in matlab [U,L]=eig(C) )1ur2urOrthogonal, unit-length eigenvectors.PCA properties11 1( )( ) ( )dTi i iid dT Tj i i i j i i i j j ji iC uuCu uu u u u u ull l l== === = =�� �r rr r r r r r r r(U eigevectors)T TU U UU I= =(u orthonormal U rotation)1:Ti iky U x=1ur2ur3ur1: 1: 1:Tk k kC U U� L1l2l003l1:3U =1:3L =(rank-k approximation)(projection)1: 1: 1: 1: 1: 1: 1:1 11 1N NT T T T T Ty i i i ik k k k k k ki iC U x x U U x x U U U U UN N= =� �= = = L =L� �� �� �PCA properties1:kCis the optimal rank-k approximation of C in Frobenius norm. I.e. it minimizes the cost-function:1221 1 1( )d d kTijil lji j lC A A with A U= = =- = L�� �Note that there are infinite solutions that minimize this norm. If A is a solution, then is also a solution.The solution provided by PCA is unique because U is orthogonal and orderedby largest eigenvalue. Solution is also nested: if I solve for a rank-k+1 approximation, I will find that the first k eigenvectors are those found by an rank-k approximation (etc.)TAR with RR
View Full Document
Unlocking...