Dimension Reduction PCA ICA CCA FLD Topic Models Yi Zhang 10 701 Machine Learning Spring 2011 April 6th 2011 Parts of the PCA slides are from previous 10 701 lectures 1 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 2 Dimension reduction Feature selection select a subset of features More generally feature extraction Not limited to the original features Dimension reduction usually refers to this case 3 Dimension reduction Assumption data approximately lies on a lower dimensional space Examples 4 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 5 Principal components analysis 6 Principal components analysis 7 Principal components analysis 8 Principal components analysis 9 Principal components analysis Assume data is centered For a projection direction v Variance of projected data 10 Principal components analysis Assume data is centered For a projection direction v Variance of projected data Maximize the variance of projected data 11 Principal components analysis Assume data is centered For a projection direction v Variance of projected data Maximize the variance of projected data How to solve this 12 Principal components analysis PCA formulation As a result 13 Principal components analysis 14 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 15 Source separation The classical cocktail party problem Separate the mixed signal into sources 16 Source separation The classical cocktail party problem Separate the mixed signal into sources Assumption different sources are independent 17 Independent component analysis Let v1 v2 v3 vd denote the projection directions of independent components ICA find these directions such that data projected onto these directions have maximum statistical independence 18 Independent component analysis Let v1 v2 v3 vd denote the projection directions of independent components ICA find these directions such that data projected onto these directions have maximum statistical independence How to actually maximize independence Minimize the mutual information Or maximize the non Gaussianity Actual formulation quite complicated 19 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 20 Recall PCA Principal component analysis Note Find the projection direction v such that the variance of projected data is maximized Intuitively find the intrinsic subspace of the original feature space in terms of retaining the data variability 21 Canonical correlation analysis Now consider two sets of variables x and y x is a vector of p variables y is a vector of q variables Basically two feature spaces How to find the connection between two set of variables or two feature spaces 22 Canonical correlation analysis Now consider two sets of variables x and y x is a vector of p variables y is a vector of q variables Basically two feature spaces How to find the connection between two set of variables or two feature spaces CCA find a projection direction u in the space of x and a projection direction v in the space of y so that projected data onto u and v has max correlation Note CCA simultaneously finds dimension reduction for two feature spaces 23 Canonical correlation analysis CCA formulation X is n by p n samples in p dimensional space Y is n by q n samples in q dimensional space The n samples are paired in X and Y 24 Canonical correlation analysis CCA formulation X is n by p n samples in p dimensional space Y is n by q n samples in q dimensional space The n samples are paired in X and Y How to solve kind of complicated 25 Canonical correlation analysis CCA formulation X is n by p n samples in p dimensional space Y is n by q n samples in q dimensional space The n samples are paired in X and Y How to solve Generalized eigenproblems 26 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 27 Fisher s linear discriminant Now come back to one feature space In addition to features we also have label Find the dimension reduction that helps separate different classes of examples Let s consider 2 class case 28 Fisher s linear discriminant Idea maximize the ratio of between class variance over within class variance for the projected data 29 Fisher s linear discriminant 30 Fisher s linear discriminant Generalize to multi class cases Still maximizing the ratio of between class variance over within class variance of the projected data 31 Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher s Linear Discriminant Topic Models and Latent Dirichlet Allocation 32 Topic models Topic models a class of dimension reduction models on text from words to topics 33 Topic models Topic models a class of dimension reduction models on text from words to topics Bag of words representation of documents 34 Topic models Topic models a class of dimension reduction models on text from words to topics Bag of words representation of documents Topic models for representing documents 35 Latent Dirichlet allocation A fully Bayesian specification of topic models 36 Latent Dirichlet allocation Data words on each documents Estimation maximizing the data likelihood difficult 37
View Full Document
Unlocking...