DOC PREVIEW
CMU CS 10701 - Recitation

This preview shows page 1-2-17-18-19-36-37 out of 37 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 37 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Dimension Reduction (PCA, ICA, CCA, FLD, To p i c M o d e l s )Yi Zhang10-701, Machine Learning, Spring 2011April 6th, 2011Parts of the PCA slides are from previous 10-701 lectures1Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation2Dimension reduction Feature selection – select a subset of features More generally, feature extraction◦ Not limited to the original features◦ “Dimension reduction” usually refers to this case3Dimension reduction Assumption: data (approximately) lies on a lower dimensional space Examples:4Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation5Principal components analysis6Principal components analysis7Principal components analysis8Principal components analysis9Principal components analysis Assume data is centered For a projection direction v◦ Variance of projected data10Principal components analysis Assume data is centered For a projection direction v◦ Variance of projected data◦ Maximize the variance of projected data 11Principal components analysis Assume data is centered For a projection direction v◦ Variance of projected data◦ Maximize the variance of projected data ◦ How to solve this ? 12Principal components analysis PCA formulation As a result …13Principal components analysis14Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation15Source separation The classical “cocktail party” problem◦ Separate the mixed signal into sources16Source separation The classical “cocktail party” problem◦ Separate the mixed signal into sources◦ Assumption: different sources are independent17Independent component analysis  Let v1, v2, v3, … vddenote the projection directions of independent components ICA: find these directions such that data projected onto these directions have maximum statistical independence18Independent component analysis  Let v1, v2, v3, … vddenote the projection directions of independent components ICA: find these directions such that data projected onto these directions have maximum statistical independence How to actually maximize independence?◦ Minimize the mutual information◦ Or maximize the non-Gaussianity◦ Actual formulation quite complicated !19Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation20Recall: PCA Principal component analysis◦ Note: ◦ Find the projection direction v such that the variance of projected data is maximized◦ Intuitively, find the intrinsic subspace of the original feature space (in terms of retaining the data variability)21Canonical correlation analysis Now consider two sets of variables x and y◦ x is a vector of p variables◦ y is a vector of q variables◦ Basically, two feature spaces How to find the connection between two set of variables (or two feature spaces)?22Canonical correlation analysis Now consider two sets of variables x and y◦ x is a vector of p variables◦ y is a vector of q variables◦ Basically, two feature spaces How to find the connection between two set of variables (or two feature spaces)?◦ CCA: find a projection direction u in the space of x, and a projection direction v in the space of y, so that projected data onto u and v has max correlation◦ Note: CCA simultaneously finds dimension reduction for two feature spaces23Canonical correlation analysis CCA formulation◦ X is n by p: n samples in p-dimensional space◦ Y is n by q: n samples in q-dimensional space◦ The n samples are paired in X and Y24Canonical correlation analysis CCA formulation◦ X is n by p: n samples in p-dimensional space◦ Y is n by q: n samples in q-dimensional space◦ The n samples are paired in X and Y How to solve? … kind of complicated …25Canonical correlation analysis CCA formulation◦ X is n by p: n samples in p-dimensional space◦ Y is n by q: n samples in q-dimensional space◦ The n samples are paired in X and Y How to solve? Generalized eigenproblems !26Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation27Fisher’s linear discriminant Now come back to one feature space In addition to features, we also have label◦ Find the dimension reduction that helps separate different classes of examples !◦ Let’s consider 2-class case28Fisher’s linear discriminant Idea: maximize the ratio of “between-class variance” over “within-class variance” for the projected data29Fisher’s linear discriminant30Fisher’s linear discriminant Generalize to multi-class cases Still, maximizing the ratio of “between-class variance” over “within-class variance” of the projected data:31Outline Dimension reduction Principal Components Analysis Independent Component Analysis Canonical Correlation Analysis Fisher’s Linear Discriminant Topic Models and Latent DirichletAllocation32Topic models Topic models: a class of dimension reduction models on text (from words to topics)33Topic models Topic models: a class of dimension reduction models on text (from words to topics) Bag-of-words representation of documents34Topic models Topic models: a class of dimension reduction models on text (from words to topics) Bag-of-words representation of documents Topic models for representing documents35Latent Dirichlet allocation A fully Bayesian specification of topic models36◦ Data: words on each documents◦ Estimation: maximizing the data likelihood – difficult!Latent Dirichlet


View Full Document

CMU CS 10701 - Recitation

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download Recitation
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Recitation and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Recitation 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?