DOC PREVIEW
CMU CS 10701 - Hierarchical Bayesian Models for Text Classification

This preview shows page 1-2 out of 7 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

10-701 Project Final ReportHierarchical Bayesian Modelsfor Text ClassificationYangbo [email protected] IntroductionHierarchical structure is a natural and effective way of organizing information. Well knownexamples include the Dewey Decimal System, Yahoo Directory and computer file systems.We view such a hierarchy as a tree, which is consisted of a root node, certain levels ofintermediate nodes and leaf nodes. Suppose the documents to be classified could be fit intoa topic tree. Intuitively, if we know the parents of a leaf node, we can describe the leaf moreaccurately. Therefore, we can use hierarchical information to improve the performance oftext classification. In this project, we propose a new method called General HierarchicalShrinkage (GHS), and compare it with the original Hierarchical Shrinkage (HS) methodand Naive Bayes.The rest of this paper is organized as follows. Section 2 reviews related work. Section 3describes the GHS algorithm and some technical details. Section 4 presents experimen-tal results, and compares GHS with HS and Naive Bayes, along with some preliminarydiscussion. Section 5 concludes this paper.2 Related WorkThis project is inspired by the Hierarchical Shrinkage method in [McCallum 1998]. In HS,we first train a Naive Bayes model for each class of documents. Each class is representedwith a leaf node in the topic tree. Given a non-leaf node A, the model of A is the mean of allleaf nodes in the subtree with A as its root. Therefore, the model of root node is the meanof all classes. Furthermore, we add a “super root” on top of the original root node, witha uniform conditional distribution. After building the tree, we assume the model of eachclass is a linear combination of all the nodes along the path from leaf to the super root. Theweights for linear combination can be optimized using a simple Expectation Maximization(EM) method.The HS method arises from a general parameter estimator called Shrinkage estimator orJames-Stein estimator, which was discovered by [Stein 1956], and later extended by [James& Stein 1961]. The basic idea of Shrinkage Estimator is as follows: when estimating agroup of parameters (θ1, ..., θn), we can reduce the mean square error (MSE) by shrinking{θi} towards their mean¯θ =Piθi, even if {θi} are completely independent. Since thisstatement contradicts to people’s common sense, it’s called Stein’s Paradox. We will brieflyreview it in Section 3.2.[Koller & Sahami 1997] proposes another hierarchical method for text classification, whichis called Pachinko Machine (PM). PM also group classes into a topic tree, and computethe model of each node based on all documents belonging to it. However, PM does notcombine different nodes together to produce mixture models. Instead, it takes a greedytop-down search strategy to locate the “best” leaf-node for the document. The search startat root. At each node A, PM picks a sub-branch of A according to certain criteria. PMrepeats this action until it reaches a leaf. Therefore, the accuracy of the entire process isthe product of accuracy on all levels. For example, if the accuracy on each level is 0.9, andthere are three levels, then the final accuracy is 0.93= 0.73.3 Methods3.1 Naive BayesWe assume a document is generated by two steps: first choose a class cjwith probabilityP (cj), then generate its bag of words according to the conditional distribution P (w|cj).Based on this assumption, we use the algorithm in Table 6.2 of [Mitchell 1997] to trainNaive Bayes classifiers.Given a labeled document di, the probability that it belongs to cjis P (cj|di) ∈ {0, 1}. Weestimate the prior distribution of class cj:P (cj) =|D|Xi=1P (cj|di)|D|(1)where |D| is the number of documents.The conditional distribution is estimated by:P (wt|cj) =1 +P|D|i=1N(wt, di)P (cj|di)|V | +P|V |s=1P|D|i=1N(ws, di)P (cj|di)(2)where |V | is the vocabulary size, N(wt, di) is the term-frequency (TF) os wtin di.After the classifier is built, we classify future documents as:c(di) = arg maxcjP (cj|di) = arg maxcjYwt∈diP (wt|cj)P (cj)P (wt)(3)3.2 James-Stein EstimatorThe James-Stein estimator is simple to state, but hard to believe at first glance. Assumethere are a group of variables {xi}, i = 1, . . . , n, which follow Gaussian distributionN(µ, σ2I), where I is the identity matrix. We are interested in estimating the set of pa-rameters µ based on observation x = X. A natural and intuitive estimate is the maximumlikelihood estimation (MLE) ˆµ = X. [Stein 1956] demonstrated that, in terms of mini-mizing mean square error (MSE) E(kˆµ − µk2), the James-Stein estimator is better thanMLE.The original James-Stein estimator shrinks µ towards a prior µ = 0, when n > 2:ˆµ = (1 −(n − 2)σ2kXk2)X (4)Notice that when n ≤ 2, MLE is the best.A generalized James-Stein estimator can shrink µ towards non-zero prior, like the mean¯X =Pixi, when n > 3:ˆµ = X + (1 −(n − 3)σ2kX −¯Xk2)(X −¯X) (5)The reason why people are shocked by Stein’s claim is that each ˆµiis affected by allvariables in x, even if they are completely independent. For example, let µ1be the weightof cookies in a given box, µ2be the height of Mount Everest, and µ3be the speed of light,assume the results of our measurement x follow Gaussian distribution described above. TheJames-Stein estimator can get better MSE than maximum likelihood estimator. It meansthat the expectation of total MSE is reduced, while the MSE of each individual µicould bebetter or worse.3.3 General Hierarchical Shrinkage ModelRecall that in HS method, the final model θjof class cjis a linear combination of all thenodes on the path from leaf to root. The model of each intermediate node in the tree isagain a linear combination of its children. Therefore, θjis actually a linear combination ofall classes:θj=|C|Xk=1λkjθk,|C|Xk=1λkj= 1 (6)where |C| is the number of classes.The weights {λkj} is constrained by the hierarchical structure. For example, if classes ck1and ck2are siblings of cj(i.e. They share the same parent node with cj), then λk1j≡ λk2j.Based on above observation, a straightforward generalization of HS method is to give {λkj}more freedom. The maximum freedom for {λkj} is that they can take any non-negativevalue, as long asP|C|k=1λkj= 1. Like in HS method, we can still train the weights usingEM algorithm, although the number of weights increases from |C||L| in HS (|L| is thedepth of tree) to |C|2in GHS. The training algorithm is


View Full Document

CMU CS 10701 - Hierarchical Bayesian Models for Text Classification

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download Hierarchical Bayesian Models for Text Classification
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Hierarchical Bayesian Models for Text Classification and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Hierarchical Bayesian Models for Text Classification 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?