Oznur Tastan 10601 Machine Learning Recitation 3 Sep 16 2009OutlineSlide 3Text classificationText classification spam filteringSlide 6Slide 7Text Classification: ExamplesRepresenting text for classificationRepresenting text: a list of wordsSlide 11‘Bag of words’ representation of textBag of words representationBag of wordsSlide 15Slide 16Multinomial distributionMultinomial DistributionSlide 19Slide 20Slide 21Multinomial distribution and bag of wordsConjugate distributionDrichlet distributionDirichlet DistributionPseudo Count and priorGenerative modelSlide 28Polynomial Curve FittingSum-of-Squares Error Function0th Order Polynomial1st Order Polynomial3rd Order Polynomial9th Order PolynomialWhich of the predicted curve is better?What do we really want?Slide 37Slide 38ExampleGeneral strategyTest set methodSlide 42How good is the prediction?Train test set splitMore data is betterSlide 46Slide 47Slide 48Slide 49Train/test set splitCross ValidationLOOCV (Leave-one-out Cross Validation)Slide 53Slide 54Slide 55K-fold cross validationModel SelectionReferencesOznur Tastan 10601 Machine LearningRecitation 3Sep 16 2009Outline•A text classification example–Multinomial distribution–Drichlet distribution•Model selection–Miro will be continuing in that topicText classification exampleText classification•We are not into classification yet.•For the sake of example, I’ll briefly go over what it is. Classification Task:You have an input x, you classify which label it has y from some fixed set of labels y1,...,ykText classification spam filteringInput: document DOutput: the predicted class y from {y1,...,yk }Spam filtering:Classify email as ‘Spam’, ‘Other’.P (Y=spam | X)Text classificationInput: document DOutput: the predicted class y from {y1,...,yk }Text classification examples:Classify email as ‘Spam’, ‘Other’. What other text classification applications you can think of?Text classificationInput: document xOutput: the predicted class y y is from {y1,...,yk }Text classification examples:Classify email as ‘Spam’, ‘Other’.Classify web pages as ‘Student’, ‘Faculty’, ‘Other’Classify news stories into topics‘Sports’, ‘Politics’..Classify business names by industry.Classify movie reviews as ‘Favorable’, ‘Unfavorable’, ‘Neutral’ … and many more.Text Classification: Examples Classify shipment articles into one 93 categories. An example category ‘wheat’ARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONSBUENOS AIRES, Feb 26Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1,655.8, Feb 872.0, March 164.6, total 2,692.4 (4,161.0). Maize Mar 48.0, total 48.0 (nil). Sorghum nil (nil) Oilseed export registrations were: Sunflowerseed total 15.0 (7.9) Soybean May 20.0, total 20.0 (nil)The board also detailed export registrations for subproducts, as follows....Representing text for classificationARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONSBUENOS AIRES, Feb 26Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1,655.8, Feb 872.0, March 164.6, total 2,692.4 (4,161.0). Maize Mar 48.0, total 48.0 (nil). Sorghum nil (nil) Oilseed export registrations were: Sunflowerseed total 15.0 (7.9) Soybean May 20.0, total 20.0 (nil)The board also detailed export registrations for sub-products, as follows.... yHow would you represent the document?Representing text: a list of wordsargentine, 1986, 1987, grain, oilseed, registrations, buenos, aires, feb, 26, argentine, grain, board, figures, show, crop, registrations, of, grains, oilseeds, and, their, products, to, february, 11, in, …Common refinements: remove stopwords, stemming, collapsing multiple occurrences of words into one…. yRepresenting text for classificationARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONSBUENOS AIRES, Feb 26Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1,655.8, Feb 872.0, March 164.6, total 2,692.4 (4,161.0). Maize Mar 48.0, total 48.0 (nil). Sorghum nil (nil) Oilseed export registrations were: Sunflowerseed total 15.0 (7.9) Soybean May 20.0, total 20.0 (nil)The board also detailed export registrations for sub-products, as follows.... yHow would you represent the document?‘Bag of words’ representation of textARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONSBUENOS AIRES, Feb 26Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1,655.8, Feb 872.0, March 164.6, total 2,692.4 (4,161.0). Maize Mar 48.0, total 48.0 (nil). Sorghum nil (nil) Oilseed export registrations were: Sunflowerseed total 15.0 (7.9) Soybean May 20.0, total 20.0 (nil)The board also detailed export registrations for sub-products, as follows.... grain(s) 3oilseed(s) 2total 3wheat 1maize 1soybean 1tonnes 1... ...word frequencyBag of word representation:Represent text as a vector of word frequencies.Bag of words representationdocument iFrequency (i,j) = j in document iword jA collection of documentsBag of wordsWhat simplifying assumption are we taking?Bag of wordsWhat simplifying assumption are we taking?We assumed word order is not important.‘Bag of words’ representation of textARGENTINE 1986/87 GRAIN/OILSEED REGISTRATIONSBUENOS AIRES, Feb 26Argentine grain board figures show crop registrations of grains, oilseeds and their products to February 11, in thousands of tonnes, showing those for future shipments month, 1986/87 total and 1985/86 total to February 12, 1986, in brackets: Bread wheat prev 1,655.8, Feb 872.0, March 164.6, total 2,692.4 (4,161.0). Maize Mar 48.0, total 48.0 (nil). Sorghum nil (nil) Oilseed export registrations were: Sunflowerseed total 15.0 (7.9) Soybean May 20.0, total 20.0 (nil)The board also detailed export registrations for sub-products, as
View Full Document