CS224N Final ProjectSummarizing Movie ReviewsDan Fingal, Jeff Michels, Jamie NicolsonJune 3, 2004 (1 late day)1 AbstractText summarization is a classic problem in natural language processing. Given abody of text, is there an automated way to generate a few sentences that sum upits content? Using movie reviews downloaded from RottenTomatoes.com, alongwith summary sentences provided by the site, we attempt to find statisticalmachine learning methods to find acceptable summary sentences in previouslyunseen movie reviews. The task is inherently difficult due to the relativelyunstructured nature of online movie reviews, the large variability in writingstyles, and the presence of many possible “good” sentences among which onlyone will be tagged as the correct “RottenTomatoes” choice. Our best systemfirst classifies each review as either positive or negative in opinion and thenuses a unigram language model along with a ranking support vector machineto guess the correct sentence with 26% precision. Human precision on this taskwas tested to be 40%. In addition, many of the “incorrect” sentences that thesystem returns are subjectively plausible summaries.2 Research Questions2.1 Text SummarizationOne approach to the problem of text summarization is to search the documentfor sentences that use language that normally occurs in statements of broaddescription, judgement, or reflection. Our aim is to build a system that, givena list of the sentences in the document, can detect which of these qualify asproviding a summary of that document’s content. We attempt to do this withina machine learning context, in which the system distills the relevant featuresthrough training on a large corpus of tagged data, as opposed to a heuristicmethod, in which hand engineered detectors provide the discriminating factors.Kupiec et al. [3] extract summaries from technical documents using a verysmall feature space. Their features are sentence length; the presence of cer-tain fixed phrases, such as “In conclusion”; the position of the sentence within1a paragraph; and the presence of capitalized words. When their algorithm istold how many summary sentences there are, it achieves 42% accuracy on testdata. Technical documents, however, are far more structured than online moviereviews. Kupiec uses fixed phrases such as “Therefore,” “In conclusion,” etc.along with section titles to help determine whether or not a sentence wouldmake a good summary. Movie reviews typically do not have labeled subsectionsor catch phrases such as these. In addition, technical articles more often aredivided into paragraphs that have introductory and concluding sentences, so alot of mileage can be gotten from simply looking at the position of the sentencewithin the paragraph. Movie reviews, on the other hand, often contain mis-leading information. A review may begin with a description of the reviewer’sexpectations, only to later go on to describe how the movie did not live up tothem. These factors combine to make movie review summarization a difficulttask.2.2 ReviewsProduct reviews provide an excellent category on which to direct our system ofsummary sentence detection. Generally, reviews will describe the product underconsideration, its positive and negative qualities, and then provide the overallsentiment of the reviewer in one or two lines. It would be useful, therefore, fora seeker of product information to be able to cut right to the chase and get thisstatement of sentiment for a large number of reviews. Such sentences are moreuseful to the consumer than simple positive or negative categorization, as theynot only clearly provide a more nuanced measure of the reviewer’s rating of theproduct, but also distill down the essential reasons that the reviewer felt theway he or she did about the product.2.3 RottenTomatoes.comRottenTomatoes.com is a movie review megasite. For each movie that has everseen widespread release, RottenTomatoes maintains a page with a variety ofinformation and links related to that movie. Central to these pages are thevast quantities of links to reviews by RottenTomato approved critics. For eachof these links, RottenTomatoes displays a rating (fresh or rotten) and a quotedrawn from the review that characterizes its general sentiment. These quotesare of the same essential nature as described above; they convey to the readerthe intensity which which the reviewer classifies the movie, and give the readera justifying reason for the expressed opinion.This structure provides us with an exellent bank of tagged data with whichwe can test the systems that we implement. For every review linked to by thesite, there is a single target summary sentence hand labeled by RottenTomatoes.The central focus of our project was therefore to build a system which, afterbeing trained to recognize sentences tagged as summary or non-summary, wouldbe able to properly classify sentences in reviews it had not seen before. We madethe simplifying assumption that any given review contains exactly one “best”2Figure 1: Data Preparation Stagesummary sentence as labeled by RottenTomatoes.com, as this reduced the sub-jectivity inherent in the summarization task. This did, naturally, likely result inapparently lower performance as a “pretty good” sentence that happened notto be the one that RT.com chose was labeled as an incorrect response. It wasour hope that the classifier we built would fare decently well in the strict task offinding the tagged summary sentence, and in the process select sentences thatoverall were reasonable, even if those sentences were not tagged as the target inour corpus.3 Algorithms and Implementation3.1 Data CollectionTo set about acheiving these goals, we first had to acquire the data to trainand test on. We created a spider (getRottenTomatoes.py) to traverse theRottenTomatoes.com website, acquiring every review referenced by the site onits portal for each movie as well as the blurb RottenTomatoes.com selected tosummarize the review. These HTML pages were of little use in their raw form,however. We needed to scrub the data down to the core features we would beusing in our learning algorithm.The process of taking a webpage and turning into data was four steps long.In the first step (htmlstrip.cpp), we eliminated any html or scripting code,leaving only plaintext.In the second step (dodiffs.pl), we attempted to eliminate plaintext thatoccurs on all pages originating from the same site. To do
View Full Document
Unlocking...