Twitter Sentiment AnalysisDefining SentimentRelated WorkData CollectionClassifiersNaive BayesFrequency-based feature selectionMutual InformationΧ2 Feature selectionMaximum Entropy Feature Extractors1. UnigramNaive Bayes with Three ClassesError AnalysisMaxEnt Error AnalysisConclusion and Future ImprovementsReferencesAppendixA. Code DetailsB. Web ApplicationAlec Go ([email protected])Lei Huang ([email protected])Richa Bhayani ([email protected])CS224N - Final Project ReportJune 6, 2009, 5:00PM (3 Late Days)Twitter Sentiment AnalysisIntroductionTwitter is a popular microblogging service where users create status messages (called"tweets"). These tweets sometimes express opinions about different topics.The purpose of this project is to build an algorithm that can accurately classify Twittermessages as positive or negative, with respect to a query term. Our hypothesis is that wecan obtain high accuracy on classifying sentiment in Twitter messages using machinelearning techniques.Generally, this type of sentiment analysis is useful for consumers who are trying to researcha product or service, or marketers researching public opinion of their company.Defining SentimentFor the purposes of our research, we define sentiment to be "a personal positive or negativefeeling." Here are some examples:Sentiment Query TweetPositive jquerydcostalis: Jquery is my newbest friend.Neutral San Franciscoschuyler: just landed at SanFranciscoNegative examjvici0us: History examstudying ugh.For tweets that were not clearcut, we use the following litmus test: If the tweet could everappear as a newspaper headline or as a sentence in Wikipedia, then it belongs in the neutralclass. For example, the following tweet would be marked as neutral because it is fact froma newspaper headline, even though it projects an overall negative feeling about GM:ThomasQuinlin: RT @Finance_Info Bankruptcy filing could put GM on road to profits (AP)http://cli.gs/9ua6Sb #FinanceRelated WorkThere have been many papers written on sentiment analysis for the domain of blogs andproduct reviews. (Pang and Lee 2008) gives a survey of sentiment analysis. Researchershave also analyzed the brand impact of microblogging (Jansen). We could not find anypapers that analyzes machine learning techniques in the specific domain of microblogs,probably because the popularity of Twitter is very recent.Overall, text classification using machine learning is a well studied field (Manning andSchuetze 1999). (Pang and Lee 2002) researched the effects of various machine learningtechniques (Naive Bayes (NB), Maximum Entropy (ME), and Support Vector Machines (SVM)in the specific domain of movie reviews. They were able to achieve an accuracy of 82.9%using SVM and a unigram model.Researchers have also worked on detecting sentiment in text. (Turney 2002) presents asimple algorithm, called semantic orientation, for detecting sentiment. (Pang and Lee 2004)present a hierarchical scheme in which text is first classified as containing sentiment, andthen classified as positive or negative.Work (Read, 2005) has been done in using emoticons as labels for positive and sentiment.This is very relevant to Twitter because many users have emoticons in their tweets.Twitter messages have many unique attributes, which differentiates our research fromprevious research:1. Length. The maximum length of a Twitter message is 140 characters. From our trainingset, we calculated that the average length of a tweet is 14 words, and the average length ofa sentence is 78 characters. This is very different from the domains of previous research,which was mostly focused on reviews which consisted of multiple sentences.2. Available data. Another difference is the sheer magnitude of data. In (Pang and Lee2002), the corpus size 2053. With the Twitter API, it is much easier to collect millions oftweets for training.3. Language model. Twitter users post messages from many different mediums, includingtheir cell phones. The frequency of misspellings and slang in tweets is much higher thanother domains.ProcedureData CollectionThere are not any existing data sets of Twitter sentiment messages. We collected our ownset of data. For the training data, we collected messages that contained the emoticons :)and :( via the Twitter API.The test data was manually. A set of 75 negative tweets and 108 positive tweets weremanually marked. A web interface tool was built to aid in the manual classification task.See Appendix A for more details about the data.ClassifiersSeveral different classifiers were used. A Naive Bayes classifier was built from scratch.Third-party libraries were used for Maximum Entropy and Support Vector Machines. Thefollowing table summarizes the results.Table 1. Accuracy results from various classifiersTraining size also has an effect on performance. Figure 1 shows the effect of training sizeon accuracy.Figure 1. Effect of training size on different classifiers.Naive BayesNaive Bayes is a simple model for classification. It is simple and works well on textcategoration. We adopt multinomial Naive Bayes in our project. It assumes each feature isconditional independent to other features given the class. That is,where c is a specific class and t is text we want to classify. P(c) and P(t) is the priorprobabilities of this class and this text. And P(t | c) is the probability the text appears giventhis class. In our case, the value of class c might be POSITIVE or NEGATIVE, and t is just asentence.The goal is choosing value of c to maximize P(c | t):Where P(wi| c) is the probability of the ith feature in text t appears given class c. We needto train parameters of P(c) and P(wi| c). It is simple for getting these parameters in NaiveBayes model. They are just maximum likelihood estimation (MLE) of each one. Whenmaking prediction to a new sentence t, we calculate the log likelihood log P(c) +∑ilogP(wi|c) of different classes, and take the class with highest log likelihood as prediction.In practice, it needs smoothing to avoid zero probabilities. Otherwise, the likelihood will be0 if there is an unseen word when it making prediction. We simply use add-1 smoothing inour project and it works well.Feature selectionFor unigram feature, there are usually 260,000 different features. This is a very largenumber. It makes model higher variance. (Since more complicated model has highervariance). So it will need much more training data to avoid overfitting. Our training setcontains hundreds of thousands
View Full Document
Unlocking...