CSCI 5417 Information Retrieval Systems Jim MartinToday 11/15Sentiment, Style, Identity, Opinion ClassificationWhy is it hardExampleDiscussionClassificationMoviesPolitical SentimentSlide 10Slide 11Naïve Bayes: Top 20 wordsSlide 13ResultsChoosing a VocabularyWordsSlide 17Other parts of speechIdentifying Polarity WordsFinding Sentiment-bearing Adjectives Hatzivassiloglou & McKeown 1997Slide 21Slide 22Hatzivassiloglou & McKeown 1997SummaryFine-Grained/Structural SentimentChallengesQuiz Questions?Quiz hintsHomeworkCSCI 5417Information Retrieval SystemsJim MartinLecture 2311/15/201101/14/19 CSCI 5417 - IR 2Today 11/15Sentiment analysisQuiz questions?Extra HW01/14/19 CSCI 5417 - IR 3Sentiment, Style, Identity, Opinion ClassificationSentiment AnalysisMovie: is a review positive or negativeProducts (new iPhone)Sentiment over time (is voter anger increasing or decreasing?)Politics (is this editorial left or right?)Prediction (election outcomes, market trends). Will stock go up after this news report?Style/EmotionIs this conversation (or blog) friendly, aggressive, polite, flirtatious, deceitful, threatening01/14/19 CSCI 5417 - IR 4Why is it hardThe insurance folks were nice enough to set me up with a rental car until I get my settlement offer. Perfect, since I was planning to rent one to go to Vancouver this weekend anyway, and now its free. They paid for a "standard" size car, which means huge. I asked for something smaller with better fuel economy and ended up with a Kia Rondo in "velvet blue." It is indeed the color of Isabella Rossellini's bathrobe in Blue Velvet.Every time I drive a rental car I'm a bit appalled. My antique vehicle not only got better gas mileage than most new cars, but it had leg room and head room and ample windows for seeing out. New cars have tiny, low windows with blind spots all over the place. This Kia is ridiculous. It seems to be made for very tall people with very short legs. High ceilings, but the back seat is practically up against the front seat, and the hauling capacity is not better than, say, a Prius.01/14/19 CSCI 5417 - IR 5ExampleSo what exactly is the point of this compact, yet tall, mid-size SUV? Is it stylish? I can't see any practical reason it is designed this way. It is certainly not an off-road vehicle. I imagine it's front-wheel drive and a bitch to drive in snow. Does simply taking up a lot of space appeal to people? I'm sure it's a fine car, in a general sense, but whatever happened to "smart" design?01/14/19 CSCI 5417 - IR 6DiscussionWhat makes classification hard?Sentiment can be subtle:Perfume review in “Perfumes: the Guide”:“If you are reading this because it is your darling fragrance, please wear it at home exclusively, and tape the windows shut.”“She runs the gamut of emotions from A to B” (Dorothy Parker on Katherine Hepburn)Order effectsThis film should be brilliant. It sounds like a great plot, the actors are first grade, and the supporting cast is good as well, and Stallone is attempting to deliver a good performance. However, it can’t hold up.01/14/19 CSCI 5417 - IR 7ClassificationCoarse-grained classification of sentimentDocument-level classification according to some simple (usually binary) schemePolitical biasLikes/hatesFine-grained classification of sentiment-bearing mentions in a textPositive/negative classification of opinions about entities mentioned in a textPerhaps with intensity01/14/19 CSCI 5417 - IR 8MoviesA well-studied problem is the problem of classifying movie reviewsAs either +/-Or on a scale01/14/19 CSCI 5417 - IR 9Political SentimentTwo examples of classifiersUsing words as featuresWith a Naïve Bayes or SVM classifierTo make a binary decisionAbout the political stance of a text01/14/19 CSCI 5417 - IR 10Political SentimentBitterlemon.comA website designed to “contribute to mutual understanding [between Palestinians and Israelis] through the open exchange of ideas”Can we label a Pro-Israeli or Pro-Palestinian perspective from editorial texts1. “The inadvertent killing by Israeli forces of Palestinian civilians – usually in the course of shooting at Palestinian terrorists – is considered no different at the moral and ethical level than the deliberate targeting of Israeli civilians by Palestinian suicide bombers.”2. “In the first weeks of the Intifada, for example, Palestinian public protests and civilian demonstrations were answered brutally by Israel, which killed tens of unarmed protesters.”01/14/19 CSCI 5417 - IR 11Political SentimentA naïve Bayes classifier applied to this domain achieved accuracy around 90%Recall with naïve Bayes we have a table with priors on the class P(C)and tables of conditionals of words given the class P(word|C) for all the words in the vocabulary for the entire corpus01/14/19 CSCI 5417 - IR 12Naïve Bayes: Top 20 wordsPalestinian palestinian, israel, state, politics, peace, international, people, settle, occupation, sharon, right, govern, two, secure, end, conflict, process, side, negotiateIsraeli israel, palestinian, state, settle, sharon, peace, arafat, arab, politics, two, process, secure, conflict, lead, america, agree, right, gaza, govern01/14/19 CSCI 5417 - IR 13Political SentimentGoal: label a speech as a bill as for our against a bill in congressData: transcripts of all debates in US House of Representatives in 2005From GovTrack (http://govtrack.us) websiteEach speech segment (sequence of uninterrupted utterances by speaker)Labeled by the vote (“yea” or “nay”) castLabeled by SVM classifier, using all word unigrams as features01/14/19 CSCI 5417 - IR 14ResultsMajority baseline 58.37#(“support”) − #(“oppos”) 62.67SVM classifier 66.0501/14/19 CSCI 5417 - IR 15Choosing a VocabularyKey task: VocabularyEssentially feature selectionThe previous examples used all the words in the vocabularyCan we do better by focusing on subset of words?How to find words, phrases, patterns that express sentiment or polarity?01/14/19 CSCI 5417 - IR 16WordsAdjectivespositive: honest important mature large patientRon Paul is the only honest man in Washington. Kitchell’s writing is unbelievably mature and is only likely to get better. To humour me my patient father agrees yet again to my choice of film 1601/14/19 CSCI 5417 - IR