Automatic recognition of discourse relationsCan RST analysis be done automatically?ME’02Cleverly labeling data through patterns with cue phrasesExtraction patternsMain ideaSlide 7DiscussionB-GMR’07: Offer several improvements over ME’02SL’07Two classifiersExplicit note here, not in the previous papersTesting on explicit relationsTraining on manually labeled, unmarked dataUsing the Penn discourse tree bankSlide 16Automatic recognition of discourse relationsLecture 3Can RST analysis be done automatically?In the papers we’ll read the question is really about local rhetorical relationsPart of the problem is the availability of training data for automatic labellingManual annotation is slow and expansiveLots of data can be cleverly collected, but is it appropriate (SL’07 paper)ME’02Discourse relations are often signaled by cue phrasesCONTRAST: butEXPLANATION-EVIDANCE: because But not always. In a manually annotated corpus25% of contrast and explanation-evidence relations marked explicitly by a cue phrase•Mary liked the play, John hated it•He wakes up early every morning. There is a construction site opposite his building.Cleverly labeling data through patterns with cue phrases CONTRAST[BOS…EOS][BOS But…EOS][BOS…][but…EOS][BOS…][although…EOS][BOS Although…,][…EOS]CAUSE-EXPLANATION[BOS…][because…EOS][BOS Because…,][…EOS][BOS…EOS][Thus,…EOS]Extraction patternsCONDITION[BOS If…,][…EOS][BOS If…][then…EOS][BOS…][if…EOS]ELABORATION[BOS…EOS][BOS…for example…EOS][BOS…][which…EOS]NO-RELATION-SAME-TEXTNO-RELATION-DIFF-TEXTMain ideaPairs of words can trigger a given relationJohn is good in math and sciences.Paul fails almost every class he takes.Embargo—legallyFeatures for classification the cartesian product of the words in the two text spans being annotatedProbability of word-pairs given a relationlog(W1,W2|RLk) + log(P(RLk)Classification results are well above the baselineUsing only content words did not seem to be very helpfulModel does not perform that well on manually annotated examplesDiscussionWould be interesting to see the list of the most informative word-pairs per relationIs there an intrinsic difference in clauses explicitly marked for a relation compared to those where the relation is implicit?B-GMR’07: Offer several improvements over ME’02Tokenizing and stemmingImproves accuracy Reduces model sizeVocabulary size limit/minimum frequencyUsing 6,400 most frequent words is bestUsing a stoplistPerformance deteriorates (as in the original ME’02 paper!)Topic segmentation for better example collectionSL’07Using automatically labeled examples to classify rhetorical relationsIs it a good idea?The answer is no, as already hinted by the other papersTwo classifiersWord-pair based Naïve BayesMulti-feature (41) BoosTexter modelPositional Length Lexical POSTemporalCohesion (pronouns and ellipsis)Explicit note here, not in the previous papersThe distribution of different relations in the automatically extracted corpus does not reflect the true distributionIn all studies data is downsampledTesting on explicit relationsResults deteriorate for both machine learning approachesStill better than randomNatural data does not seem suitable for trainingDo not generalize well to examples which occur naturally without unambiguous discourse markersTraining on manually labeled, unmarked data Less training data is availableWorse for the Naïve Bayes classiferGood for the Boostexter modelWhy?Semantic redundancy between discourse markers and the context they appear in?Using the Penn discourse tree bankImplicit relationsNot that good performanceExplicit relationsPerformance closer to that in automatically collected test setCheap data collection for this task probably not that good idea after
View Full Document