Some interesting directions in Automatic SummarizationToday’s lectureRelevant Papers :More about DUC 2002 data…DUC 2002 input categoriesTf-idf summary - d061Tf-idf summary - d064Tf-idf summary - d065Multi-strategy summarizationProduce multiple summaries and choose – LCC GISTexterRoute to a specific module – Columbia’s multi-document summarizerFeatures - Single EventFeatures - BiographiesFeatures – Weakly related documentsCharacterizing/ Classifying inputsInput difficulty and Performance Confidence EstimationInput to summarizerInput difficulty & Content coverage scoresDid system performance vary with DUC 2001 input categories?Input type influenced scores obtainedCohesive inputs are easier to summarizeInputs can be easy or difficult =>First step..Features – Simple length-based‏Features – Word distributions in inputFeatures – Document similarity and relatednessFeatures – Document similarity and relatednessFeatures – Log likelihood ratio based‏What makes some inputs easy?Input difficulty hypothesis for systemsClassification ResultsSummary Evaluation without Human ModelsNeed for cheap, fast measuresCompare with input – No human modelsWhat features should we use?Features - Divergence between input and summaryFeatures – Use of topic words from the inputFeatures – Similarity between input and summaryFeatures - Summary ProbabilityAnalysis of featuresResultsEvaluation without human modelsOther summarization tasks of interestSlide 44What you have learnt today..1Some interesting directions in Automatic SummarizationAnnie LouisCIS 43012/02/082Today’s lectureMulti-strategy summarizationIs one method enough?Performance Confidence EstimationWould be nice to have an indication of expected system performance on an inputEvaluation without human modelsCan we come up with cheap and fast evaluation measures?Beyond generic summarizationQuery focused, updates, blogs, meeting, speech..3Relevant Papers :Lacatusu et al. LCC’s GISTexter at DUC 2006: Multi-Strategy Multi-Document Summarization. In Proceedings of the Document Understanding Workshop (DUC-2006) McKeown et al. Columbia multi-document. summarization: Approach and evaluation. In Proceedings of the Document Understanding Conference (DUC01), 2001. Nenkova et al. Can You Summarize This? Identifying Correlates of Input Difficulty for Multi-Document Summarization. In Proceedings of ACL-08: HLT4More about DUC 2002 data…/project/cis/nlp/tools/Summarization_Data/Inputs2002 Newswire texts Has 3 categories of inputs ! ?5DUC 2002 input categoriesSingle event - 30 inputsEg: d061- Hurricane GilbertSame place, roughly same time, same actionsMultiple distinct events – 15 inputsEg: d064 - Opening of Mac Donald’s at Russia, Canada, South Korea..Different places, different times, different agentsBiographies – 15 inputsEg: d065 – Dan Quayle, Bush’s nominee for vice presidentOne person – one event, background info – events from the pastDo you think a single method will do well for all ?6Tf-idf summary - d061Hurricane Gilbert Heads Toward Dominican Coast . Tropical Storm Gilbert formed in the eastern Caribbean and strengthened into a hurricane Saturday night. Gilbert Reaches Jamaican Capital With 110 Mph Winds . Hurricane warnings were posted for the Cayman Islands, Cuba and Haiti. Hurricane Hits Jamaica With 115 mph Winds; Communications. Gilbert reached Jamaica after skirting southern Puerto Rico, Haiti and the Dominican Republic. Gilbert was moving west-northwest at 15 mph and winds had decreased to 125 mph. What Makes Gilbert So Strong? With PM-Hurricane Gilbert, Bjt . Hurricane Gilbert Heading for Jamaica With 100 MPH Winds . Tropical Storm Gilbert7Tf-idf summary - d064First McDonald's to Open in Communist Country . Police Keep Crowds From Crashing First McDonald's . McDonald's and Genex contribute $1 million each for the flagship restaurant. A Bolshoi Mac Attack in Moscow as First McDonald's Opens . McDonald's Opens First Restaurant in China . McDonald's hopes to open a restaurant in Beijing later. The 500-seat McDonald's restaurant in a three-story building is operated by McDonald's Restaurant Shenzhen Ltd., a wholly owned subsidiary of McDonald's Hong Kong. McDonald's Hong Kong is a 50-50 joint venture with McDonald's in the United States. McDonald's officials say it is not a question that8Tf-idf summary - d065Tucker was fascinated by the idea, Quayle said. But Dan Quayle's got experience, too. Quayle's Triumph Quickly Tarnished . Quayle's Biography Inflates State Job; Quayle Concedes Error . Her statement was released by the Quayle campaign. But he would go no further in describing what assignments he would give Quayle. ``I will be a very close adviser to the president,'' Quayle said. ``You're never going to see Dan Quayle telling tales out of It was everything Quayle had hoped for. Quayle had said very little and he had said it very well. There are windows into the workings of the9Multi-strategy summarizationMultiple summarization modules within a single systemBetter than a single method How to employ a multi-strategy system?Use all methods, produce multiple summaries, choose bestUse a router and summarize by only one specific method10Produce multiple summaries and choose – LCC GISTexterTask - Query focused summarizationQuery is decomposed by 3 methodsSent to a QA system and a multi-document summarizer6 different summariesSelect the best summaryTextual entailment + pyramid scoring11Route to a specific module – Columbia’s multi-document summarizerFeatures to classify an input asSingle eventBiographyLoosely connected documentsThe result of classification is used to route the input to one of 3 different summarizers12Features - Single EventTo identifyTime span between publication dates < 80 daysMore than 50% documents published on same dayTo summarizeExploit redundancy, cluster similar sentences into themesRank themes based on size, similarity, ranking of contained sentences by lexical chainsSelect phrases from each themeGenerate sentences13Features - BiographiesTo identifyFrequency of most frequent capitalized letter > X (compensate for NE)Frequency of personal pronouns > YTo summarizeTarget individual mentioned in sentence ?Another individual found in the sentence ?Position of most prominent capitalized word in the sentence14Features – Weakly