The Information Retrieval Cycle Source Selection An Introduction to Information Retrieval and Question Answering Resource Query Formulation Query Search Jimmy Lin College of Information Studies University of Maryland Ranked List Selection query reformulation vocabulary learning relevance feedback Wednesday December 8 2004 Documents Examination source reselection Supporting the Search Process Source Selection Ad hoc retrieval find me documents like this Identify positive accomplishments of the Hubble telescope since it was launched in 1991 Query Formulation Query Compile a list of mammals that are considered to be endangered identify their habitat and if possible specify what threatens them Search Indexing Acquisition Index Ranked List Selection Collection Question answering Documents Examination Factoid Documents List Delivery IR is an Experimental Science Definition Formulate a research question the hypothesis Design an experiment to answer the question Perform the experiment Report the results Rinse repeat Who is Aaron Copland What is a quasar Does morphological analysis improve retrieval performance Does expanding the query with synonyms improve retrieval performance Corresponding experiments Are the results significant What countries export oil Name U S cities that have a Shubert theater Example questions Compare with a baseline control Does the experiment answer the question Who discovered Oxygen When did Hawaii become a state Where is Ayer s Rock located What team won the World Series in 1992 What experiments Delivery Types of Information Needs Resource Documents Build a stemmed index and compare against unstemmed baseline Expand queries with synonyms and compare against baseline unexpanded query What s missing here 1 IR Test Collections Three components of a test collection Where do they come from Collection of documents corpus Set of information needs topics Sets of documents that satisfy the information needs relevance judgments Series of annual evaluations started in 1992 Organized into tracks Test collections are formed by pooling Metrics for assessing performance TREC Text REtrieval Conferences Gather results from all participants Corpus topics judgments can be reused Precision Recall Other measures derived therefrom Roots of Question Answering Information Retrieval IR Information Retrieval IR Can substitute document for information Information Extraction IE IR systems Limitations Information Extraction IE IE systems Type Date Location Damage Deaths Templates pre defined questions Extracted information answers Limitations Templates are domain dependent and not easily portable One size does not fit all Answers questions indirectly Does not attempt to understand the meaning of user s query or documents in the collection Central Idea of Factoid QA Identify documents of a specific type Extract information according to pre defined templates Place the information into frame like database records Weather disaster Use statistical methods Rely on frequency of words in query document collection Retrieve complete documents Return ranked lists of hits based on relevance Determine the semantic type of the expected answer Who won the Nobel Peace Prize in 1991 is looking for a PERSON Retrieve documents that have keywords from the question Retrieve documents that have the keywords won Nobel Peace Prize and 1991 Look for named entities of the proper type near keywords Look for a PERSON near the keywords won Nobel Peace Prize and 1991 2 An Example Generic QA Architecture Who won the Nobel Peace Prize in 1991 But many foreign investors remain sceptical and western governments are withholding aid because of the Slorc s dismal human rights record and the continued detention of Ms Aung San Suu Kyi the opposition leader who won the Nobel Peace Prize in 1991 The military junta took power in 1988 as pro democracy demonstrations were sweeping the country It held elections in 1990 but has ignored their result It has kept the 1991 Nobel peace prize winner Aung San Suu Kyi leader of the opposition party which won a landslide victory in the poll under house arrest since July 1989 The regime which is also engaged in a battle with insurgents near its eastern border with Thailand ignored a 1990 election victory by an opposition party and is detaining its leader Ms Aung San Suu Kyi who was awarded the 1991 Nobel Peace Prize According to the British Red Cross 5 000 or more refugees mainly the elderly and women and children are crossing into Bangladesh each day Question analysis Question word cues Question Analyzer IR Query Document Retriever Documents Passage Retriever Passages Answer Extractor Answer Type Answers Using WordNet What is the service ceiling of an U 2 Who person organization location e g city When date Where location What Why How length NUMBER Head noun cues NL question What city which country what year Which astronaut what blues band wingspan diameter radius Scalar adjective cues altitude ceiling How long how fast how far how old Extracting Named Entities More Named Entities Person Mr Hubert J Smith Adm McInnes Grace Chan Currency 400 yen 100 DM 450 000 Title Chairman Vice President of Technology Secretary of State Linear 10 feet 100 miles 15 centimeters Country USSR France Haiti Haitian Republic Area a square foot 15 acres City New York Rome Paris Birmingham Seneca Falls Volume 6 cubic feet 100 gallons Province Kansas Yorkshire Uttar Pradesh Weight 10 pounds half a ton 100 kilos Business GTE Corporation FreeMarkets Inc Acme Duration 10 day five minutes 3 years a millennium University Bryn Mawr College University of Iowa Frequency daily biannually 5 times 3 times a day Organization Red Cross Boys and Girls Club Speed 6 miles per hour 15 feet per second 5 kph Age 3 weeks old 10 year old 50 years of age 3 How do we extract NEs Heuristics and patterns Fixed lists gazetteers Machine learning approaches Does it work Where do lobsters like to live Where do hyenas live Answer Type Hierarchy Limitations on a Canadian airline in Saudi Arabia in the back of pick up trucks Where are zebras most likely found near dumps in the dictionary Why can t ostriches fly What s the population of Maryland Because of American economic sanctions three Conclusion Question answering is an exciting research area Lies at the intersection of information retrieval and natural language processing A real world application of NLP technologies The dream a vast repository of knowledge we can talk to We re a long way from