Page 1 of 16 LBSC 796/INFM 718R: Information Retrieval Systems Spring, 2005 Midterm Exam Name: ________________________________ • Please show sufficient work to demonstrate your understanding of the material. This is mostly for your benefit, because it will allow partial credit to be awarded. • This exam has seven questions, six of which are divided into multiple parts. • You will have until 8:45pm to complete the exam. • Good luck! Question Points Total 1 16 2 10 3 48 4 20 5 17 6 12 7 2 Total 125 Page 2 of 16 Question 1. Evaluation (16 points) An information retrieval system returns the following ranked list for a particular query: 1234567891011 12 13 14 15 16 17 18 19 201234567891011 12 13 14 15 16 17 18 19 20 Colored blocks represent relevant documents; white blocks represent irrelevant documents. From the known relevance judgments, you know that there are eight relevant documents in total. A. (4 points) What is the Mean Average Precision (MAP)? B. (2 points) What is the R-precision? C. (2 points) What is Precision at 10?Page 3 of 16 D. (8 points) Plot the ROC curve (precision-recall curve), both uninterpolated and interpolated versions: RecallPrecision0.5 1.01.00.500RecallPrecision0.5 1.01.00.500 Page 4 of 16 Question 2. More Evaluation (10 points) Assume a document retrieval system produced the following interpolated ROC curve (precision-recall curve) on a particular query (based on 20 hits): RecallPrecision0.5 1.01.00.500RecallPrecision0.5 1.01.00.500 You know that there are ten relevant documents. A. (2 points) What is the precision after the system has retrieved three relevant documents? B. (2 points) Going down the hit list, I discover that I’ve retrieved n documents, and all of them are relevant. What’s the maximum possible value of n?Page 5 of 16 C. (6 points) Where are the relevant documents in the hit list? Mark a relevant document with an R in the corresponding box. Leave irrelevant documents unmarked. 1234567891011 12 13 14 15 16 17 18 19 201234567891011 12 13 14 15 16 17 18 19 20 Page 6 of 16 Question 3. Boolean and Vector Space Retrieval (48 points) Assume the following fragments comprise your document collection: Doc 1: Interest in real estate speculation Doc 2: Interest rates and rising home costs Doc 3: Kids do not have an interest in banking Doc 4: Lower interest rates, hotter real estate market Doc 5: Feds’ interest in raising interest rates rising Assume the following are stopwords: an, and, do, in, not A. (10 points) Construct the term-document matrix for the above documents that can be used in Boolean retrieval. The index terms have already been arranged for you alphabetically in the following table: Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 banking costs estate feds have home hotter interest kids lower market raising rates real rising speculationPage 7 of 16 B. (2 points each) What documents would be returned in response to the following queries? interest NOT rates ( interest AND rates ) NOT ( rising OR kids ) ( ( real AND estate ) OR home ) AND ( interest AND rates ) ( kids AND home ) Page 8 of 16 Doc 1: Interest in real estate speculation Doc 2: Interest rates and rising home costs Doc 3: Kids do not have an interest in banking Doc 4: Lower interest rates, hotter real estate market Doc 5: Feds’ interest in raising interest rates rising stopwords: an, and, do, in, not C. (20 points) Construct the vector space term-document matrix for the above documents (repeated from before) using tf.idf term weighting. Normalize your vectors. The following blank tables are provided for your convenience. You can use as many or as few of them as you wish. Clearly indicate your final answer. Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 banking costs estate feds have home hotter interest kids lower market raising rates real rising speculationPage 9 of 16 Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 banking costs estate feds have home hotter interest kids lower market raising rates real rising speculation Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 banking costs estate feds have home hotter interest kids lower market raising rates real rising speculation Page 10 of 16 D. (4 points each) Simulate the retrieval of documents in response to the following queries. Indicate the order in which documents will be retrieved, and the similarity score between the query and each document. interest rising real estate interest E. (2 points) Consider Doc 5: “Feds’ interest in raising interest rates rising.” Do the two instances of the term “interest” have the same meaning? What problem is this an example of?Page 11 of 16 Question 4. On hobbits (20 points) This question is about hobbits. Let’s say that hobbits come in two different eye colors: blue and brown. Let’s also say that some hobbits have small feet, and some hobbits have large feet (by hobbit standards, of course). Now, suppose we know the following things: • Brown-eyed hobbits are more common than blue-eyed hobbits. The probability that a random hobbit has brown eyes is 70%. • Blue-eyed hobbits tend to have larger feet. 65% of blue-eyed hobbits have “large feet”. • Brown-eyed hobbits tend to have smaller feet. 70% of brown-eyed hobbits have “small feet”. 500 hobbits totalBlue-eyed hobbits Brown-eyed hobbits1. 2.Blue-eyed hobbitswith big feetBlue-eyed hobbitswith small feetBrown-eyed hobbitswith big feetBrown-eyed hobbitswith small feet3. 4. 5. 6.500 hobbits totalBlue-eyed hobbits Brown-eyed hobbits1. 2.Blue-eyed hobbitswith big feetBlue-eyed hobbitswith small feetBrown-eyed hobbitswith big feetBrown-eyed hobbitswith small feet3. 4. 5. 6. A. (1 point) Are eye color and feet size independent for hobbits? (2 points each) For the following questions, write your answer in the box indicated. Out of 500 hobbits: B. How many blue-eyed hobbits would you expect? (Box 1) C. How many brown-eyed hobbits would you expect? (Box 2) D. How many blue-eyed hobbits with big feet would you expect? (Box 3) E. How many blue-eyed hobbits with small feet would you expect? (Box 4) F. How many brown-eyed hobbits with big feet would you expect? (Box 5) G. How many brown-eyed hobbits with small feet would you expect? (Box 6) Page