CS6322: Information Retrieval Sanda Harabagiu Lecture 2: The term vocabulary and postings listsCS 6322 Information Retrieval Recap of the previous lecture Basic inverted indexes: Structure: Dictionary and Postings Key step in construction: Sorting Boolean query processing Intersection by linear time “merging” Simple optimizations Overview of course topicsCS 6322 Information Retrieval Plan for this lecture Elaborate basic indexing Preprocessing to form the term vocabulary Documents Tokenization What terms do we put in the index? Postings Faster merges: skip lists Positional postings and phrase queriesCS 6322 Information Retrieval Recall the basic indexing pipeline Tokenizer Token stream. Friends Romans Countrymen Linguistic modules Modified tokens. friend roman countryman Indexer Inverted index. friend roman countryman 2 4 2 13 16 1 Documents to be indexed. Friends, Romans, countrymen.CS 6322 Information Retrieval Parsing a document What format is it in? pdf/word/excel/html? What language is it in? What character set is in use? Each of these is a classification problem, which we will study later in the course. But these tasks are often done heuristically … Sec. 2.1CS 6322 Information Retrieval Complications: Format/language Documents being indexed can include docs from many different languages A single index may have to contain terms of several languages. Sometimes a document or its components can contain multiple languages/formats French email with a German pdf attachment. What is a unit document? A file? An email? (Perhaps one of many in an mbox.) An email with 5 attachments? A group of files (PPT or LaTeX as HTML pages) Sec. 2.1CS 6322 Information Retrieval TOKENS AND TERMSCS 6322 Information Retrieval Tokenization Input: “Friends, Romans and Countrymen” Output: Tokens Friends Romans Countrymen A token is an instance of a sequence of characters Each such token is now a candidate for an index entry, after further processing Described below But what are valid tokens to emit? Sec. 2.2.1CS 6322 Information Retrieval Tokenization Issues in tokenization: Finland’s capital Finland? Finlands? Finland’s? Hewlett-Packard Hewlett and Packard as two tokens? state-of-the-art: break up hyphenated sequence. co-education lowercase, lower-case, lower case ? It can be effective to get the user to put in possible hyphens San Francisco: one token or two? How do you decide it is one token? Sec. 2.2.1CS 6322 Information Retrieval Numbers 3/20/91 Mar. 12, 1991 20/3/91 55 B.C. B-52 My PGP key is 324a3df234cb23e (800) 234-2333 Often have embedded spaces Older IR systems may not index numbers But often very useful: think about things like looking up error codes/stacktraces on the web (One answer is using n-grams: Lecture 3) Will often index “meta-data” separately Creation date, format, etc. Sec. 2.2.1CS 6322 Information Retrieval Tokenization: language issues French L'ensemble one token or two? L ? L’ ? Le ? Want l’ensemble to match with un ensemble Until at least 2003, it didn’t on Google Internationalization! German noun compounds are not segmented Lebensversicherungsgesellschaftsangestellter ‘life insurance company employee’ German retrieval systems benefit greatly from a compound splitter module Can give a 15% performance boost for German Sec. 2.2.1CS 6322 Information Retrieval Tokenization: language issues Chinese and Japanese have no spaces between words: 莎拉波娃现在居住在美国东南部的佛罗里达。 Not always guaranteed a unique tokenization Further complicated in Japanese, with multiple alphabets intermingled Dates/amounts in multiple formats フォーチュン500社は情報不足のため時間あた$500K(約6,000万円) Katakana Hiragana Kanji Romaji End-user can express query entirely in hiragana! Sec. 2.2.1CS 6322 Information Retrieval Tokenization: language issues Arabic (or Hebrew) is basically written right to left, but with certain items like numbers written left to right Words are separated, but letter forms within a word form complex ligatures ← → ← → ← start ‘Algeria achieved its independence in 1962 after 132 years of French occupation.’ With Unicode, the surface presentation is complex, but the stored form is straightforward Sec. 2.2.1CS 6322 Information Retrieval Stop words With a stop list, you exclude from the dictionary entirely the commonest words. Intuition: They have little semantic content: the, a, and, to, be There are a lot of them: ~30% of postings for top 30 words But the trend is away from doing this: Good compression techniques (lecture 5) means the space for including stopwords in a system is very small Good query optimization techniques (lecture 7) mean you pay little at query time for including stop words. You need them for: Phrase queries: “King of Denmark” Various song titles, etc.: “Let it be”, “To be or not to be” “Relational” queries: “flights to London” Sec. 2.2.2CS 6322 Information Retrieval Normalization to terms We need to “normalize” words in indexed text as well as query words into the same form We want to match U.S.A. and USA Result is terms: a term is a (normalized) word type, which is an entry in our IR system dictionary We most commonly implicitly define equivalence classes of terms by, e.g., deleting periods to form a term U.S.A., USA USA deleting hyphens to form a term anti-discriminatory, antidiscriminatory antidiscriminatory Sec. 2.2.3CS 6322 Information Retrieval Normalization: other languages Accents: e.g., French résumé vs. resume. Umlauts: e.g., German: Tuebingen vs. Tübingen Should be equivalent Most important criterion: How are your users like to write their queries for these words? Even in languages that standardly have accents, users often may not type them Often best to normalize to a de-accented term Tuebingen, Tübingen, Tubingen Tubingen Sec. 2.2.3CS 6322 Information Retrieval Normalization: other languages Normalization
View Full Document
Unlocking...