Search in Googles Ngrams J S Roger Jang

Search in Google's Ngrams J. -S. Roger Jang (張智星) jang@mirlab. org http: //mirlab. org/jang MIR Lab, CSIE Dept. National Taiwan University

Linggle Goals � Linguistic search � Computer-assisted language learning Demos � http: //linggle. com � Query examples present a method _ propose * to at/in the afternoon discuss ? about the issue v. death penalty to v. education to v. ? prep. ? det. difficulty/difficulties 2/13

Google‘s Web 1 T N-gram Dataset Web 1 T � Blog for Web 1 T � Statistics 1, 024, 908, 267, 229 words of running text 1, 176, 470, 663 five-word sequences (appear at least 40 times) 13, 588, 391 unique words (appear at least 200 times) � Applications: machine translation, speech recognition, spelling correction, language learning, and others From sentences to n-grams � Document � N-grams 3/13

Offline Task: Inverted Index Documents (n-gram entries) � have a book � for good � good to have � good job Inverted index 0 � book 0 � for 1 � good 1 2 3 � have 0 2 � job 3 � to 2 �a Dictionary Postings 4/13

Online Task 1: Query Expansion Supported queries � Wild cards _ listen _ music * a * book � Alternatives ? discuss ? about the issue / in/at the afternoon Query expansion � To expand a query until it is composed of literals and “_ discuss ? about the issue in/at the afternoon a * book give * a * 5/13

Online Task 2: Merge Postings Find intersection of postings � give 2 5 7 54 78 97 267 �_ � joy 3 7 45 23 97 345 890 Ordering checkup Print based on descending frequency 6/13

Typical Approach Offline task: Inverted index � Extract all words from the n-gram dataset � Create a dictionary of all sorted unique words � Generate a posting for each word in the dictionary Online task: Query processing � Expand the query until it contains only words or "_". � Extract words from the query. � Retrieve each word (by binary search or the likes) and its posting from one of the n-gram sets. � Combine postings to have the candidate output set. � Generate final output by considering ordering, etc. � Sort and print the output ordered by frequency. 7/13

3 Steps for Query Expansion (1/2) First, expand "? ". � abc ? x/y/z _ ? */p abc _ _ */p x/y/z _ */p Second, expand "/". � abc x/y/z _ */p abc abc x x y y z z _ _ _ * p * p 8/13

3 Steps for Query Expansion (2/2) Last, expand "*". � give * a * give give give a a _ _ _ _ _ a a a _ _ _ Note that it will be easier to write a recursive function for each of the above expansions. _ _ _ a a _ _ a 9/13

Recursive Formula for Query Expansion Expand "? " � expand 1({? a, b, c}) is the union of expand 1({b, c}) a+expand 1({b, c}) Expand "/" � expand 2({a/b, c, d/e}) is the union of a+expand 2({c, d/e}) b+expand 2({c, d/e}) Expand "*" � expand 3({*, b, c, d}) is the union of expand 3({b, c, d}) _+_+expand 3({b, c, d}) 10/13

Web Resources Tutorials by Stanford NLP-Professor Dan Jurafsky & Chris Manning � Inverted index This HW does not need to do tokenization, normalization, stemming, stop words � Merge postings 11/13

How to Optimize Your Program Strategies for speedup � Use tries instead of binary search (***) � When sorting, use pointers instead of moving entries (***) � Do not use STL sets for postings. Write your own function for merging postings (**) � Read a file into memory before further processing (*) � Use constant-size arrays since n-gram files are fixed (*) Strategies for saving memory � Process tokens on the fly 12/13

Suggested schedule Time is limited! � Finish inverted index this week � Finish query expansion and overall test next week 13/13