CSCE 590 Web Scraping Information Extraction II Topics

CSCE 590 Web Scraping – Information Extraction II Topics n Information Retrieval framework revisited Readings: n Scrapy User manual – l https: //scrapy. org/doc/ – 1– l https: //doc. scrapy. org/en/1. 3/ Slide from Speech and Language Processing -- Jurafsky and Martin March 16, 2017 CSCE 590 Web. Scraping Sp 17

Figure 23. 1 Google as an IR engine – 2– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 2 Architecture of an IR system – 3– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Google Page. Rank http: //en. wikipedia. org/wiki/Page. Rank – 4– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Google Page. Rank continued “ Page. Rank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important". – 5– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

False or spoofed Page. Rank This spoofing technique, also known as 302 Google Jacking, was a known failing or bug in the system. Any page's Page. Rank could have been spoofed to a higher or lower number of the webmaster's choice and only Google has access to the real Page. Rank of the page. Spoofing is generally detected by running a Google search for a URL with questionable Page. Rank, as the results will display the URL of an entirely different site (the one redirected to) in its results. – 6– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Wolfram Alpha http: //www. wolframalpha. com/examples/Words. And. Ling uistics. html n-grams it was the best of times it was the worst of times Who was the president in 1888? What is pi? graphs on 11 vertices *Who was Grover Cleveland's vice-president? – 7– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Bing http: //www. bing. com – 8– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

– 9– Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Vector Space Model - Represents the terms that occur within the collection Term weights In a document if • keyword 1 occurs 4 times • keyword 2 occurs 7 times • keyword 3 occurs 0 times • … • keywordn occurs 3 times Then the vector representing the document is (4, 7, 0, … 3) – 10 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Sim(query, document). Dot-product Vector length Term-by-document matrix – 11 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 3 Visualization of the vector model – 12 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Inverse Document Frequency Term Weighting • Document importance • Inverse Document Frequency (Spark Jones 1972) • “Assign higher weights to more discriminative words” • Where N = total # of documents and ni = # documents that contain term i • – 13 – Tf-idf weighting (term-frequency x idf) Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Tf-idf weighted cosine Also used in summarization (page 794) Other topics: • Stemming • Stop-list – 14 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 4 Rank-Specific P and R for list of docs – 15 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 5 Interpolated Precision – 16 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 6 – 17 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Ways to improve User Queries Relevance feedback Query expansion thesaurus – 18 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 7 Factoid Question Answering Question Wolfram. Alpha Where is the Louvre? Bing Google 10 10 10+map What is the abbreviation for limited partnership? - 9 10 What are the names of Odin’s ravens? - 9 9 What currency is used in China? - 10 10 - 7 10 what kind of nuts are used in marzipan? What is the official language of Algeria? What is the telephone number of the University of South Carolina? How many pounds are in a stone? – 19 – Slide from Speech and Language Processing -- Jurafsky and Martin SCState 10 10 10 CSCE 590 Web. Scraping Sp 17

Figure 23. 8 Architecture – 20 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 9 Question Typology – 21 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 9 continued – 22 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 10 – 23 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 11 – 24 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 12 – 25 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 13 Example Summarization – 26 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 14 – 27 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 15 – 28 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 16 – 29 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 17 Features used in Supervised Classifiers – 30 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 18 – 31 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 19 Summarization Tagging Subject(S) Object(O) Oblique(X) – 32 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 20 Rewriting References – 33 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 21 Examples – 34 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Figure 23. 22 – 35 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17

Google translate http: //translate. google. com/translate_t# What is the number of graphs on 10 vertices? Translation: English » Arabic ﺍﻟﻘﻤﻢ؟ 10 ﻓﻲ ﺍﻟﺒﻴﺎﻧﻴﺔ ﺍﻟﺮﺳﻮﻡ ﻣﻦ ﻋﺪﺩ ﻫﻮ ﻣﺎ Translation: Arabic » English What is the number of charts in the 10 top? – 36 – Slide from Speech and Language Processing -- Jurafsky and Martin CSCE 590 Web. Scraping Sp 17