CS 276 A Text Information Retrieval Mining and

CS 276 A Text Information Retrieval, Mining, and Exploitation Lecture 14 21 November, 2002

Recap n Last 3 lectures n n n n search on the web link analysis anchor text and other heuristics mirror and spam detection crawl and index infrastructure behavioral ranking Today n n web characterization (“web archaeology”) size and shape of the web

What is the size of the web ? n Issues n The web is really infinite n n n Dynamic content, e. g. , calendar Soft 404: www. yahoo. com/<anything> is a valid page Static web contains syntactic duplication, mostly due to mirroring (~20 -30%) Some servers are seldom connected Who cares? n n n Media, and consequently the user Engine design Engine crawl policy. Impact on recall.

What can we attempt to measure? n The relative size of search engines n n The notion of a page being indexed is still reasonably well defined. Already there are problems n n Document extension: e. g. Google indexes pages not yet crawled by indexing anchortext. Document restriction: Some engines restrict what is indexed (first n words, only relevant words, etc. ) The coverage of a search engine relative to another particular crawling process. n. The ultimate coverage associated to a particular crawling process and a given list of seeds. n

Statistical methods n Random queries n Random searches n Random IP addresses n Random walks

URL sampling via Random Queries n n Ideal strategy: Generate a random URL and check for containment in each index. Problem: Random URLs are hard to find!

Random queries [Bhar 98 a] n Sample URLs randomly from each engine n 20, 000 random URLs from each engine n n n Test if present in other engines. n n Issue random conjunctive query with <200 results Select a random URL from the top 200 results Query with 8 rarest words. Look for URL match Compute intersection & size ratio Intersection = x% of E 1 = y% of E 2 E 1/E 2 = y/x E 1 E 2

Issues n Random narrow queries may bias towards long documents (Verify with disjunctive queries) n Other biases induced by process

Random searches n Choose random searches extracted from a local log [Lawr 97] or build “random searches” [Note 02] n n Use only queries with small results sets. Count normalized URLs in result sets. Use ratio statistics Advantage: n Might be a good reflection of the human perception of coverage

Random searches [Lawr 98, Lawr 99] n n n 575 & 1050 queries from the NEC RI employee logs 6 Engines in ’ 98, 11 in ‘ 99 Implementation: n n n Restricted to queries with < 600 results in total Counted URLs from each engine after verifying query match Computed size ratio & overlap for individual queries Estimated index size ratio & overlap by averaging over all queries Issues n n n Samples are correlated with source of log Duplicates Technical statistical problems (must have non-zero results, ratio average)

Queries from Lawrence and Giles study n n n n n 1. adaptive access control 2. neighborhood preservation topographic 3. hamiltonian structures 4. right linear grammar 5. pulse width modulation neural 6. unbalanced prior probabilities 7. ranked assignment method 8. internet explorer favourites importing 9. karvel thornber 10. zili liu n n n n n 11. softmax activation function 12. bose multidimensional system theory 13. gamma mlp 14. dvi 2 pdf 15. john oliensis 16. rieke spikes exploring neural 17. video watermarking 18. counterpropagation network 19. fat shattering dimension 20. abelson amorphous computing

Size of the Web Estimation [Lawr 98, Bhar 98 a] n Capture – Recapture technique n Assumes engines get independent random subsets of the Web E 2 contains x% of E 1. Assume, E 2 contains x% of the Web as well Knowing size of E 2 compute size of the Web Size of the Web = 100*E 2/x E 2 E 1 WEB Bharat & Broder: 200 M (Nov 97), 275 M (Mar 98) Lawrence & Giles: 320 M (Dec 97)

Random IP addresses [Lawr 99] n n Generate random IP addresses Find, if possible, a web server at the given address Collect all pages from server Advantages n Clean statistics, independent of any crawling strategy

Random IP addresses [ONei 97, Lawr 99] n HTTP requests to random IP addresses n n n Ignored: empty or authorization required or excluded [Lawr 99] Estimated 2. 8 million IP addresses running crawlable web servers (16 million total) from observing 2500 servers. OCLC using IP sampling found 8. 7 M hosts in 2001 n n Netcraft [Netc 02] accessed 37. 2 million hosts in July 2002 [Lawr 99] exhaustively crawled 2500 servers and extrapolated n n Estimated size of the web to be 800 million Estimated use of metadata descriptors:

Issues n n Virtual hosting Server might not accept http: //102. 93. 22. 15, No guarantee all pages are linked to root page Power law for # pages/host generates bias

Random walks [Henz 99, Bar. Y 00, Rusm 01] n n View the Web as a directed graph from a given list of seeds. Build a random walk on this graph n n Includes various “jump” rules back to visited sites Converges to a stationary distribution n n Time to convergence not really known Sample from stationary distribution of walk Use the “strong query” method to check coverage by SE “Statistically clean” method at least in theory! Could work even for infinite web (assuming convergence) under certain metrics.

Issues n n List of seeds is a problem. Practical approximation might not be valid n n Non-uniform distribution, subject to link spamming Still has all the problems associated with “strong queries”

Recent Measurements Source: http: //www. searchengineshowdown. com/stats/change. shtml

Conclusions n n No sampling solution is perfect. Lots of new ideas. . . . but the problem is getting harder Quantitative studies are fascinating and a good research problem

Questions about the web graph n How big is the graph? n n How many links on a page (outdegree)? How many links to a page (indegree)? Can one browse from any web page to any other? How many clicks? Can we pick a random page on the web? n (Search engine measurement. )

Questions about the web graph n n n Can we exploit the structure of the web graph for searching and mining? What does the web graph reveal about social processes which result in its creation and dynamics? How different is browsing from a “random walk”?

Why? n Exploit structure for Web algorithms n n n Crawl strategies Search Mining communities Classification/organization Web anthropology n n Prediction, discovery of structures Sociological understanding

Web snapshots n n Altavista crawls (May 99/Oct 99/Feb 00) 220/317/500 M pages 1. 5/2. 1 B/5 B hyperlinks Compaq CS 2 connectivity server n n n back-link information 10 bytes/url, 3. 4 bytes/link, 0. 15 s/access given pages, return their in/out neighborhood

Algorithms n n Weakly connected components (WCC) Strongly connected components (SCC) Breadth-first search (BFS) Diameter

Challenges from scale n Typical diameter algorithm: number of steps ~ pages links. n For 500 million pages, 5 billion links, even at a very optimistic 0. 15 s/step, we need ~4 billion seconds. Hopeless. n n Will estimate diameter/distance metrics.

Scale n On the other hand, can handle tasks linear in the links (5 billion) at ~1 s/step. n n n E. g. , breadth-first search First eliminate duplicate pages/mirrors. Linear-time implementations for WCC and SCC.

May 1999 crawl n n n 220 million pages after duplicate elimination. Giant WCC has ~186 million pages. Giant SCC has ~56 million pages. n n n Cannot browse your way from any page to any other Next biggest SCC ~150 K pages Fractions roughly the same in other crawls.

Tentative picture SCC 56 M pages WCC 186 M pages Where did this come from? Disconnected debris 34 M pages

Breadth-first search (BFS) n Start at a page p n n get its neighbors; their neighbors, etc. Get profile of the number of pages reached by crawling out of p, as a function of distance d Can do this following links forwards as well as backwards

BFS experiment n n Start at 1000+ random pages For each start page, build BFS (reachability vs. distance) profiles going forwards, and backwards

Reachability How many pages are reachable from a random page?

Net of BFS experiments n BFS out of a page n n either dies quickly (~100 pages reached) “explodes” and reaches ~100 million pages n n n somewhat over 50% of starting pages SCC pages ~25% of total, reach >56 M pages Qualitatively the same following in- or outlinks

Interpreting BFS expts n Need another 100 -56 = 44 M pages reachable from SCC n n n gives us 100 M pages reachable from SCC Likewise, need another ~44 M pages reachable from SCC going backwards These together don’t account for all 186 M pages in giant WCC.

Web anatomy …/~newbie/ www. ibm. com /…/…/leaf. htm

Distance measurements n n n For random pages p 1, p 2: Pr[p 1 reachable from p 2] ~ 1/4 Maximum directed distance between 2 SCC nodes: >28 Maximum directed distance between 2 nodes, given there is a path: > 900 Average directed distance between 2 SCC nodes: ~16 Average undirected distance: ~7

Exercise n Given the BFS and component size measurements, how can we infer all of the above measurements?

Bibliometric Laws n Lotka’s Law of Scientific Productivity n n Number of authors making n contributions is proportional to 1/n 2 Bradford’s Law of Scattering Journals in a field are in groups of size k, kn 2 contributing equal numbers of (useful) articles. (Roughly: For every good journal there are n mediocre journals with articles that are 1/n th as useful) n n Zipf’s Law (Sociological Models, [Zipf 49]) n n Freq(t) proportional to 1/rank(t) where is close to 1 Also by Yule (Linguistic Vocabulary, [Yule 44]) and by Pareto (Economic Theory, [Pa 1897])

Power laws on the Web n Inverse polynomial distributions: Pr[k] ~ c/k for a constant c. log Pr[k] ~ c - log k n Thus plotting log Pr[k] against log k should give a straight line (of negative slope).

Zipf-Pareto-Yule Distributions on the Web n In-degrees and out-degrees of pages [Kuma 99, Bara 99, Brod 00] n Connected component sizes n n Both directed & undirected Host in-degree and out-degree n n n [Brod 00] Both in terms of pages and hosts Also within individual domains [Bhar 01 b] Number of edges between hosts [Bhar 01 b]

In-degree distribution Probability that a random page has k other pages pointing to it is ~k -2. 1 (Power law) Slope = -2. 1

Out-degree distribution Probability that a random page points to k other pages is ~k -2. 7 Slope = -2. 7

Connected components Largest WCC = 186 M, SCC = 56 M Connected component sizes: