CS 345 Data Mining Crawling the Web Web

CS 345 Data Mining Crawling the Web

Web Crawling Basics Start with a “seed set” of to-visit urls get next url Web get page to visit urls visited urls extract urls web pages

Crawling Issues o o Load on web servers Insufficient resources to crawl entire web n o o o Which subset of pages to crawl? How to keep crawled pages “fresh”? Detecting replicated content e. g. , mirrors Can’t crawl the web from one machine n Parallelizing the crawl

Polite Crawling o Minimize load on web servers by spacing out requests to each server n o E. g. , no more than 1 request to the same server every 10 seconds Robot Exclusion Protocol n n Protocol for giving spiders (“robots”) limited access to a website www. robotstxt. org/wc/norobots. html

Crawl Ordering o o o Not enough storage or bandwidth to crawl entire web Visit “important” pages first Importance metrics n In-degree o n More important pages will have more inlinks Page Rank o o To be discussed later For now, assume it is a metric we can compute

Crawl Order o o Problem: we don’t know the actual indegree or page rank of a page until we have the entire web! Ordering heuristics n n Partial in-degree Partial page rank Breadth-first search (BFS) Random Walk -- baseline

stanford. edu experiment o 179 K pages Overlap with best x% by indegree x% crawled by O(u) Source: Cho et al (1998)

Larger study (328 M pages) BFS crawling brings in high quality pages early in the crawl Source: Najork and Wiener (2001)

Maintaining freshness o o o How often do web pages change? What do we mean by freshness? What strategy should we use to refresh pages?

How often do pages change? o o Cho et al (2000) experiment 270 sites visited (with permission) n n o identified 400 sites with highest “Page. Rank” contacted administrators 720, 000 pages collected n n n 3, 000 pages from each site daily start at root, visit breadth first (get new & old pages) ran only 9 pm - 6 am, 10 seconds between site requests

Average change interval Source: Cho et al (2000)

Modeling change o o Assume changes to a web page are a sequence of random events that happen independently at a fixed average rate Poisson process with parameter Let X(t) be a random variable denoting the number of changes in any time interval t Pr[X(t)=k] = e- t( t)k/k! for k = 0, 1, … “Memory-less” distribution

Poisson processes o Let us compute the expected number of changes in unit time E[X(1)] = kke k/k! = o o is therefore the average number of changes in unit time Called the rate parameter

Time to next event o o o Let T be a random variable denoting the time to the next event Verify that Pr[T>t] = e- t (t>0) The corresponding density function is f(t) = e- t o The distribution of change intervals should follow an exponential distribution

Change intervals of pages fraction of changes with given interval for pages that change every 10 days on average Poisson model Source: Cho et al (2000) interval in days

Change Metrics (1) - Freshness of element ei at time t is if ei is up-to-date at time t otherwise Freshness of the database S at time t is F( S ; t ) = 1 N N F( p ; t ) i=1 i (Assume “equal importance” of pages) web database pi pi. . . F(pi; t ) = 1 0 . . . o

Change Metrics (2) - Age of element ei at time t is A( pi ; t ) = 0 if ei is up-to-date at time t t-(modification time pi ) otherwise A( S ; t ) = 1 N N A( p ; t ) i=1 i (Assume “equal importance” of pages) web database pi pi. . . Age of the database S at time t is . . . o

Change Metrics (3) - Delay o o o Suppose crawler visits page p at times 0, 1, … Suppose page p is updated at times t 1, t 2, … tk between times 0 and 1 The delay associated with update ti is D(ti) = 1 - ti The total delay associated with the changes to page p is D(p) = i D(ti) At time 1 the delay drops back to 0 Total crawler delay is sum of individual page delays

Comparing the change metrics F(p) 1 0 time A(p) 0 time D(p) update refresh

Resource optimization problem o o Crawler can fetch M pages in time T Suppose there are N pages p 1, …, p. N with change rates 1, …, N How often should the crawler visit each page to minimize delay? Assume uniform interval between visits to each page

A useful lemma Lemma. For page p with update rate , if the interval between refreshes is , the expected delay during this interval is 2/2. Proof. Number of changes generated at between times t and t+dt = dt Delay for these changes = -t Total delay = = 2/2 t 0 dt

Optimum resource allocation Total number of accesses in time T = M Suppose allocate mi fetches to page pi Then Interval between fetches of page pi = T/mi Delay for page pi between fetches = Total delay for page p = Minimize subject to

Method of Lagrange Multipliers To maximize or minimize a function f(x 1, …xn) subject to the constraint g(x 1, …xn)=0 Introduce a new variable and define h = f - g Solve the system of equations: for i = 1, …, n g(x 1, …xn)=0 n+1 equations in n+1 variables

Optimum refresh policy Applying the Lagrange multiplier method to our problem, we have

Optimum refresh policy o o To minimize delay, we must allocate to each page a number of visits proportional to the square root of its average rate of change Very different answer to minimze the freshness and age metrics; see references.

Estimating the rate parameter o Simple estimator n n n o Visit page N times in time interval T Suppose we see that page has changed X times Estimate = X/T What is the problem with this estimator?

A better estimator o The page may have actually changed more than once between visits n o We therefore tend to understimate A better estimator is n For N=10, X=3, we get: o o o = 0. 3 with the simple estimator = 0. 34 with the improved estimator. Details in Cho and Garcia-Molina (2000)

Detecting duplicate pages o o Duplicates waste crawler resources We can quickly test for duplicates by computing a fingerprint for every page (e. g. , MD 5 hash) n o Make fingerprint long enough to make collisions very unlikely Problem: pages may differ only in ads or other formatting changes n Also an issue in counting changes for rate estimation

Detecting approximate duplicates o Can compare the pages using known distance metrics e. g. , edit distance n o o Takes far too long when we have millions of pages! One solution: create a sketch for each page that is much smaller than the page Assume: we have converted page into a sequence of tokens n Eliminate punctuation, HTML markup, etc

Shingling o o o Given document D, a w-shingle is a contiguous subsequence of w tokens The w-shingling S(D, w) of D, is the set of all w-shingles in D e. g. , D=(a, rose, is, a, rose) S(D, W) = {(a, rose, is, a), (rose, is, a, rose), (is, a, rose, is)} Can also define S(D, w) to be the bag of all shingles in D n We’ll use sets in the lecture to keep it simple

Resemblance o o Let us define the resemblance of docs A and B as: In general, 0 rw(A, B) 1 n n o Note rw(A, A) = 1 But rw(A, B)=1 does not mean A and B are identical! What is a good value for w?

Sketches o Set of all shingles is large n o o Can we create a document sketch by sampling only a few shingles? Requirement n o Bigger than the original document Sketch resemblance should be a good estimate of document resemblance Notation: Omit w from rw and S(A, w) n r(A, B)=rw(A, B) and S(A) = S(A, w)

Choosing a sketch o Random sampling does not work! n n o For s=1: E[|M(A) M(B)|] = 1/n n n o E. g. , suppose we have identical documents A & B each with n shingles M(A) = set of s shingles from A, chosen uniformly at random; similarly M(B) But r(A, B) = 1 So the sketch overlap is an underestimate Verify that this is true for any value of s

Choosing a sketch o Better method: M(A) is “smallest” shingle in A n n n o Assume set of shingles is totally ordered Also define M(AB) = min(M(A) M(B)) = smallest shingle across A and B Define r’(A, B) = |M(AB) M(A) M(B)| For identical documents A & B n r’(A, B) = 1 = r(A, B)

Example o o |S(A)|=|S(B)|=100, |S(A) S(B)|= 80 |S(A) S(B)| = 100+100 -80 = 120 r(A, B) = 80/120 = 2/3 Suppose M(AB) = x n o o o x could be any one of 120 shingles Pr[x M(A) M(B)] = Pr[x S(A) S(B)] = 80/120 = 2/3 E[r’(A, B)] = E[M(AB) M(A) M(B)] = 1(2/3) + 0(1/3) = 2/3 In general, for any documents A and B, it’s easy to generalize: E[r’(A, B)] = r(A, B)

Larger sketches o With a sketch of size 1, there is still a high probability of error n o For set W of shingles, let MINs(W) = set of s smallest shingles in W n o We can improve the estimate by picking a larger sketch of size s Assume documents have at least s shingles Define n n n M(A) = MINs(S(A)) M(AB) = MINs(M(A) M(B)) r’(A, B) = |M(AB) M(A) M(B)| / s

Larger sketches o Easy to show that E[r’(A, B)] = r(A, B) n n o Similar to the case s = 1 Repeat argument for each element of M(AB) By increasing sample size (s) we can make it very unlikely r’(A, B) is significantly different from r(A, B) n 100 -200 shingles is sufficient in practice

Corner case o Some docs have fewer than s shingles MINs(W) = set of smallest s elements of W, if |W|³ s W, otherwise o o o M(A) = MINs(S(A)) M(AB) = MINs(M(A) M(B)) r’(A, B) = |M(AB) M(A) M(B)| / |M(AB)|

Random permutations o One problem n n o Test is biased towards “smaller” shingles May skew results Solution n Let be the set of all shingles Pick a permutation : uniformly at random Define M(A) = MINs( (S(A)))

Random permutation

Implementation o Size of each shingle is still large n n o e. g. , each shingle = 7 English words = 4050 bytes 100 shingles = 4 K-5 K Compute a fingerprint f for each shingle (e. g. , Rabin fingerprint) n n 40 bits is usually enough to keep estimates reasonably accurate Fingerprint also eliminates need for random permutation

Finding all near-duplicates o Naïve implementation makes O(N^2) sketch comparisons n o Divide-Compute-Merge (DCM) n n n o Suppose N=100 million Divide data into batches that fit in memory Operate on individual batch and write out partial results in sorted order Merge partial results Generalization of external sorting

DCM Steps doc 1: s 11, s 12, …, s 1 k doc 2: s 21, s 22, …, s 2 k … Invert s 11, doc 1 s 12, doc 1 … s 1 k, doc 1 s 21, doc 2 … sort on shingle_id t 1, doc 11 t 1, doc 12 … t 2, doc 21 t 2, doc 22 … Invert and pair doc 11, doc 12, 1 doc 11, doc 12, 2 doc 11, doc 12, 1 doc 11, doc 13, 10 … … Merge doc 11, doc 13, 1 … sort on <docid 1, docid 2> doc 11, doc 12, 1 doc 11, doc 13, 1 … doc 21, doc 22, 1 …

Finding all near-duplicates 1. 2. 3. 4. 5. 6. 7. Calculate a sketch for each document For each document, write out the pairs <shingle_id, doc. Id> Sort by shingle_id (DCM) In a sequential scan, generate triplets of the form <doc. Id 1, doc. Id 2, 1> for pairs of docs that share a shingle (DCM) Sort on <doc. Id 1, doc. Id 2> (DCM) Merge the triplets with common docids to generate triplets of the form <doc. Id 1, doc. Id 2, count> (DCM) Output document pairs whose resemblance exceeds the threshold

Some optimizations o o “Invert and Pair” is the most expensive step We can speed it up eliminating very common shingles n n o Common headers, footers, etc. Do it as a preprocessing step Also, eliminate exact duplicates up front

Mirrors o Replication at a higher level n n o Entire collections of documents Java APIs, perl manuals, … Use document comparison as a building block