Distance functions and IE 3 William W Cohen

Distance functions and IE - 3 William W. Cohen CALD

Announcements • No meeting this Wed March 24 • March 25 Thus – talk from Carlos Guestrin on max-margin Markov nets – Newell-Simon Hall 1507 at 9: 30 am – no wait! – make that Wean Hall 4625 • Writeups: – today: “distance metrics for text” – three papers

Record linkage: definition • Record linkage: determine if pairs of data records describe the same entity – I. e. , find record pairs that are co-referent – Entities: usually people (or organizations or…) – Data records: names, addresses, job titles, birth dates, … • Main applications: – Joining two heterogeneous relations – Removing duplicates from a single relation – Storing results of information extraction in a database, or answering queries that involve information extracted from different places • Key step: measuring similarity of two strings – TFIDF metric (WHIRL) – Edit distance (Monge-Elkan)

The data integration problem

Levenshtein distance - example • distance(“William Cohen”, “Willliam Cohon”) s W I L L gap I A M _ C O H E N alignment t op cost WI L L L I A M _ C O H O N C C I C C C C S C 0 0 1 1 1 1 2 2

Computing Levenshtein distance D(i, j)= min M C C O H N D(i-1, j-1) + d(si, tj) //subst/copy D(i-1, j)+1 //insert D(i, j-1)+1 //delete C 1 1 2 3 4 5 O 2 2 3 4 H 3 3 2 3 E 4 4 3 3 N 5 5 4 3 = D(s, t)

Smith-Waterman distance c o h e n d o r f m 0 0 0 0 0 c 1 0 0 0 0 c 0 0 0 0 0 o 0 2 1 0 0 0 2 1 0 h 0 1 4 3 2 1 1 1 0 n 0 0 3 3 5 4 3 2 1 s 0 0 2 2 4 4 3 2 1 k 0 0 1 1 3 3 3 2 1 i 0 0 2 2 1 dist=5

Affine gap distances - 3 D(i, j) = max D(i-1, j-1) + + d(si, tj) //subst/copy D(i-1, j)-1 IS(I-1, j-1) + d(si, tj) //insert D(i, j-1)-1 //delete IT(I-1, j-1) + d(si, tj) IS(i, j) = max D(i-1, j) - A IS(i-1, j) - B IT(i, j) = max D(i, j-1) - A IT(i, j-1) - B Best score in which si is aligned with a ‘gap’ Best score in which tj is aligned with a ‘gap’

Record linkage: definition • Record linkage: determine if pairs of data records describe the same entity – I. e. , find record pairs that are co-referent – Entities: usually people (or organizations or…) – Data records: names, addresses, job titles, birth dates, … • Main applications: – Joining two heterogeneous relations – Removing duplicates from a single relation – Storing results of information extraction in a database, or answering queries that involve information extracted from different places • Key step: measuring similarity of two strings – TFIDF metric (WHIRL) – Edit distance (Monge-Elkan)

Inference in WHIRL • Explode p(X 1, X 2, X 3): find all DB tuples <p, a 1, a 2, a 3> for p and bind Xi to ai. • Constrain X~Y: if X is bound to a and Y is unbound, – find DB column C to which Y should be bound – pick a term t in X, find proper inverted index for t in C, and bind Y to something in that index • Keep track of t’s used previously, and don’t allow Y to contain one.

String distance metrics so far. . . • Term-based (e. g. TF/IDF as in WHIRL) – Distance depends on set of words contained in both s and t – so sensitive to spelling errors. – Usually weight words to account for “importance” – Fast comparison: O(n log n) for |s|+|t|=n • Edit-distance metrics – Distance is shortest sequence of edit commands that transform s to t. – No notion of word importance – More expensive: O(n 2) • Other metrics – Jaro metric & variants – Monge-Elkan’s recursive string matching – etc? • Which metrics work best, for which problems?

Jaro metric • Jaro metric is (apparently) tuned for personal names: – Given (s, t) define c to be common in s, t if it si=c, tj=c, and |i-j|<min(|s|, |t|)/2. – Define c, d to be a transposition if c, d are common and c, d appear in different orders in s and t. – Jaro(s, t) = average of #common/|s|, #common/|t|, and 0. 5#transpositions/#common – Variant: weight errors early in string more heavily • Fast to compute

Jaro metric

Winkler-Jaro metric

String distance metrics so far. . . • Term-based (e. g. TF/IDF as in WHIRL) – Distance depends on set of words contained in both s and t – so sensitive to spelling errors. – Usually weight words to account for “importance” – Fast comparison: O(n log n) for |s|+|t|=n • Edit-distance metrics – Distance is shortest sequence of edit commands that transform s to t. – No notion of word importance – More expensive: O(n 2) • Other metrics – Jaro metric & variants – Monge-Elkan’s recursive string matching – etc? • Which metrics work best, for which problems?

So which metric should you use? Second. String (Cohen, Ravikumar, Fienberg): • Java toolkit of string-matching methods from AI, Statistics, IR and DB communities • Tools for evaluating performance on test data • Exploratory tool for adding, testing, combining string distances – e. g. Second. String implements a generic “Winkler rescorer” which can rescale any distance function with range of [0, 1] • URL – http: //secondstring. sourceforge. net • Distribution also includes several sample matching problems.

Second. String distance functions • Edit-distance like: – Levenshtein – unit costs – untuned Smith-Waterman – Monge-Elkan (tuned Smith-Waterman) – Jaro and Jaro-Winkler – Less ad hoc Jaro variants • Term-based – TFIDF – Jaccard distance:

Second. String distance functions • Edit-distance like: – Levenshtein – unit costs – untuned Smith-Waterman – Monge-Elkan (tuned Smith-Waterman) – Jaro and Jaro-Winkler

Results - Edit Distances Monge-Elkan is the best on average. .

Edit distances

Second. String distance functions • Term-based, for sets of terms S and T: – TFIDF distance – Jaccard distance: – Language models: construct PS and PT and use

Second. String distance functions • Term-based, for sets of terms S and T: – TFIDF distance – Jaccard distance – Jensen-Shannon distance • smoothing toward union of S, T reduces cost of disagreeing on common terms • unsmoothed PS, Dirichlet smoothing, Jelenik-Mercer – “Simplified Fellegi-Sunter”

Results – Token Distances

Second. String distance functions • Hybrid term-based & edit-distance based: – Monge-Elkan’s “recursive matching scheme”, segmenting strings at token boundaries (rather than separators like commas) – Soft. TFIDF • Like TFIDF but consider not just tokens in both S and T, but tokens in S “close to” something in T (“close to” relative to some distance metric) • Downweight close tokens slightly

Results – Hybrid distances

Results - Overall

Prospective test on two clustering tasks

An anomolous dataset

An anomalous dataset: census

An anomalous dataset: census Why?