Fusing Data with Correlations Ravali Pochampally Anish Das

Fusing Data with Correlations Ravali Pochampally, Anish Das Sarma, Luna Dong, Alexandra Meliou, Divesh Srivastava AT&T Research

Fusion in web data extraction extractor extractor <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> How can we purge wrong triples from the dataset? Applications: Building knowledge bases, answer questions, facilitate data mining 1

The data fusion problem Contribution: Fusion techniques that consider source quality and correlations bad good Knowledge triple S 1 S 2 <Daniel Radcliffe, played role, Harry Potter> ✓ ✓ <Daniel Radcliffe, spouse, Bonnie Wright> ✓ ✓ <Daniel Radcliffe, acted in, Frankenstein> S 3 S 4 ✓ ✗ ✓ <Emma Watson, acted in, Harry Potter> ✓ ✓ ✓ <J. K. Rowling, acted in, Harry Potter> ✓ ✓ ✓ <Richard Harris, played role, Dumbledore> ✗ ✓ <Michael Gambon, played role, Dumbledore> ✓ <Tim Burton, directed, Harry Potter> ✓ ✓ <Daniel Craig, acted in, Harry Potter> ✓ ✓ <Rupert Grint, acted in, Harry Potter> ✓ ✓ correlated ✗ ✗ ✓ anti-correlated 2

This talk 2 techniques Prec. Rec: consider source quality Prec. Rec. Corr: consider correlations extractor approximations future directions evaluation extractor extractor <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> diagnosis 3

source quality correlations evaluation future directions High-level intuition Researcher affiliation S 1 S 2 S 3 UMich ATT UMich Dewitt MSR UWisc Bernstein MSR MSR Carey UCI ATT BEA Franklin UCB UMD Jagadish 4

source quality correlations evaluation future directions High-level intuition Researcher affiliation S 1 S 2 S 3 UMich ATT UMich Dewitt MSR UWisc Bernstein MSR MSR Carey UCI ATT BEA Franklin UCB UMD Jagadish Voting: Trust the majority 5

source quality correlations evaluation future directions High-level intuition Researcher affiliation S 1 S 2 S 3 UMich ATT UMich Dewitt MSR UWisc Bernstein MSR MSR Carey UCI ATT BEA Franklin UCB UMD Jagadish Quality-based: More votes to accurate sources 6

source quality correlations evaluation future directions Source quality in extraction Actors/actresses in “Harry Potter” films S 1 S 2 Daniel Radcliffe ✓ ✓ Emma Watson ✓ ✓ J. K. Rowling S 3 ✓ ✓ Daniel Craig ✓ Rupert Grint ✓ high recall ✗ ✗ ✓ high precision Considering source quality: -- More likely to be correct if extracted by high-precision source. -- More likely to be wrong if not extracted by high-recall source. med prec/rec 7

source quality correlations evaluation future directions Source quality metrics probability to return a true triple o Recall: o False positive rate: probability to return a false triple A source is good if ri > qi 8

source quality correlations evaluation future directions Accounting for quality o Compute score § If for each triple: extracts it, multiply by • Good source higher score • Bad source lower score § If does not extract it, multiply by • Good source lower score • Bad source higher score 9

source quality correlations evaluation future directions Correlation scenarios Triple provided by good sources with recall r and FPR q o Copying: o Overlapping § On true triples: § On false triples: o Complementary sources: Correlations capture richer information than copying relationships 10

source quality correlations evaluation future directions Correlation in web extraction [Dong et al. PVLDB 2014] Significant negative correlation 11

source quality correlations evaluation future directions Considering correlations o Positive correlation: o Negative correlation: joint recall Exact solution: We can express these probabilities using an exponential number of correlation parameters 12

source quality correlations evaluation future directions Aggressive approximation o Partial independence assumptions correlation between Si and the other sources linear number of parameters But: low accuracy 13

source quality correlations evaluation future directions Approximation levels add parameters closer approximation no independence assumptions partial independence assumptions exact solution high accuracy exponential size elastic approximation aggressive approximation trade efficiency for accuracy low accuracy linear size 14

source quality correlations evaluation future directions Elastic approximation 3 steps 3 iterations achieve near-optimal accuracy Iterations of the elastic approximation 15

source quality correlations evaluation future directions Comparisons o Our techniques: Prec. Rec & Prec. Rec. Corr o Union-K § A triple is correct if at least K% of sources provide it o 3 -Estimate [Galland et al. WSDM 2010] § Iteratively computes trustworthiness o LTM [Zhao et al. PVLDB 2012] § Uses graphical models and Gibbs sampling 16

source quality correlations evaluation future directions Three real-world datasets o Restaurant: [Marian et al. DE Bull, 2011] § 7 sources § 93 triples o Book: [Dong et al. PVLDB, 2009] § 879 sources § 225 triples o Re. Verb: [Fader et al. EMNLP, 2011] § 6 extractors § 2407 triples 17

source quality correlations evaluation future directions Restaurant 18

source quality correlations evaluation future directions Book 19

source quality correlations evaluation future directions Re. Verb 20

source quality correlations evaluation future directions Synthetic data: low precision 21

source quality correlations evaluation future directions Synthetic data: high precision 22

source quality correlations evaluation future directions Synthetic data: low recall 23

source quality correlations evaluation future directions Synthetic data: correlations 24

source quality correlations evaluation future directions Error diagnosis <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> <subject, predicate, object> 25

Contributions o Fusion techniques that consider source quality and correlations o The number of correlation parameters grows exponentially, but we provide a scalable solution o Evaluation on real-world and synthetic data shows that our techniques are more effective than the state-of-the-art 26