Efficient Estimation of Inclusion Coefficient using Hyper Log

  • Slides: 26
Download presentation
Efficient Estimation of Inclusion Coefficient using Hyper. Log Sketches Azade Nazi, Bolin Ding, Vivek

Efficient Estimation of Inclusion Coefficient using Hyper. Log Sketches Azade Nazi, Bolin Ding, Vivek Narasayya, Surajit Chaudhuri

Inclusion Coefficient (INC) •

Inclusion Coefficient (INC) •

Applications • Data integration • Relaxed containment measure • Detecting data quality issues such

Applications • Data integration • Relaxed containment measure • Detecting data quality issues such as missing values • Foreign-key and FD detection • INC is an important feature in detection algorithms • For example, [Rostin et al, 2009] and [Chen et al. , 2014]

Exact vs. Approximate INC Computation • Exact INC computation is expensive • Full scan/join

Exact vs. Approximate INC Computation • Exact INC computation is expensive • Full scan/join of data [Lopes et al. , 2002] • Too expensive for large database schema and tables • Worse if to be computed for all pairs of columns (foreign-key detection) • Approximate (estimated) INC: sketch-based approaches • Based on Bottom-k sketch [Cohen and Kaplan, 2007]: Jaccard -> INC [Zhang et al. , 2010] • This paper: better accuracy with error bounds (based on Hyperloglog sketch [Flajolet et al. , 2007])

Problem Definition • … …

Problem Definition • … …

Problem Definition •

Problem Definition •

Our Approach: BML (Binomial Mean-Lookup) •

Our Approach: BML (Binomial Mean-Lookup) •

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Hyperloglog Sketch Construction (Recap)

Key Ideas: Estimating INC from HLL Sketches • ?

Key Ideas: Estimating INC from HLL Sketches • ?

Binomial Mean-Lookup Estimator Binomial Mean Estimation ? Lookup

Binomial Mean-Lookup Estimator Binomial Mean Estimation ? Lookup

Binomial Mean-Lookup Estimator Binomial Mean Estimation ? Lookup

Binomial Mean-Lookup Estimator Binomial Mean Estimation ? Lookup

Error Bound Estimation Binomial Mean Estimation ? Chernoff bound Lookup

Error Bound Estimation Binomial Mean Estimation ? Chernoff bound Lookup

Experiments • Datasets

Experiments • Datasets

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments Execution Time k = 1536 Average Error (TPCDS-300)

Experiments: Small vs. Large Columns

Experiments: Small vs. Large Columns

Experiments: Small vs. Large Columns

Experiments: Small vs. Large Columns

Experiments: Small vs. Large Columns

Experiments: Small vs. Large Columns

Experiments: Foreign Key Detection • Plug estimated INC into [Rostin et al, 2009] and

Experiments: Foreign Key Detection • Plug estimated INC into [Rostin et al, 2009] and [Chen et al. , 2014]

Conclusion and Future Work • BML estimator for INC using Hyperloglog sketches • A

Conclusion and Future Work • BML estimator for INC using Hyperloglog sketches • A Maximum-Likelihood estimation schema • A novel error estimation method to produce data-dependent bound • Better accuracy than Bottom-k and can be used for FK detection • To be extended for other correlation measures

Thank You! Questions? Comments?

Thank You! Questions? Comments?

Log Parser Log Parser Log Parser Log ParserLog Parser Log Parser Log Parser Log Parser
HYPER PRESENTATION TEMPLATE HYPER PRESENTATION TEMPLATE HYPER PRESENTATIONHYPER PRESENTATION TEMPLATE HYPER PRESENTATION TEMPLATE HYPER PRESENTATION
Correlation Coefficient Using Technology Remember Correlation Coefficient indicatesCorrelation Coefficient Using Technology Remember Correlation Coefficient indicates
2 Log 2x32 2 Log 222 Log 22 Log 2x32 2 Log 222 Log 2
Ex log ab log ab log a bEx log ab log ab log a b
Ex log ab log ab log a bEx log ab log ab log a b
Table 1 Estimation Results log IPA log FOODMTable 1 Estimation Results log IPA log FOODM
Inclusion Equality and Inclusion Equality and inclusion hasInclusion Equality and Inclusion Equality and inclusion has
INCLUSION INCLUSION What is inclusion The practice ofINCLUSION INCLUSION What is inclusion The practice of
Inclusion What is inclusion Inclusion means that allInclusion What is inclusion Inclusion means that all
KAPPAK COEFFICIENT OF AGREEMENT 2 interrater reliability coefficientKAPPAK COEFFICIENT OF AGREEMENT 2 interrater reliability coefficient
Coefficient of Determination SECTION 3 3 Coefficient ofCoefficient of Determination SECTION 3 3 Coefficient of
Squared coefficient of variation l The squared coefficientSquared coefficient of variation l The squared coefficient
Binomial Coefficient Definition of Binomial coefficient For nonnegativeBinomial Coefficient Definition of Binomial coefficient For nonnegative
Binomial Coefficient Dynamic Programming The binomial coefficient isBinomial Coefficient Dynamic Programming The binomial coefficient is
CFD Results CD Drag Coefficient The drag coefficientCFD Results CD Drag Coefficient The drag coefficient
HTML HTML Hyper Text Markup Language Hyper TextHTML HTML Hyper Text Markup Language Hyper Text
Hyper Text 1965 Nelson Hyper Card 1987 AppleHyper Text 1965 Nelson Hyper Card 1987 Apple
Hyper Commercialism Digital Convergence Hyper Commercialism means emphasisHyper Commercialism Digital Convergence Hyper Commercialism means emphasis
Hyper FlashHyper RAM Hyper Flash S 26 KSSHyper FlashHyper RAM Hyper Flash S 26 KSS
Forecast of Altair Hyper MeshHyper Works Hyper MeshForecast of Altair Hyper MeshHyper Works Hyper Mesh
Hyper Gram EKG Programs Yellow Card The HyperHyper Gram EKG Programs Yellow Card The Hyper
TOPIC Hyper Commercialism and Digital Convergence Hyper CommercialismTOPIC Hyper Commercialism and Digital Convergence Hyper Commercialism
Hyper Text Transfer Protocol HTTP Hyper Text TransferHyper Text Transfer Protocol HTTP Hyper Text Transfer