Extracting Structured Data from Web Page Arvind Arasu

Extracting Structured Data from Web Page Arvind Arasu, Hector Garcia-Molina ACM SIGMOD 2003

Outline • Introduction • Model, Problem Formulation • Equivalence Classes – Observations and Properties • Build Template and Extract Values • Experiments • Conclusion

Introduction • Keyword: Schema (Data having a structure) • Problem Definition: automatically extracting schema encoded in a given collection of pages, without any human input • Cue: characteristic of pages belonging to the same site and encoding data of the same schema, is that data encoding in a consistent manner =＞ a common template by plugging-in value

Figuration

Goal and Challenge • Previous IE Techniques rely on heuristic by human. ex. wrapper • Goal: to deduce the template without human – Time consuming and error-prone – Optional attributes are ignored • Challenge: – No obvious way of differentiating what text is template or data – The schema of data in pages isn’t flat but more complex and semi-structured of attributes

Model, Problem Formulation • • • Structured Data Model of Page Creation Optionals and Disjunctions Problem Statement Miscellaneous Terminology, Definition

Structured Data • Token: A token is some basic unit of text • Structured Data: any set of data values conforming to a common schema or type • Define “Type”: – 1. Basic Type (β): string of tokens eg. ＜html＞, text – 2. Ordered List Type: tuple constructor of order “n” eg. ＜T 1, T 2, …, Tn＞, T 1, T 2, …, Tn : type – 3. Define Type: set constructor – eg. {T} , T: type

Define term value and example • Define “instance”: – 1. An instance of basic type, β, is any string of tokens – 2. An instance of type ＜T 1, T 2, …, Tn＞ is a 　　tuple of the form ＜i 1, i 2, …, in＞, where attributes i 1, i 2, …, in are instances of types. T 1, T 2, …, Tn – 3. An instance of type {T}, is any set of elements {e 1, e 2, …, em}, such ei is an instance of type T • Instance → Value; String → a string of tokens • Example: – Schema S 1= – Value =

Schemas and Values as Trees

Model of Page Creation • Definition: A template T for a schema S (as shown TS), is defined as a function that maps each type constructor τ of S into an ordered set of strings T(τ ), such that, – Ifτis the tuple constructor of order n, T(τ) is an order set of n+1 string <Cτ1, Cτ2 , Cτ3, …Cτ(n+1) > – Ifτis the set constructor, T(τ) is a string Sτ

Example • A template T for schema S 1 is given by the mapping: – T(t 1)=<A, B, C, D> – T(t 2)=H – T(t 3)=<E, F, G>

Encoding of a value x S • 1. if x β, then λ (T, x)→x • 2. if x <x 1, x 2, …, xn＞τt λ (T, x) → C 1 λ (T, x 1) C 2 …λ (T, xn) Cn+1 • 3. if x {e 1, e 2, …, em}τs , τs S λ (T, x) → λ (T, e 1) S λ (T, e 2) …. S λ (T, em)

Example of Schema S 1

Optionals and Disjunctions • Optional: – If T is a type, optional type (T)? ≡{T}τ |τ| = 0 or 1 • Disjunction: – If T 1 and T 2 is type, disjunction type (T 1| T 2) ≡ ＜{T 1}τ1, {T 2}τ2 ＞τ |τ1|+|τ2| = 1

Problem Statement • Extract Problem: n pages, each page pi = λ(T, xi) (1 ≤ i ≤ n), is created from some unknown deduction template T and values {x 1, . . . , xn} from the set of pages alone

Example of correct solution of EXTRACT

Example of correct solution of EXTRACT (cont. ) T(te 1)=<li>Reviewer Name, Rating, Text, </li> T(te 2)= T(te 3)=<html><body>Book Name, Reviewers<ol>, </ol></body></html>

Miscellaneous Terminology, Definition • A token is a word or a HTML tag • An occurrence of a token in page (resp. value, template) is called a page-token (resp. valuetoken, template-token) • Each page token is created from either a template-token or a value-token • 2 page-token in Pe have the same role iff they have been generated by the same templatetoken

Overview Approach - EXALG (ECGM) Stage 2 Stage 1

Equivalence Classes Pages P = { p 1, … , pn } , pi = λ(TS, xi) TS = {τ1, … , τk }: type constructor • Definition (Occurrence Vector): – The occurrence-vector of a token t, is defined as the vector <f 1, f 2, …, fn>, where fi is the number of occurrences of t in pi • Definition (Equivalence Classes): All tokens of equivalence class have the same occurrence vector. – Ex. ε 1: { <html>, <body>, Book, Reviews, <ol>, </body>, </html> } <1, 1, 1, 1> – Ex. ε 2: {Data, Mining, Jeff, 2, Jane, 6} <0, 1, 0, 0> – Ex. ε 3: { <li>, Reviewer, Rating, Text, </li> } <1, 2, 1, 0>

Equivalence Classes: Observations • Observation 1 : – • Observation 2: – • Tokens associated with the same type constructor τj in T that have unique-roles occur in the same equivalence class. (used to decide EQ valid or not) For real pages, an equivalence class of large size and support is usually valid Definition – – Support of token: #(page contain) Size of EQ class: #(token of EQ)

Properties of EQ class • Definition (Ordered Equivalence Classes): – An EQ class is ordered, if its tokens can be ordered <t 1, t 2, …, tm>, such that, for every page pi and every pair of tj, tk (1 j k m) • If tj occurs at least l times in pi, the lth occurrence of tj in pi occurs before the lth occurrence of tk in pi and • If tj occurs at least (l+1) times in pi, the (l+1)th occurrence of tj in pi is after the lth occurrence of tk in pi. • Definition (Nesting of EQ classes): – A pair of EQ classes εi and εj is nested if, • The span of any occurrence of εi does not overlap with the span of any occurrence of εj , or • The span of all occurrences of εi is within Pos(p) of some occurrence of εj for some fixed p; or vice-versa.

EQ Classes: Observations (Cont. ) • Observation 3 : – • A valid equivalence class is ordered and a pair of two valid equivalence classes is nested. Handling Invalid Equivalence Classes – Detect the existence of invalid LFEQs using violation of ordered and nesting – Yes, discard some of LFEQs and break other into smaller LFEQs

Differentiating roles of tokens • By Path – different roles of tokens are in different path of HTML parse tree • By Position – different roles of tokens locates at different Position (non-empty) • Observation 4: – In practice, two page-tokens with different occurrence paths have different roles. • Observation 5: – For a valid EQ class . The role of an occurrence of t, which is within Pos(l) of some occurrence of is different from the role of an occurrence of t which is within Pos(m) (m l) of some occurrence of .

DIFFFORM (step 1) and DIFFEQ (step 4) • These module are used to add more tokens to LFEQ by “differentiating” roles – Ex. Name has multiple “role”, one occurs in Book Name and the other occurs in Reviewer Name • Differentiate the multiple roles : – The multiple tokens occur in different path from root in the HTML parse tree (DIFFFORM) – The multiple tokens occur in different “Position” with respect to LFEQ εe 1(DIFFEQ) • dtoken (differentiated tokens): – ex. Name 5 and Name 14 are regarded as different tokens Name. A and Name. B

Stage 1: ECGM Find dtoken from path in html parse tree Find LFEQ Detect and remove invalid LFEQ (using violation of order and nesting) Find dtoken from position in valid LFEQ

Running Example • ECGM: – OUTPUT: set of LFEQs of dtokens and page represented as string of dtokens – Two parameters used to consider LFEQs • SIZETHRES=3, SUPTHRES=3

Iteration 1: Diff. FORM, Find. EQ • <1, 1, 1, 1>={<html>, <body>, Book, Name, Reviews, <ol>, </body>, </html>} Use path • <2, 2, 2, 2>={, } : <html><body> • <3, 6, 3, 0>={, } : <html><body><ol> • <1, 2, 1, 0>={<li>, Reviewer, Name, Rating, Text, </li>} • <1, 0, 0, 0>={Database} Not LFEQ • <0, 1, 0, 0>={Data, Mining, Jeff, Jane} • <0, 0, 1, 0>={Query, Opt. } • <0, 0, 0, 1>={Transactions} • <1, 0, 1, 0>={John}

Iteration 1: Diff. EQ • <1, 1, 1, 1>={<html>, <body>, Book, Name, Reviews, <ol>, </body>, </html>} Use position • : at pos 2 or pos 4 • : at pos 4 or pos 5 • εe 1 : <1, 1, 1, 1>= { <html><body>Book Name, Reviews<ol>, </ol></body></html> } 8 → 13 • <1, 2, 1, 0>={<li>, Reviewer, Name, Rating, Text, </li>} • : at pos 1 or pos 3 or pos 4 • : at pos 3 or pos 4 or pos 5 • εe 3: <1, 2, 0, 1>={ <li>Reviewer Name, Rating , Text, </li>} 6 → 12

Stage 2: Construct Schema from ECGM • Input to this module is {ε 1 , ε 2 , … , εm } • The ANALYSIS consist of 2 modules – CONSTTEMP and EXVAL • CONSTTEMP , εi = { d 1, d 2, … , dl } – Start the basic ε 1= { <html>, <body>, … , </body>, </html> } – recursively constructs a template Tεi , corresponding toεi , and template Tεi, p, corresponding to each nonempty position p ofεi – Checks if the set of strings, Pos. String(εi , p), corresponding has some recognizable pattern

• Construct Schema S’ fromεe 1: { <html>, <body>, , Book, Name, , , Reviews, , <ol>, </body>, </html> } → T(τe 1) = <Te 1, 1, Te 1, 2><C 11, C 12, C 13>

Cont. • Pos. String(εe 1+ , 6) is a string of dtokens for every occurrence of εe 1+, which matches Pattern 5 of table; →T(Te 1, 1)= β • Pos. String(εe 1+ , 10) is always a string of 0 or more occurrences of εe 3+, which matches Pattern 1 → T(Te 1, 2) ={τe 3} → T(τe 3) = < Te 3, 1, Te 3, 2, Te 3, 3 >< C 31, C 32, C 33, C 34 > <li>Reviewer Name Rating Text </li>

(Cont. ) • The three non-empty positions are all Basic Type β →T(Te 3, 1)= β →T(Te 3, 2)= β →T(Te 3, 3)= β S = < β, { <β, β, β, >τe 3 } >τe 1

Example of correct solution of EXTRACT

Evaluation Data sets: http: //www-db. stanford. edu/~arvind/extract/ Leaf attribute Am in schema Sm • Correct: the set of Am in the page is equal to the set of extracted value Ae in the page • Partially Correct: the set of Am in the page is not equal to the set of extracted value Ae in the page, but as part of value of Ae • Incorrect: not correct and Partially correct

Assumption • The 4 assumptions: (A 1) A large number of tokens occurring in template have unique roles (A 2) The EQ class derived from a type constructor is recognized as an LFEQ (A 3) Irregularity in encoded data that leads to invalid EQ class (A 4) The separators are around data values. In this model, strings associated with type construction are non-empty position

Result • 18 or 40% of input collections our System correctly extracted all the attribute • Around 80% of the attributes were extracted correctly • Normalized average • Input size <=10 • Parameter = 3

Conclusion • EXALG: use 2 novel concepts – equivalence classes and – differentiate roles, to discovery the template • Impact of the failed assumption is limit to a few attributes • Future work: – Develop techniques for crawling, indexing, and providing querying support for the structured pages in the web – Develop techniques for automatically annotating the extracted data, possibly using the words that appear in the template