Information Extraction on the Web ChiaHui Chang Department
Information Extraction on the Web Chia-Hui Chang Department of Computer Science & Information Engineering National Central University chia@csie. ncu. edu. tw
Outline What is information extraction? Document types Applications Wrapper induction Automatic Wrapper generator Conclusions
What’s information extraction? An information extraction system is a cascade of transducers or modules that at each step add structure and often lose information, hopefully irrelevant, by applying rules that are acquired manually and/or automatically. Example-- Parser n input a sequence of lexical items and perhaps small-scale structures (phrases) and output a set of parse tree fragments, possibly complete
Modules Text Zoner turn a text into a set of text segments Preprocessor turn a text or text segment into a sequence of sentences, each of which is a sequence of lexical items, where a lexical item is a word together with its lexical attributes Filter turn a set of sentences into a smaller set of sentences by filtering out the irrelevant ones
Document types Plain text: (一句一句，平鋪直述) n n 利用lexical、semantic analysis。 Auto. Slog(Riloff 93), LIEP(Huffman 95), CRYSTAL(Soderland 95), HASTEN(Krupka 95)。 Web page：(半結構性文件) n n 利用html語法特性-tag。 觀察所得之heuristics: Layout。
Applications Meta Search Engines Information Agents n 以特定目的為導向，例: w 新聞代理人(News spider) w 網羅新聞 w 購物比價 w 找 作 n Shop. Bot (Doorenbos 97), Software LEGO(Hsu 99)。
Information Integration Systems Abstracted Information Mediation Human & Computer Users User Services: • Query • Monitor • Update Information Integration Service Mediator Wrapper Unprocessed, Unintegrated Details Agent/Module Coordination Wrapper Text, Hierarchical Images/Video, & Network Spreadsheets Databases Mediator SQL Relational Databases ORB Object & Knowledge Bases Heterogeneous Data Sources Semantic Integration Translation and Wrapping
What is a wrapper? Wrapper n An extracting program to extract desired information from Web pages. Semi-Structure Doc. – wrapper→ Structure Info.
Web Wrappers Web wrappers wrap. . . n n “Query-able’’ or “Search-able’’ Web sites Web pages with large itemized lists The primary issues are: n How to build the extractor quickly?
Free Text Extraction v. s. Semi-structured Text Extraction Example: to extract attributes --- job title, employer and phone number --from a job item list n Free text extraction can depend on NL knowledge “The department of computer science at Cranberry Lemon University has a faculty position opening. Please call (555)333 -5555 for more details. ” n Semistructured text extraction? --- depend on appearance and regularity “Faculty position, department of computer science, Cranberry Lemon University. Call (555)333 -5555”
Wrapper Representations Delimiter-based finite state automata <HTML><TITLE>Some Country Codes</TITLE><BODY> <B>Congo</B><I>242</I><BR> <B>Egypt</B><I>20</I><BR> <B>Belize</B><I>501<I><BR> <B>Spain</B><I>34</I><BR> </BODY></HTML> skip extract 1 2 3 4 <B> </B> <I> </I>
Related Work Shopbot n Doorenbos, Etzioni, Weld, AA-97 Ariadne n Ashish, Knoblock, Coopis-97 WIEN n Kushmerick, Weld, IJCAI-97
Related Work (Cont. ) Soft. Mealy wrapper representation n Hsu, IJCAI-99 STALKER n n Muslea, Minton, Knoblock, AA-99 A hierarchical FST IEPAD n Chang, WWW 01
WIEN n n n HLRT (Head-Left-Right-Tail) Labeling: by Page. Oracle, Lable. Oracle. PAC analysis Extract 48% web pages successfully. Weakness: w Missing attributes, attributes not in order, tabular data. . etc.
Softmealy Chun-Nan Hsu, 1998 Arizona State University
Softmealy n Finite-State Transducers for Semi. Structured Text Mining Labeling: use a interface to label example by manually. n FST (Finite-State Transducer) n n n Sigle-pass Multi-pass
Soft. Mealy wrapper representation n n Uses finite-state transducer where each distinct attribute permutations can be encoded as a successful path Replaces delimiters with contextual rules that describes the context delimiting two adjacent attributes
Example
Output 4種情形
Finite State Transducer skip extract b U -U N 多解決了 (N, M)、 (N, A, M) 2個情形 skip -N skip e extract M skip -A extract A
Find the starting position -- Single Pass 新增的定義
Taxonomy Tree
Stalker Muslea, Minton, Knoblock, AA-99 A Hierarchical FST
STALKER n “STALKER: Learning Extraction Rules for Semi-structured, Web-based Information Sources”. AAAI-98, Muslea. Embeded Catalog Description is a treelike structure.
EC Tree of a page
Multi-Pass or Hierarchical Wrapper Pass 1: extract U 先extract Body Pass 2: extract N Pass 3: extract A 再extract Tuples Pass 4: extract M
Rule Generating Extract Credit info. 1 st : terminals: {; reservation _Symbol_ _Word_} Candidate: {; <i> _Symbol_ _Html. Tag_} perfect Disj: {<i> _Html. Tag_} positive example: D 3, D 4 2 nd: uncover{D 1, D 2} Candicate: {; _Symbol_}
Possible Rules
Features Process is performed in a hierarchical manner. 沒有Attributes not in order的問題。 Use disjunctive rule 可以解決Missing attributes的問題。
Comparison Both : n n can handle irregular missing attributes. 對於未見過的attribute，需要training Single-pass : n n n 允許的attribute permutations 有限 Single-pass is good for tabular pages 比較快 Multi-pass: n n n Attribute permutations沒有影響 Multi-pass is good for tagged-list pages 比較慢
Comparison Quote Server n n n Stalker: 10 example tuples, 79%, 500 test WIEN: the collection beyond learn’s capablity Soft. Mealy: multi-pass 85%, single-pass 97% Internet Address Finder n n n Stalker: 80% ~ 100%, 500 test WIEN: the collection beyond learn’s capablity Soft. Mealy: multi-pass 68%, single-pass 41%,
Comparison Okra(tabular pages) n n n Stalker: 97%, 1 example tuple WIEN: 100% , 13 example tuples, 30 test Soft. Mealy: single-pass 100%, 1 example tuple, 30 test Big-book(tagged-list pages) n n n Stalker: 97%, 8 example tuples WIEN: perfect, 18 example tuples, 30 test Soft. Mealy: single-pass 97%, 4 examples, 30 test multi-pass 100%, 6 examples, 30 test
- Slides: 34