Web Document Analysis How can Natural Language Processing

Web Document Analysis: How can Natural Language Processing Help in Determining Correct Content Flow? Hassan Alam, Fuad Rahman and Yuliya Tarnikova Human Computer Interaction Group BCL Technologies Inc. Santa Clara, CA 95050 www. bcltechnologies. com fuad@bcltechnologies. com

Overview of the talk l l l l Web document re-authoring HTML data structure and segmentation Merging and the “mess” Semantic Relatedness of Textual Segments Spoken Language User Interface Toolkit (SLUITK) How do we do it? Some applications Conclusion and future work

Web Page Data Structure

Merging R Us l While merging two segments, the only information available to the merging algorithm is the proximity map and broad content classification. l It is not uncommon that sometimes totally unrelated content can easily meet these tests, resulting in the failure of the merging algorithm.

e. Merging Questions? How do we determine if two separate web document segments contain related information? l What is the definition of 'relatedness'? l If other segments are geometrically embedded within closely related segments, can we determine if this segment is also related to the surrounding segments? l When a hyperlink is followed and a new page is accessed, how do we know which exact segment within that new page is directly related to the link we just followed? l

Natural Language Processing l Syntax l Semantics l Context l Anaphora l Tokenizing l Theme

Our Answer l Lexical Chains

Lexical Chains A lexical chain is a sequence of related words in a narrative. It can be composed of adjacent words or sentences or can cover elements from the complete narrative. l Cohesion is a way of connecting different parts of text into a single theme: is a list of semantically related words, constructed by the use of coreference, ellipses and conjunctions. l This aims to identify the relationship between words that tend to co-occur in the same lexical context. l

Lexical Chains l Coreference: The grammatical relation between two words that have a common referent – Example: You said you would come In the given sentence, both ‘you’ s have the same referent. l Ellipsis: Omission or suppression of parts of words or sentences – Example: 'the virtues I admire', for, 'the virtues 'which' I admire' l Conjecture: Reasoning that involves the formation of conclusions from incomplete evidence – Example: Scientists supposed that large dinosaurs lived in swamps

What is SLUI TK? SLUI is a set of tools that allows programmers to rapidly develop applications with Natural Language Processing Functionality

SLUI TK Steps for the Programmer to Follow while Setting up the Toolkit

SLUI TK An innovative way to assist programmers with no linguistic knowledge in developing programs that can understand, process, and act upon spoken Natural Language (NL) input

Our Frame Can you suggest some internet sites or books that give details on lowering the LDL by 50 points without including information on cancer risks?

BCL Database l l l Sentences collected from email messages received between June 2000 and May 2001 Deleted attachments, html and other tags, header files, and senders’ information. Also deleted were salutations and greetings Total of 34, 640 lines and 170, 000 words We constantly update our corpus with new emails from our customers.

Our Lexical Chains

Relatedness Factor

An Application: Web Page Reauthoring

Segment Scores

Example Output

Future Work Only a single main theme can be handled per document. In future we are going to address a more generic solution that can handle documents with multiple themes. l Integration of this NLP method in building commercial summarizers and in aiding existing web page summarization techniques based on structural analysis alone is already well underway. l Determining the flow of web information between different web pages as the browser loads up new pages following hyperlinks. l Aiding geometric web parsers in determining the correct logical layout by complementing geometric information with linguistic coherence. l

Conclusions A novel approach of determining semantic relationship among segments of web documents using lexical chain computation. l Two related papers in ICDAR 2003 l – One will explore the application of lexical chains in building a commercial summarizer capable of summarizing any document – The other will concentrate on a hybrid approach to web page summarization, combining structural and NLP techniques.