Challenges in Web Document Summarization Some Myths and

Challenges in Web Document Summarization: Some Myths and Reality A. Rahman H. Alam Document Analysis and Recognition Team (DART) BCL Computers Inc. Santa Clara, Calif, USA

Basic Problem Statement • What are web based documents? • What is summarization? • Textual summarization vs. content summarization • What myths do we have about summarization? • What is the reality?

Why Summarization? • Display area of handheld devices i. e. PDAs and Cell phones is too small for useful web browsing • Download times is still too slow for comfortable browsing using wireless devices • Cost factor is still too high

Where is the Money? • 1. 2 billion web pages • 2 hours/site to adapt an existing page for wireless, it will take 2. 4 billion work-hours • At $20 per hour is assumed, this effort requires an investment of around $50 billion

Current need? • Viewing website using small screen handheld devices • Since web sites are written using HTML codes, we need to translate these to systems that the wireless devices can support.

Myths • Web summarization is easy • • • No scanning No image processing No Word or character level recognition HTML has structural elements Already in electronic formats

Current Solutions • Handcrafting: – Custom Web Sites are typically crafted by hand by a set of content experts • Transcoding: – Thranscoding replaces HTML tags with suitable device specific tags (HDML, WML etc)

Handcrafting • • • Take an existing website and make it available to wireless access. Aether Systems, Mshift and 2 Roam currently offer these types of solutions. Use a proprietary graphical interface to ease the development of wireless applications from scratch. Covigo and i. Converse offer these type of solutions. Let the user do all coding in languages such as C++ or Java. Thin. Air. Apps offers this type of solution.

Handcrafting • • • Labor intensive Expensive. Typically less than 1% of a web site gets converted to wireless content.

Transcoding • • Transcoding was introduced in Japan during 1999 -2000. It was widely rejected by the Japanese users. Recently, Google and Pixo introduced this solution for the US market, but have so far failed to attract attention of end users.

The Alternate Solution • • Separate the content into smaller segments Generate a summary of these segments Prioritize these summaries from individual segments Put together to form a summary of the overall document

Summarization vs. Transcoding • • Long displays Long download times Finding information difficult No mapping of the importance of content in the original document

Steps to Summarization • Segmentation Ccontent – A tree • Problems – – – – – Tables Frames Java Script Graphics Other Artifacts Over segmentation Under segmentation Poor coding Browsers are too good! CTable CRow CCol CTable Etc… etc…. . CCol CTable Etc…

Steps to Summarization • Visual cues • Size of font • Headlines • Boldness • Color • Links, • Flashing • Italic (I) • Emphasized • Underlines. • Labeling – – Main Story Links Navigation Bars Advertisement Bars – Other Stories – Forms – Images • Problems • Graphics • OCR • Java scripts • CSS

Steps to Summarization • Labeling => Segment Summary: Extraction of a low level summary of the segment • Priority: Estimating importance of these segments • Table of Content (TOC) => Document Summary: Putting together a summary of the document

Conclusion • Content can be used effectively to summarize web documents • Content summarization is more complex than textual summarization • HTML structure is a good starting point, but not enough to understand context • Summarization offers significant advantages over transcoding • Summarization also helps in faster browsing experience • There is a lot of money in this!