Filipiniana Web Nestor Michael C Tiglao Computer Networks

Filipiniana. Web Nestor Michael C. Tiglao Computer Networks Lab (CNL) University of the Philippines 17 th APAN Meetings & Joint Techs Workshop Jan. 30, 2004

World Wide Web Enormous growth (10 billion pages) n Imagine the Web without search engines n Need for intelligent document discovery mechanisms n

Web Crawlers n Programs that retrieve Web pages Two kinds: General-purpose crawlers n Focused crawlers n

Sample Query: anthrax

Result 1

Result 2

Focused Crawler n Selectively seek out pages that are relevant to a pre-defined set of topics n Topics are specified by sample documents

Research on Search Engines Implemented the focused crawler on a Linux cluster using Beowulf and MPI (2002) n Philippine-specific search engine using the open. Mosix platform (2003) n

Focused Crawler Architecture User Interface Results Distiller Sample Document Crawl Tables Classifier Crawler

Focused Crawler Design

Flowchart

Performance (Crawl Time)

Why another search engine? n Existing Philippine search engines: Yehey. com, Alleba, Tanikalang Ginto, Pugad. com and Edsa. World n n actually web directories We need a better search engine

Unique Situation Many Philippine-related sites are not registered under the. ph domains n Many sites are hosted outside the Philippines n English as the de facto language n

System Design (Gagambot)

Filters n ph Domain filter n n gov. ph, edu. ph Language filter iso 639, iso-8859 -1/latin 1 and windows-1252 n subset of Unicode characters utf-8 and us-ascii n

Filters 2 n Geo. URL filter Location-to-URL reverse directory n Finds URLs by their proximity to a given location (www. geourl. org) n n Bayesian filter n Analyzes the textual content of the HTML document

Filipiniana. Web

Current Plans Develop Filipiniana. Web on a grid platform n Better filtering techniques n Integrate focused crawling n Support for other object formats: documents, images, XML, etc. n

Conclusion n Filipiniana. Web is a work-in-progress and a proof-of-application n Grid infrastructure will help provide the computational and resource requirements of a production-level search engine