Design a fulltext search engine for a website

Design a full-text search engine for a website based on Lucene Presented by: Lijia Li, Yingyu Wu, Xiao Zhu

Outline • • • Introduction Our goal System architecture Conclusion and future work Show demo

Introduction • With the development of the network, the amount of information on the Internet showed explosive growth, increased the difficulty of finding the target information, the search engine has brought great convenience to people looking for information, internet has become an indispensable tool.

Our goal • In this project, our goal is to implement a fulltext retrieval engine based on Lucene.

Full-text retrieval engine • The full-text search engine based on the entire text retrieval technology for indexing and searching. • Features: (1) The unstructured index file database (2) Flexible retrieval methods (3) Support nature language retrieval (4) Retrieval efficiency

System Architecture • Search Engine is used to provide searching service to users. Our search engine has two main parts: online and offline.

Users Enter keyword Online User Interface analyzer Result sorting Search module Search Index File Index module website Request webpage Website database crawler offline

Lucene Why • The index file format independent of the application platform • Inverted index • Object-oriented system architecture • Chinese parser (Smartchinese. Analyzer, IKAnalyzer) • Implement a set of powerful Query engine(Range. Query, Fuzzy. Query……) • Open Source

Web Crawler Analysis robots. txt Collection of start URL Analysis URL Get robots. txt Unprocessed URL queue Extract Links Architecture of web crawler Page database Page fetch module Page analysis module Internet

Work flow of web crawler 1. Extract the initial URL into unprocessed URL queue 2. Get a URL address from the head of the queue 3. Download pages according to their URL 4. Extract hyperlink from the download page 5. Extracted hyperlinks added to unprocessed URL queue 6. Check whether the unprocessed URL queue is null if yes the program will be terminated otherwise step 2 will be executed. 7. Loop

Index Aset of documents to be index Read and Analysis document no yes Date of index ealier than the creation data yes Call the corresponding document parser to parse document Whether Indexed? yes Whether exist same type no Determine the type of document no Parse document Build index file Work flow

Document indexing steps 1. Creating a Index. Writer instance Index. Writer writer = new Index. Writer(index. Path, analyzer, boolean, max. Field. Length) 2. Creating a recode of Document doc = new Document() 3. Add Field Object in recode of Document doc. add(new Filed(string, tokenstream)) 4. Write recode of Document in Index writer. add. Document(doc); 5. Close Index Writer Object, end indexing writer. close()

Flow chart of searching start Accept search string from user Query. Parser analyze search string, output Query object Set up Searcher Index. Searcher object search related document in Index File Output related document end Example： User input: “ 大连理 计算机”, “america ohio” After Query. Parser: “大连理 ” AND“计算机 ”, “america” AND “ohio”

Highlight search key word 1. Get position value of search key word 2. Get fragment of search key word, according position value of search key word 3. Use HTML and CSS attributes to highlight search key word

Conclusion and future work • What we learn through this project is how to use web crawler and Lucene to implement a full-text search engine. • Working on hadoop • Thank you!