ODISSEA open distributed search engine architecture A PeertoPeer

ODISSEA open distributed search engine architecture A Peer-to-Peer Architecture for Scalable Web Search and Information Retrieval Torsten Suel, Chandan Mathur, Jo-Wen Wu, Jiangong Zhang, Alex Delis, Mehdi Kharrazi, Xiaohui Long, Kulesh Shanmugasundaram Daniel Porta (d. porta@web. de) Seminar "Peer-to-peer Information Systems"

Talk Outline l l l l Motivation Design Overview System Design Details Target Applications Implementation Details Efficient Query Processing Open Questions Seminar "Peer-to-peer Information Systems" 2

Motivation l l l Today, main part of the web search infrastructure is supplied by only a few large crawl-based search engines Strong research in the field of P 2 P systems over the last few years Computers have/will become faster and the network bandwidth has increased/will grow This raises two issues l Vast data in P 2 P networks requires the ability to search in these networks l Significant computing resources provided by a P 2 P system could be used to search content residing inside or outside the system ODISSEA attempts both issues by a „distributed global indexing and query execution service“ Seminar "Peer-to-peer Information Systems" 3

Design Overview l l l ODISSEA is different from many other approaches to P 2 P search It assumes a two-layered search engine architecture and a global index structure distributed over the nodes of the system In a global index, as contradiction to a local index, a single node holds the entire inverted index for a particular term Seminar "Peer-to-peer Information Systems" 4

Two Layer Approach l Lower layer provides l maintanance of the global index structure under document insertions and updates l Maintanance of node joins and failures l Efficient execution of simple search queries WWW l ODISSEA Search server Upper layer interacts with P 2 P-based lower layer via two classes of clients l Update clients (e. g crawler, web server) l Query clients (user implemented optimized query execution plan) Seminar "Peer-to-peer Information Systems" queries crawler 5

Two Layer Approach l l Enables a large variety of (client-based) search tools that more fully exploit client computing resources. Those tools could share the same lower-layer web search infrastructure. Tools are developed using an open API, which accesses the search infrastructure When processing a query, this could in the most general case (i. e where no pre-evaluation is done on server-side) result in large amounts of data to be transferred to the query client Seminar "Peer-to-peer Information Systems" 6

Global vs. Local index l A B l C l l search client posting = [Doc. ID, Position, additional information] A: chair Inverted list is a list of postings B: table that represents all occurencies of a term in the document collection C Inverted index for terms is the set of the corresponding inverted lists Suppose a query „chair AND table“. Then the query will be processed as follows search client Seminar "Peer-to-peer Information Systems" 7

Global vs. Local index l l Local index organization is very inefficient in very large networks (e. g. web) if result quality is the major concern, because the query has to be transmitted to all nodes and all of them have to respond But in a global index organization large amounts of data need to be transmitted between nodes when l Initially building the index l Evaluating a query bad response time Can be overcome with smart algorithmic techniques, as you will see later Choice depends on the types of queries and the frequency of document updates, as well as on the question of how dynamic the system is Seminar "Peer-to-peer Information Systems" 8

Crawling and Fault Tolerance Crawling approach l Client-based, non P 2 P crawlers have the advantage that they can be easily altered in the case that some web site operators have complains about the bot l Smart crawling strategies beyond BFS are hard to implement in a P 2 P environment unless there is a centralized scheduler P 2 P systems and fault tolerance l System design relies on the assumption of a more stable P 2 P environment, since otherwise administration (insert, update, replication) would be too expencive Seminar "Peer-to-peer Information Systems" 9

Target Applications l l Full-text search in large document collections located within in P 2 P communities Search in large intranet environments Web Search: a powerful API supports the anticipated shift towards client-based search tools which better exploit the resources of todays desktop machines Search middleware: Instead of inserting documents, clients could directly insert index entries. This might speed up query execution, since for a document only certain „strong“ keywords can be inserted. But a drawback could be that the identification of such keywords lies in client‘s hand Seminar "Peer-to-peer Information Systems" 10

Implementation Details l l l Currently, a first system is being implemented in Java, using Pastry as a P 2 P substrate (lower layer) and a DHT mapping for hashing IDs to the appropriate IP-address Each node runs an indexer that stores inverted list in compressed form in a Berkeley DB (which contains a B+ tree), each document is also stored in a Berkeley DB Using MD 5, all documents and term lists are hashed to a 80 -bit ID that is used for lookups in the system Seminar "Peer-to-peer Information Systems" 11

Implementation Details Parsing and Routing Postings l New or updated documents are parsed at the node where they reside, as determined by the DHT mapping l Parser generates for each term a posting that is routed via several intermediate nodes, as determined by the topology of the Pastry network, until it reaches its destination node l An index structure of a node is split up in a small structure (residing in main memory) that is eventually merged with a bigger structure on disk to avoid disk accesses during inserts/updates lower amortized complexity Seminar "Peer-to-peer Information Systems" 12

Implementation Details Groups and Splits l Initially, all objects (documents, indexes) whose first w bits (here w=16) coincide are placed into a common group identified by this w-bit string l Locally, each group maintains a Berkeley DB with all objects it contains l When a group (of documents) becomes too large (here >1 GB), it is split into two groups identified by a (w+1)-bit string leaving a stub structure pointing to the new groups that are assigned to new nodes l If index structures for terms are too large (here >100 MB), they are split into two lists according to the document IDs they contain Seminar "Peer-to-peer Information Systems" 13

Implementation Details Replication l Performed at group level by attaching „/0“, „/1“, etc. to the group label (e. g. 0100101/2) l This new label is then what is really presented to Pastry/DHT during lookups l All replicas of a group form a „clique“ that communicate periodically to update their status l If a group replica fails, the others are in charge of detecting this and if necessary perform repair l Each node can contain several distinct group replicas and therefore participate in several cliques l Postings are first routed to only one replica that is then in charge of forwarding them to the others over a period of a few minutes Seminar "Peer-to-peer Information Systems" 14

Implementation Details Faults, Unavailability and Synchronization l When a node leaves the system, its group replicas eventually have to be replaced to maintain the desired degree of replication l A node has failed if it has been unavailable for an extended period of time l Create new replicas for a failed node or if a certain number of nodes are unavailable l Former unavailable nodes have to synchronize its index structures using logs of missing updates Seminar "Peer-to-peer Information Systems" 15

Efficient Query Processing Information Thoeretic Background l Let d be a document, q = q 0…qm-1 a query consisting of m terms and F be a function that assigns d (depending on q) a value F(d, q). Such a function is called a ranking function. l The top-k ranking problem for a query q is finding the k documents with the highest values F(d, q). l A common form of such a function looks like this l Since queries typically have at most only 2 search terms, the following algorithm focuses on the top-k ranking problem and queries with exactly 2 search terms (for one-term queries, there is in fact nothing to do) Seminar "Peer-to-peer Information Systems" 16

Efficient Query Processing Fagin‘s Algorithm (FA) l Intuitively, an item that is ranked in the top is likely to be ranked very high in at least one of the contributing subcategories l Assume a query q = q 0 AND q 1 and postings of the form (d, f(d, qi)) that are sorted by the second component with highest values on top l Also assume that the inverted lists for q 0 and q 1 are located on the same machine, so that no network communication is required l Goal: compute the top k documents as fast as possible Seminar "Peer-to-peer Information Systems" 17

Efficient Query Processing 1. Scan both lists from the beginning, by reading one element from each list in every step, until there are k documents that have been encountered in both lists (here assume k=2) 1 2 3 4 5 A 0. 9 0. 8 0. 7 0. 69 0. 67 B 0. 6 0. 5 0. 4 0. 3 0. 2 6 5 3 1 7 2. Compute the scores of these k documents. Also, for each document that was encoutered in only one of the lists, perform a 0. 1 lookup into the other list to determine the 8 score of the document. 3. Return the k documents with the highest score (here d 1, d 5) Seminar "Peer-to-peer Information Systems" 18

Efficient Query Processing Threshold Algorithm (TA) l Scan both lists simultaneously and read (d, f(d, q 0)) from the first and (d‘, f(d‘, q 1)) from the second list l Compute t = f(d, q 0) + f(d‘, q 1) l For each d in one of the lists perform immediately a lookup in the other list in order to compute its complete score l Algorithm terminates, when k documents have been found that have higher scores than the current value of t Because it does not make sense to scan two lists simultaneously while they are distributed in a P 2 P network, the above techniques have to be adapted. This leads us to the following protocol that aims at minimize the data to be transferred. Seminar "Peer-to-peer Information Systems" 19

Efficient Query Processing A simple distributed pruning protocol (DPP) Node A (holding the shorter list) sends the first x postings to node B. Let rmin be the smallest value f(d, q 0) transmitted Node A now performs lookups into its own list for the postings A B received from B and determines the Node B now transmitts to A all postings overall top k among its first x postings with documents f(d, q 1) > rk - rmin, together with the total scores of the k documents from the previous step Seminar "Peer-to-peer Information Systems" Node B receives the postings from A and performs a lookup into its own list in order to compute the total scores. Retain the k documents with the highest scores. Let rk be the smallest value among these. 20

Efficient Query Processing DPP-Example for k=2 and x=3: A containing term q 0: (d 1, 0. 9), (d 2, 0. 8), (d 3, 0. 7), (d 4, 0. 69), (d 5, 0. 67) B containing term q 1: (d 6, 0. 6), (d 5, 0. 5), (d 3, 0. 4), (d 1, 0. 3), (d 7, 0. 2), (d 8, 0. 1) A computes: (d 6, 0. 6+ ---- ), (d 5, 0. 5+0. 67), A to B: rmin = 0. 7 (d 1, 0. 9), (d 2, 0. 8), (d 3, 0. 7) A B (d 1, 1. 2) 0. 9 + 0. 3) (d 32, 1. 1) 0. 8 + ----) (d 3, 0. 7 + 0. 4) rk = 1. 1 Top 2 documents: 1. (d 1, 1. 2) 2. (d 5, 1. 17) B retains: computes: rk – rmin = 1. 1 - 0. 7 = 0. 4 B to A: (d 6, 0. 6), (d 5, 0. 5), because f(d 6, 5, q 1) > 0. 4 together with (d 1, 1. 2), (d 3, 1. 1) Seminar "Peer-to-peer Information Systems" 21

Efficient Query Processing Problems with the DPP l works only with queries containing 2 search terms l random lookups can cause disk accesses, since large index structures reside on hard disk bad response time l How must the value of x be chosen? (x should be the number of postings transmitted from A and B, s. t. DPP works correct without extra roundtrip; depends on the k and length of the inverted lists) l By deriving appropriate formulae based on extensive testing l By sampling-based methods that estimate the number of documents appearing in both lists Seminar "Peer-to-peer Information Systems" 22

Efficient Query Processing Evaluation of DPP l 900 two-term queries selected form a set of over 1 million l Testing corpora: 120 million web pages (1. 8 TB) that were crawled by their own crawler l Value of x determined by experiments on TA l Computation within nodes are not taken into account l Commmunication costs and estimated times of DPP for the top-10 documents and standard cosine measure: Seminar "Peer-to-peer Information Systems" 23

Future Work l l Framework for generating optimized query execution plans for multi-keyword queries New algorithmic techniques for the index synchronization problem New strategies for load balancing and rebuilding of lost replicas More experimental evaluation concerning different types of queries Seminar "Peer-to-peer Information Systems" 24

Questions? „The general question remains whether the near future will see massive P 2 P-based systems for challenging applications such as web search and large-scale IR, beyond the current simple applications such as file sharing. “ Seminar "Peer-to-peer Information Systems" 25