Parallel and Distributed IR Eric Brown Parallel Computing

Parallel and Distributed IR Eric Brown

Parallel Computing w w SISD：single instruction stream, single data stream. SIMD：single instruction stream, multiple data stream. MISD：multiple instruction stream, single data stream. MIMD：multiple instruction stream, multiple data stream.

Performance Measures S= Running time of best available sequential algorithm -------------------------------Running time of parallel algorithm 1 1 S<= <= f +(1 -f)/N f S = N

Parallel IR w Introduction： n n Develop new retrieval strategies that directly lend themselves to parallel implementation. Adapt existing, well studied information retrieval algorithms to parallel processing.

MIMD Architecture

MIMD Architecture w Inverted Files n Logical Document Partitioning l n Essentially the same basic underlying inverted file index as in the original sequential algorithm. Physical Document Partitioning l Each subcollection has its own inverted file and the search processes shard nothing during query evaluation.

MIMD Architecture w Logical document partitioning requires less communication than physical document partitioning with similar parallelization, and so is likely to provide better overall performance. w Physical document partitioning, on the other hand, offers more flexibility and conversion of an existing IR system into a parallel IR system is simpler using physical document partition.

MIMD Architectures w Term partitioning n When term partitioning is used with an inverted file is created for the document collection and the inverted lists are spread across the processors. w Assuming each processor has its own I/O channel and disks when term distribution in the documents and the queries are more skewed, document partition performs better. When terms are uniformly distributed in user queries, term partition performs better.

MIMD Architecture

SIMD Architecture w Signature Files

SIMD Architecture w Signature Files

SIMD Architecture w Signature Files

SIMD Architectures w Inverted Files

SIMD Architectures

SIMD Architectures w Inverted Files

SIMD Architectures

Distributed IR w Introduction n A distributed computing system can be viewed as a MIMD parallel processor with relatively slow inter-processor communication channel and the freedom to employ a heterogeneous collection of processors in the system.

Distributed IR w Introduction n n Distributed Model is very similar to the MIMD parallel processing model. The main difference here is that subtasks run on different computers and the communication between the subtasks is performed using network protocol such as TCP/IP.

Collection Partitioning w The procedure used to adding documents to search servers in a distributed IR system depends a number of factors. n Consider whether or not the system is centrally administered.

Collection Partitioning n When the distribute system is centrally administered, more options are available. The first option is simple replication of the collection across all of the search servers. l The second option is random distribution of the documents. l The final option is explicit semantic partitioning of the documents. l

Source Selection w Source selection is the process of determining which of the distributed document collections are most likely to contain relevant documents for the current query, and therefore should receive the query for processing. w The basic technique is to treat each collection as if it were a single large document, index the collections, and evaluate the query against the collections to produce a ranked listing of collections.

Query Processing Query processing in a distributed IR system proceeds as follows： Select collection to search. Distribute query to selected collections. Evaluate query at distributed collection in parallel. Combine results from distributed collection into final result.

Web Issues w The parallel and distributed techniques described above can then be used directly as if the Web were any other large document collection. This is the approach currently taken by most of the popular Web search services.

Trends and Research Issues w The trend in parallel hardware is the develop of general MIMD machines. w Many challenges remain in the area of parallel and distributed text retrieval. n n The first challenge is measuring retrieval effectiveness on large text collections. The second significant challenge is interoperability, or building distributed IR systems form heterogeneous components.