Chapter 1 Motivation and Introduction Introduction What is

Chapter 1 Motivation and Introduction

Introduction �What is parallel computing? �What is a parallel computer? - multicomputers - centralized multiprocessors (symmetrical multiprocessor or SMP) �What is parallel programming? �Is parallel programming really necessary? �Why should I program using MPI and Open. MP?

Parallelism Exploitation �Algorithm Level �Program Statements Level �Instruction Level �Computer Architecture Level �Operating System level

Algorithm Level �Basic design principles �Developing heuristic parallel algorithms for specific application models

Program Statements Level �Subroutine �Loop (while-loop, do-while loop, for- loop) �Statement �External files �Function library

Task/Data Dependence Graphs �A directed graph in which each vertex represents a task to be completed. �An edge from vertex u to vertex v means that task u must be completed before task v begins. That is, task v is dependent on task u. �If there is no path from u to v, then the tasks are independent and may be performed concurrently.

Task Dependence Graph

Divide larger tasks into smaller subtasks

Parallelism in Data Dependence Graphs (a)A graph exhibiting data parallelism (b)A graph exhibiting functional parallelism (c)A purely sequential dependence graph

Data Parallelism

Functional Parallelism

Pipelining An automobile assembly line is an example of pipeline

A pipeline to compute partial sums. Each circle represents a process. The leftmost stage inputs a[0], outputs its value as p[0], and pass p[0] to the next stage. All other stages i input a[i], collect p[i-1] from their predecessors, add the two values, and output the sum as p[i].

Data Clustering

A sequential algorithm to find K centers that optimally categorize N documents

Dependence diagram for the document clustering algorithm.

Opportunities for data parallelism �Each document may be input in parallel �Each document vector may be generated in parallel �The original cluster centers may be generated in parallel �The closed cluster center to each document vector and that vector’s contribution to the overall performance function may be computed in parallel

Opportunities for functional parallelism �Document input and vector generation tasks and the center generation tasks

Four paths for the development of applications software for parallel computers �Extend an existing compiler �Extend a sequential programming language �Add a new parallel language layer �Create a parallel language - occam - Fortran 90, C*, High Performance Fortran