COMP 73307336 Advanced Parallel and Distributed Computing Speculative

COMP 7330/7336 Advanced Parallel and Distributed Computing Speculative Decomposition Characteristics of Tasks Dr. Xiao Qin Auburn University http: //www. eng. auburn. edu/~xqin@auburn. edu Slides are adopted from Drs. Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar 2

Recap: Speculative Decomposition Discrete Event Simulation • Data structure: a time-ordered event list. • Events are extracted precisely in time order, processed, and if required, resulting events are inserted back into the event list. • For example: get up, get ready, drive to work, eat lunch, work some more, drive back, eat dinner, and sleep. • Each of these events may be processed independently; however, in driving to work, you might meet with an unfortunate accident and not get to work at all. • Therefore, an optimistic scheduling of other events will have to be rolled back. 3

Speculative Decomposition: Example • The simulation of a network of nodes • To simulate the behavior of this network for various inputs and node delay parameters (note that networks may become unstable for certain values of service rates, queue sizes, etc. ). 4

Problems of Recursive Decompositions Q 1: In quicksort, recursive decomposition results in _____ tasks? Q 2: What limits concurrency? (1) Dependencies (2) Uneven task sizes 5

Hybrid Decompositions • A mix of data and recursive decompositions is more desirable. • Q 3: How to use a hybrid decomposition? 6

Characteristics of Tasks • Step 1: Decomposition • Step 2: Map tasks to processes • Task characteristics critically impact choice and performance of parallel algorithms. Q 4: What are relevant task characteristics? 1. Task generation. 2. Task sizes. 3. Size of data associated with tasks. 7

Static Task Generation • Concurrent tasks are identified a-priori. • Classic Examples: – – Matrix operations Graph algorithms Image processing applications Regularly structured problems • Q 5: What decomposition techniques can we apply? – – Data decomposition technique Speculative decomposition technique. Recursive decomposition technique. Exploratory decomposition technique. 8

Dynamic Task Generation • Tasks are generated as we perform computation. • Classic Example – Game playing: each 15 puzzle board is generated from the previous one. • Q 6: What decomposition techniques can we apply? – – Data decomposition technique Speculative decomposition technique. Recursive decomposition technique. Exploratory decomposition technique. 9

Task Sizes • Task sizes may be – Uniform (i. e. , all tasks are the same size) or – Non-uniform. Q 7: Multiplying a dense matrix with a vector. Are these tasks uniform or non-uniform? Can task sizes be determined a-priori or not? 10

Size of Data Associated with Tasks Q 8: Why data size is important? Data movement overhead. A small data size implies that an algorithm can easily communicate this task to other processes dynamically. 11

Size of Input and Output Data • Small/Large Input Data • Small/Large Output Data Q 9: The 15 -puzzle problem: large/small input and output? Small input and small output. 12

Size of Input and Output Data • Small/Large Input Data • Small/Large Output Data Q 10: Computing the minimum of a sequence: large/small input and output? Large input and small output. 13

Size of Input and Output Data • Small/Large Input Data • Small/Large Output Data Large input and large output. Q 11: The quicksort problem: large/small input and output? 14

Characteristics of Task Interactions Tasks may communicate with each other in various ways. The associated dichotomy is: • Static interactions: The tasks and their interactions are known a-priori. • Dynamic interactions: The timing or interacting tasks cannot be determined a-priori. Q 12: Which is hard to program? 15

Characteristics of Task Interactions • Static interactions vs. Dynamic interactions Q 13: Which types of task interactions are there in these applications? The 15 -puzzle problem: breadthfirst search Dense-matrix-vector multiplication 16 Static Dynamic

Characteristics of Task Interactions • Regular interactions: There is a definite pattern (in the graph sense) to the interactions. These patterns can be exploited for efficient implementation. • Irregular interactions: Interactions lack well-defined topologies. (e. g. , sparse matrix-vector multiplication) 17

Characteristics of Task Interactions Regular or Irregular interactions? A simple example of a regular static interaction pattern is in image dithering. The underlying communication pattern is a structured (2 -D mesh) one as shown here: 18

Characteristics of Task Interactions Regular or Irregular interactions? The multiplication of a sparse matrix with a vector is a good example of a static irregular interaction pattern. Here is an example of a sparse matrix and its associated interaction pattern. 19

Characteristics of Task Interactions read-only vs. read-write • Interactions may be read-only or read-write. • Read-only interactions: tasks just read data items associated with other tasks. • Read-write interactions: tasks read, as well as modify data items associated with other tasks. • Read-write interactions are harder to code. Q 14: Why? since they require additional synchronization primitives. 20

Database Query Processing read-only or read-write? Decomposing the given query into a number of tasks. Edges in this graph denote that the output of one task is needed to accomplish the next. 21

Characteristics of Task Interactions one-way vs. two-way • Interactions may be one-way or two-way. • A one-way interaction can be initiated and accomplished by one of the two interacting tasks. • A two-way interaction requires participation from both tasks involved in an interaction. • One way interactions are somewhat harder to code in message passing APIs. 22

Characteristics of Task Interactions one-way or two-way? The multiplication of a sparse matrix with a vector is a good example of a static irregular interaction pattern. Here is an example of a sparse matrix and its associated interaction pattern. 23

Summary • Speculative Decomposition • Hybrid Decompositions • Characteristics of Tasks – Task generation. – Task sizes. – Size of data associated with tasks. 24