CSCI455522 Introduction to High Performance Computing Lecture 2
- Slides: 48
CSCI-455/522 Introduction to High Performance Computing Lecture 2
Types of Parallel Computers Two principal types: • Shared memory multiprocessor • Distributed memory multicomputer Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 2
Shared vs. Distributed Memory P P BUS Memory P P Shared memory - single address space. All processors have access to a pool of shared memory. (Ex: SGI Origin, Sun E 10000) Distributed memory - each processor has it’s own local memory. Must do message passing to exchange data between processors. (Ex: CRAY T 3 E, IBM SP, clusters) P P P M M M Network
Shared Memory Multiprocessor Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 4
Conventional Computer Consists of a processor executing a program stored in a (main) memory: Main memory Instructions (to processor) Data (to or from processor) Processor Each main memory location located by its address. Addresses start at 0 and extend to 2 b - 1 when there are b bits (binary digits) in address. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 5
Shared Memory Multiprocessor System Natural way to extend single processor model - have multiple processors connected to multiple memory modules, such that each processor can access any memory module : One address space Memory module Interconnection network Processors Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 6
Simplistic View of a Small Shared Memory Multiprocessor Processors Shared memory Bus Examples: • Dual Pentiums • Quad Pentiums Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 7
Quad Pentium Shared Memory Multiprocessor Processor L 1 cache L 2 Cache Bus interface Processor/ memory bus I/O interface Memory controller I/O bus Shared memory Memory Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 8
Programming Shared Memory Multiprocessors • Threads - programmer decomposes program into individual parallel sequences, (threads), each being able to access variables declared outside threads. Example Pthreads • Sequential programming language with preprocessor compiler directives to declare shared variables and specify parallelism. Example Open. MP - industry standard - needs Open. MP compiler Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 9
• Sequential programming language with added syntax to declare shared variables and specify parallelism. Example UPC (Unified Parallel C) - needs a UPC compiler. • Parallel programming language with syntax to express parallelism - compiler creates executable code for each processor (not now common) • Sequential programming language and ask parallelizing compiler to convert it into parallel executable code. - also not now common Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 10
Shared Memory: UMA vs. NUMA P P P BUS Memory Non-uniform memory access (NUMA): Time for memory access depends on location of data. Local access is faster than non-local access. Easier to scale than SMPs (SGI Origin) Uniform memory access (UMA): Each processor has uniform access to memory. Also known as symmetric multiprocessors, or SMPs (Sun E 10000) P P P P BUS Memory Network P
Distributed Memory /Message. Passing Multicomputers Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 12
Message-Passing Multicomputer Complete computers connected through an interconnection network: Interconnection network Messages Processor Local memory Computers Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 13
Distributed Memory: MPPs vs. Clusters • Processor-memory nodes are connected by some type of interconnect network – Massively Parallel Processor (MPP): tightly integrated, single system image. – Cluster: individual computers connected by s/w CPU CPU CPU MEM MEM MEM CPU CPU MEM MEM Interconnect Network
Distributed Memory: MPPs vs. Clusters • Processor-memory nodes are connected by some type of interconnect network – Massively Parallel Processor (MPP): tightly integrated, single system image. – Cluster: individual computers connected by s/w CPU CPU CPU MEM MEM MEM CPU CPU MEM MEM Interconnect Network
Clusters • Similar to MPPs – Commodity processors and memory • Processor performance must be maximized – Memory hierarchy includes remote memory – No shared memory--message passing • Communication overhead must be minimized • Different from MPPs – All commodity, including interconnect and OS – Multiple independent systems: more robust – Separate I/O systems
Interconnection Networks • • Limited and exhaustive interconnections 2 - and 3 -dimensional meshes Hypercube (not now common) Using Switches: – Crossbar – Trees – Multistage interconnection networks Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 17
Communications Networks • Custom – Many vendors have custom interconnects that provide high performance for their system, specially MPP – CRAY T 3 E interconnect is the fastest for MPPs: lowest latency, highest bandwidth • Commodity – Used in some MPPs and all clusters – Myrinet, Gigabit Ethernet, Fast Ethernet, etc. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Types of Interconnects • Fully connected – not feasible • Array and torus – Intel Paragon (2 D array), CRAY T 3 E (3 D torus) • Crossbar – IBM SP (8 nodes) • Hypercube – SGI Origin 2000 (hypercube), Meiko CS-2 (fat tree) • Combinations of some of the above – IBM SP (crossbar & fully connected for 80 nodes) – IBM SP (fat tree for > 80 nodes) Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Two-dimensional Array (Mesh) Links Computer/ processor Also three-dimensional - used in some large high performance systems. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 22
Three-dimensional Hypercube Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 23
Four-dimensional Hypercubes popular in 1980’s - not now Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 24
Crossbar Switch Memories Processors Switches Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 25
Tree Root Links Switch element Processors Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 26
Multistage Interconnection Network Example: Omega network 2 ´ 2 switch elements (straight-through or crossover connections) 000 001 010 011 Inputs 000 001 010 011 Outputs 100 101 110 111 Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 27
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved.
Distributed Shared Memory Making main memory of group of interconnected computers look as though a single memory with single address space. Then can use shared memory programming techniques. Interconnection network Messages Processor Shared memory Computers Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 35
Flynn’s Classifications Flynn (1966) created a classification for computers based upon instruction streams and data streams: – Single instruction stream-single data stream (SISD) computer Single processor computer - single stream of instructions generated from program. Instructions operate upon a single stream of data items. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 36
Single Instruction Stream-Multiple Data Stream (SIMD) Computer • A specially designed computer - a single instruction stream from a single program, but multiple data streams exist. Instructions from program broadcast to more than one processor. Each processor executes same instruction in synchronism, but using different data. • Developed because a number of important applications that mostly operate upon arrays of data. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 37
Multiple Instruction Stream-Multiple Data Stream (MIMD) Computer General-purpose multiprocessor system - each processor has a separate program and one instruction stream is generated from each program for each processor. Each instruction operates upon different data. Both the shared memory and the messagepassing multiprocessors so far described are in the MIMD classification. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 38
The Banking Analogy • Tellers: Parallel Processors • Customers: tasks • Transactions: operations • Accounts: data
Vector/Array • Each teller/processor gets a very fine-grained task • Use pipeline parallelism • Good for handling batches when operations can be broken down into finegrained stages
SIMD (Single-Instruction-Multiple. Data) • All processors do the same things or idle • Phase 1: data partitioning and distributed • Phase 2: data-parallel processing • Efficient for big, regular datasets
Systolic Array • Combination of SIMD and Pipeline parallelism • 2 -d array of processors with memory at the boundary • Tighter coordination between processors • Achieve very high speeds by circulating data among processors before returning to memory
MIMD(Multi-Instruction-Multiple-Data) • Each processor (teller) operates independently • Need synchronization mechanism – by message passing – or mutual exclusion (locks) • Best suited for large-grained problems • Less than data-flow parallelism
Networked Computers as a Computing Platform • A network of computers became a very attractive alternative to expensive supercomputers and parallel computer systems for high-performance computing in early 1990’s. • Several early projects. Notable: – Berkeley NOW (network of workstations) project. – NASA Beowulf project. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 44
Key advantages: • Very high performance workstations and PCs readily available at low cost. • The latest processors can easily be incorporated into the system as they become available. • Existing software can be used or modified. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 45
Software Tools for Clusters • Based upon Message Passing Parallel Programming: • Parallel Virtual Machine (PVM) - developed in late 1980’s. Became very popular. • Message-Passing Interface (MPI) - standard defined in 1990 s. • Both provide a set of user-level libraries for message passing. Use with regular programming languages (C, C++, . . . ). Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 46
Beowulf Clusters* • A group of interconnected “commodity” computers achieving high performance with low cost. • Typically using commodity interconnects high speed Ethernet, and Linux OS. * Beowulf comes from name given by NASA Goddard Space Flight Center cluster project. Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 47
Cluster Interconnects • Originally fast Ethernet on low cost clusters • Gigabit Ethernet - easy upgrade path More Specialized/Higher Performance • • • Myrinet - 2. 4 Gbits/sec - disadvantage: single vendor c. Lan SCI (Scalable Coherent Interface) QNet Infiniband - may be important as infininband interfaces may be integrated on next generation PCs Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 48
Dedicated cluster with a master node Dedicated Cluster User Compute nodes Master node Up link Switch 2 nd Ethernet interface External network Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2 nd Edition, by B. Wilkinson & M. Allen, © 2004 Pearson Education Inc. All rights reserved. 1. 49
