Supercomputerst Gordon Bell Bay Area Research Center Microsoft

Supercomputers(t) Gordon Bell Bay Area Research Center Microsoft Corp. http: //research. microsoft. com/users/gbell Photos courtesy of The Computer Museum History Center Please only copy with credit! http: //www. computerhistory. org 10/21/2021 Copyright G Bell & TCM History Center 1

Supercomputer Largest computer at a given time n Technical use for science and engineering calculations n Large government defense, weather, aero laboratories are first buyers n Price is no object n Market size is 3 -5 n 10/21/2021 Copyright G Bell & TCM History Center 2

Growth in Computational Resources Used for UK Weather Forecasting 10 T • 1 T • 1010/ 50 yrs = 1. 5850 100 G • 10 G • 205 1 G • 100 M • 195 10 M • 1 M • KDF 9 100 K • 1 K • 100 • Mercury Leo 10 • • 1950 10/21/2021 YMP • 2000 Copyright G Bell & TCM History Center 3

What a difference 25 years and spending >10 x more makes! Artist’s view of 40 Tflops ESRDC c 2002 LLNL 150 Mflops machine room c 1978 10/21/2021 Copyright G Bell & TCM History Center 4

Harvard Mark I aka IBM ASCC 10/21/2021 Copyright G Bell & TCM History Center 5

“ market for maybe five computers. ” I think there is a world Thomas Watson Senior, Chairman of IBM, 1943 10/21/2021 Copyright G Bell & TCM History Center 6

The scientific market is still about that size… 3 computers When scientific processing was 100% of the industry a good predictor n $3 Billion: 6 vendors, 7 architectures n DOE buys 3 very big ($100 -$200 M) machines every 3 -4 years n 10/21/2021 Copyright G Bell & TCM History Center 7

Supercomputer price (t) Time $M 1950 1 1960 3 1970 1980 1990 2000 10/21/2021 10 30 250 1, 000 structure mainframes instruction //sm mainframe SMP pipelining vectors; SCI MIMDs: m. C, SMP, DSM ASCI, COTS MPP Copyright G Bell & TCM History Center example many. . . IBM / CDC 7600 / Cray 1 “Crays”/MPP Grid, Legion 8

Supercomputing: speed at any price, using parallelism Intra processor Memory overlap & instruction lookahead Functional parallelism (2 -4) Pipelining (10) SIMD ala ILLIAC 2 d array of 64 pe vs vectors Wide instruction word (2 -4) MTA (10 -20) MIMDs… processor replication SMP (4 -64) Distributed Shared Memory SMPs 100 MIMD… computer replication Multicomputers aka MPP aka clusters (10 K) Grid: 100 K 10/21/2021 Copyright G Bell & TCM History Center 9

High performance architectures timeline 1950. 1960. 2000 1970. Vtubes Trans. MSI(mini) n. Micr Processor overlap, lookahead Cray era SMP 1980. Micro RISC 10/21/2021 “killer micros” 6600 7600 Cray 1 X Y C T Vector-----SMP--------> mainframes---> “multis”------> DSM > Clusters > 1990. Tandm VAX Copyright G Bell & TCM History Center KSR SGI---- IBM UNIX 10

High performance architectures timeline 1950. 1960. 2000 1970. Vtubes Trans. n. Micr MSI(mini) 1980. 1990. Micro RISC Sequential programming---->---------------<SIMD Vector--//-------Parallelization--- Parallel programming multicomputers ultracomputers 10 X in price 10/21/2021 Copyright G Bell & TCM History Center <-------<--MPP era-----10 x. MPP 11

Time line of hpcc contributions 10/21/2021 Copyright G Bell & TCM History Center 12

Time line of hpcc contributions 10/21/2021 Copyright G Bell & TCM History Center 13

Lehmer UC/Berkeley precomputer number sieves 10/21/2021 Copyright G Bell & TCM History Center 14

Eniac c 1946 10/21/2021 Copyright G Bell & TCM History Center 15

Manchester: the first computer. Baby, Mark I, and Atlas 10/21/2021 Copyright G Bell & TCM History Center 16

von Neumann computers : Rand Johniac 10/21/2021 Copyright G Bell & TCM History Center 17

Gene Amdahl’s Dissertation and first computer 10/21/2021 Copyright G Bell & TCM History Center 18

IBM 10/21/2021 Copyright G Bell & TCM History Center 19

IBM Stretch c 1961 & 360/91 c 1965 consoles! 10/21/2021 Copyright G Bell & TCM History Center 20

IBM Terabit Photodigital Store c 1967 10/21/2021 Copyright G Bell & TCM History Center 21

STC Terabytes of storage c 1999 10/21/2021 Copyright G Bell & TCM History Center 22

Amdahl aka Fujitsu version of the 360 c 1975 10/21/2021 Copyright G Bell & TCM History Center 23

IBM ASCI Red @ LLNL 10/21/2021 Copyright G Bell & TCM History Center 24

CDC, ETA, Cray Research, Cray Computer 10/21/2021 Copyright G Bell & TCM History Center 25

Cray 1925 -1996 10/21/2021 Copyright G Bell & TCM History Center 26

Circuits and Packaging, Plumbing (bits and atoms) & Parallelism… plus Programming and Problems Packaging, including heat removal n High level bit plumbing… getting the bits from I/O, into memory through a processor and back to memory and to I/O n Parallelism n Programming: O/S and compiler n Problems being solved 10/21/2021 Copyright G Bell & TCM History Center n 27

Seymour Cray Computers 1951: ERA 1103 control circuits n 1957: Sperry Rand NTDS; to CDC n 1959: Little Character to test transistor ckts n 1960: CDC 1604 (3600, 3800) & 160/160 A n 1964: CDC 6600 (6 xxx series) n 1969: CDC 7600 n 10/21/2021 Copyright G Bell & TCM History Center 28

Cray Research, Cray Computer Corp. and SRC Computer Corp. 1976: Cray 1. . . (1/M, 1/S, XMP, YMP, C 90, T 90) n 1985: Cray Computer Cray 2 from Cray Research; Ga. As: Cray 3 (1993), Cray 4 n 1999: SRC Company large scale, shared memory multiprocessor using x 86 microprocessors n 10/21/2021 Copyright G Bell & TCM History Center 29

Cray contributions… n n n Creative and productive during his entire career 1951 -1996. Creator and un-disputed designer of supers from c 1960 1604 to Cray 1, 1 s, 1 m c 1977… basis for SMPvector: XMP, YMP, T 90, C 90, 2, 3 Circuits, packaging, and cooling… “the mini” as a peripheral computer Use I/O computers versus I/O processors Use the main processor and interrupt it for I/O versus I/O processors aka IBM Channels 10/21/2021 Copyright G Bell & TCM History Center 30

Cray Contributions n n n Multi-theaded processor (6600 PPUs) CDC 6600 functional parallelism leading to RISC… software control Pipelining in the 7600 leading to. . . Use of vector registers: adopted by 10+ companies. Mainstream for technical computing Established the template for vector supercomputer architecture SRC Company use of x 86 micro in 1986 that could lead to largest, sm. P? 10/21/2021 Copyright G Bell & TCM History Center 31

“Cray” Clock speed (Mhz), no. of processors, peak power (Mflops) 10/21/2021 Copyright G Bell & TCM History Center 32

CDC 1604 & 6600 10/21/2021 Copyright G Bell & TCM History Center 34

CDC 7600: pipelining 10/21/2021 Copyright G Bell & TCM History Center 35

CDC 8600 Prototype: SMP, scalar, discrete circuits, failed to achieve clock speed 10/21/2021 Copyright G Bell & TCM History Center 36

CDC STAR… ETA 10 10/21/2021 Copyright G Bell & TCM History Center 37

CDC 7600 & Cray 1 at Livermore Cray 1 CDC 7600 Disks 10/21/2021 Copyright G Bell & TCM History Center 38

Cray 1 #6 from LLNL. Located at The Computer Museum History Center, Moffett Field 10/21/2021 Copyright G Bell & TCM History Center 39

Cray 1 150 Kw. MG set & heat exchanger 10/21/2021 Copyright G Bell & TCM History Center 40

Cray XMP/4 Proc. c 1984 10/21/2021 Copyright G Bell & TCM History Center 41

Cray 2 from NERSC/LBL 10/21/2021 Copyright G Bell & TCM History Center 42

Cray 3 c 1995 processor 500 MHz 32 modules 1 K Ga. As ic’s/module 8 proc. 10/21/2021 Copyright G Bell & TCM History Center 43

c 1970: Beginning the search for parallelism SIMDs n Illiac IV n CDC Star n Cray 1 n 10/21/2021 Copyright G Bell & TCM History Center 44

Iliac IV: first SIMD c 1970 s 10/21/2021 Copyright G Bell & TCM History Center 45

SCI (Strategic Computing Initiative) funded by DARPA and aimed at a Teraflops! Era of State computers and many efforts to build high speed computers… lead to HPCC Thinking Machines, Intel supers, Cray T 3 series 10/21/2021 Copyright G Bell & TCM History Center 46

Minisupercomputers: a market whose time never came. Alliant, Convex, Ardent+Stellar= Stardent = 0, 10/21/2021 Copyright G Bell & TCM History Center 47

Cydrome and Multiflow: prelude to wide word parallelism in Merced Minisupers with VLIW attack the market n Like the minisupers, they are repelled n It’s software, and software n Was it a basically good idea that will now work as Merced? n 10/21/2021 Copyright G Bell & TCM History Center 48

Mas. Par. . . A less costly, CM 1/2 done in silicon chips n It is repelled. n S is the fatal flaw n 10/21/2021 Copyright G Bell & TCM History Center 49

Thinking Machines: 10/21/2021 Copyright G Bell & TCM History Center 50

Thinking Machines: CM 1 & CM 5 c 1983 -1993 10/21/2021 Copyright G Bell & TCM History Center 51

“ In Dec. 1995 computers with 1, 000 processors will do most of the scientific processing. ” Danny Hillis 1990 (1 paper or 1 company) 10/21/2021 Copyright G Bell & TCM History Center 52

The Bell-Hillis Bet Massive Parallelism in 1995 TMC TMC World-wide Supers Applications Petaflops / mo. Revenue 10/21/2021 Copyright G Bell & TCM History Center 53

Bell-Hillis Bet: wasn’t paid off! My goal was not necessarily to just win the bet! n Hennessey and Patterson were to evaluate what was really happening… n Wanted to understand degree of MPP progress and programmability n 10/21/2021 Copyright G Bell & TCM History Center 54

KSR 1: first commercial DSM NUMA (non-uniform memory access) aka COMA (cache-only memory architecture) 10/21/2021 Copyright G Bell & TCM History Center 55

SCI (c 1980 s): Strategic Computing Initiative funded ATT/Columbia (Non Von), BBN Labs, Bell Labs/Columbia (DADO), CMU Warp (GE & Honeywell), CMU (Production Systems), Encore, ESL, GE (like connection machine), Georgia Tech, Hughes (dataflow), IBM (RP 3), MIT/Harris, MIT/Motorola (Dataflow), MIT Lincoln Labs, Princeton (MMMP), Schlumberger (FAIM-1), SDC/Burroughs, SRI (Eazyflow), University of Texas, 10/21/2021 Copyright G Bell & TCM History Center Thinking Machines (Connection Machine), 56

Those who gave their lives in the search for parallellism Alliant, American Supercomputer, Ametek, AMT, Astronautics, BBN Supercomputer, Biin, CDC, Chen Systems, CHOPP, Cogent, Convex (now HP), Culler, Cray Computers, Cydrome, Dennelcor, Elexsi, ETA, E & S Supercomputers, Flexible, Floating Point Systems, Gould/SEL, IPM, Key, KSR, Mas. Par, Multiflow, Myrias, Ncube, Pixar, Prisma, SAXPY, SCS, SDSA, Supertek (now Cray), Suprenum, Stardent (Ardent+Stellar), Supercomputer Systems Inc. , Synapse, Thinking Machines, Vitec, Vitesse, Wavetracer. 10/21/2021 Copyright G Bell & TCM History Center 57

NCSA Cluster of 8 x 128 processors SGI Origin c 1999 10/21/2021 Copyright G Bell & TCM History Center 58

Humble beginning: In 1981… would you have predicted this would be the basis of supers? 10/21/2021 Copyright G Bell & TCM History Center 59

Intel’s ipsc 1 & Touchstone Delta 10/21/2021 Copyright G Bell & TCM History Center 60

Intel Sandia Cluster 9 K PII: 1. 8 TF 10/21/2021 Copyright G Bell & TCM History Center 61

GB with NT, Compaq, HP cluster 10/21/2021 Copyright G Bell & TCM History Center 62

The Alliance LES NT Supercluster “Supercomputer performance at mail-order prices”-- Jim Gray, Microsoft • Andrew Chien, CS UIUC-->UCSD • Rob Pennington, NCSA • Myrinet Network, HPVM, Fast Msgs • Microsoft NT OS, MPI API 192 HP 300 MHz 64 Compaq 333 MHz 10/21/2021 Copyright G Bell & TCM History Center 63

Our Tax Dollars At Work ASCI for Stockpile Stewardship Intel/Sandia: 9000 x 1 node Ppro n LLNL/IBM: 512 x 8 Power. PC (SP 2) n LANL/Cray: 6144 CPUs n Maui Supercomputer Center n – 10/21/2021 512 x 1 SP 2 Copyright G Bell & TCM History Center 64

ASCI Blue Mountain 3. 1 Tflops SGI Origin 2000 12, 000 sq. ft. of floor space 1. 6 MWatts of power 530 tons of cooling 384 cabinets to house 6144 CPU’s with 1536 GB (32 GB / 128 CPUs) 48 cabinets for metarouters 96 cabinets for 76 TB of raid disks 36 x HIPPI-800 switch Cluster Interconnect 9 cabinets for 36 HIPPI switches about 348 miles of fiber cable 10/21/2021 Copyright G Bell & TCM History Center 65

Half of SGI ASCI Computer at LASL c 1999 10/21/2021 Copyright G Bell & TCM History Center 66

LASL ASCI Cluster Interconnect 18 Separate Networks 18 16 x 16 Crossbar Switches 1 2 3 1 4 5 2 6 7 8 3 9 10 11 4 12 13 14 15 16 5 17 18 6 6 Groups of 8 Computers each 10/21/2021 Copyright G Bell & TCM History Center 67

LASL ASCI Cluster Interconnect 10/21/2021 Copyright G Bell & TCM History Center 68

3 Tera. Ops makes a difference! ASCI Blue Mountain MCNP simulation: • 1 mm resolution (256 x 250) • 100 million particles • 2 hours on 6144 CPUs Typical MCNP BNCT simulation: • 1 cm resolution (21 x 25) • 1 million particles • 1 hour on 200 MHz PC 10/21/2021 Copyright G Bell & TCM History Center 69

LLNL Architecture System Parameters • 3. 89 TFLOP/s Peak • 2. 6 TB Memory • 62. 5 TB Global disk Sector S Hi. PPI 2. 5 GB/node Memory 24. 5 TB Global Disk 8. 3 TB Local Disk 12 6 24 HPGN 24 Sector Y Each SP sector has • 488 Silver nodes • 24 HPGN Links 10/21/2021 FDDI 6 Sector K 1. 5 GB/node Memory 24 20. 5 TB Global Disk 4. 4 TB Local Disk SST Achieved >1. 2 TFLOP/s on s. PPM and Problem >70 x Larger Than Ever Solved Before! 1. 5 GB/node Memory 20. 5 TB Global Disk Copyright G Bell & TCM History Center 4. 4 TB Local Disk 70

I/O Hardware Architecture 488 Node IBM SP Sector GPFS GPFS 56 GPFS Servers System Data and Control Networks 24 SP Links to Second Level Switch 432 Silver Compute Nodes Each SST Sector Full system mode • local and global I/O file system • Application launch over full 1, 464 Silver nodes • 2. 2 GB/s global I/O performance • 1, 048 MPI/us tasks, 2, 048 MPI/IP tasks • 3. 66 GB/s local I/O performance • High speed, low latency communication 10/21/2021 Copyright G Bell & TCM History Center. STDIO interface • Separate SP first level switches • Single 71

Fujitsu VPP 5000 multicomputer: (not available in the U. S. ) Computing nodes speed: 9. 6 Gflops vector, 1. 2 Gflops scalar primary memory: 4 -16 GB memory bandwidth: 76 GB/s (9. 6 x 64 Gb/s) inter-processor comm: 1. 6 GB/s non-blocking with global addressing among all nodes I/O: 3 GB/s to scsi, hippi, gigabit ethernet, etc. n 1 -128 computers deliver 1. 22 Tflops n 10/21/2021 Copyright G Bell & TCM History Center 72

NEC SX 5: clustered SMPv (not available in the U. S. ) n SMPv computing nodes 4 - 8 processors/computer – Processor pap: 8 Gflops – Memory – I/O speed – n Cluster 10/21/2021 Copyright G Bell & TCM History Center 73

NEC Supers 10/21/2021 Copyright G Bell & TCM History Center 74

High Performance COTS n Raceway and (RACE++) Busses ANSI Standardized – Mapped Memory, Message Passing, ‘Planned Direct’ Transfers – Circuit Switched; Basic Bus Interface Unit Is a 6 (8) Port Bidirectional Switch at 40 MB/s (66 MB/s) Per Port. – Scales to » 4000 Processors – n Skychannel ANSI Standardized – 320 mb/sec; Crossbar backplane supports up to 1. 6 GB/s Throughput Non-blocking – Heart of Air Force $3 M / 256 Gflops System – 10/21/2021 Copyright G Bell & TCM History Center 75

Mercury & Sky Computers - & $ Rugged System With 10 Modules ~ $100 K; $1 K /# Scalable to several K processors; ~1 -10 Gflop / Ft 3 10 9 U Boards * 4 Ppc 750’s » 440 Specfp 95 in 3 1 Ft (18. 5 * 8 * 10. 75”) Sky 384 Signal Processor, #20 on ‘Top 500’, $3 M 10/21/2021 VME Platinum Copyright GSystem Bell & TCM History Center Mercury Sky PPC Daughtercard 76

Brookhaven/Columbia QCD c 1999 (1999 Bell Prize for performance/$) 10/21/2021 Copyright G Bell & TCM History Center 77

Brookhaven/Columbia QCD board 10/21/2021 Copyright G Bell & TCM History Center 78

HT-MT: What’s 0. 55? c 1999 10/21/2021 Copyright G Bell & TCM History Center 79

HT-MT… Mechanical: cooling and signals n Chips: design tools, fabrication n Chips: memory, PIM n Architecture: mta on steroids n Storage material n 10/21/2021 Copyright G Bell & TCM History Center 80

HTMT challenges the heuristics for a successful computer Mead 11 year rule: time between lab appearance and commercial use n Requires >2 break throughs n Team’s first computer or super n It’s government funded… albeit at a university n 10/21/2021 Copyright G Bell & TCM History Center 81