Lambiente software di ape NEXT sviluppo ed esecuzione

L'ambiente software di ape. NEXT: sviluppo ed esecuzione delle applicazioni Alessandro Lonardo I. N. F. N Roma - gruppo APE* alessandro. lonardo@roma 1. infn. it *http: //apegate. roma 1. infn. it/APE Alessandro Lonardo - 18/12/06

Index • • Machine Architecture Software Areas Programming Model Languages Example Applications Development Tools Execution Environment Alessandro Lonardo - 18/12/06

ape. NEXT Architecture The Network • 3 D mesh of computing nodes • Vertexes are processors • Each proc hosts its local memory • Each proc supports 64 bit complex, vector 2, double and integer types • Edges are 3 D torus network channels • 6 bi-dir channels per proc • Basic comm primitive is firstneighbour send-recv • Processors synchronize on communications (send starts when recv is issued) Alessandro Lonardo - 18/12/06

ape. NEXT Architecture The J&T Processor Alessandro Lonardo - 18/12/06

ape. NEXT Architecture Very Long Instruction Word Control Word Decoder&Scheduler 3 2 1 D 1 S 1 D 2 D 3 Alessandro Lonardo - 18/12/06 D 1 D 2 D 3

ape. NEXT Architecture The J&T FILU is FP, Integer and Logical unit: • MAC op: A*B+C • fully pipelined(1 result per cycle) • ~ 12 cycles latency • synthesizes to 200 MHz • 4 multipliers • 4 adders • 1. 6 GFlops on complex MAC Alessandro Lonardo - 18/12/06

ape. NEXT Software Areas • Architecture design, development and validation: simulators, no regression tools, … • Application development: compilation chain, libraries, profiler… • Execution environment: operating system, batch system, … • Applications. • System administration. Alessandro Lonardo - 18/12/06

ape. NEXT SW development team • Average ~5 persons • People in all the collaboration sites – INFN Roma & Ferrara – Desy Zeuthen – Univ. Bielefeld – INRIA (France) Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model • Single Program Multiple Data: each node executes the same program, but on its own data. – synchronization barriers at global condition evaluations, with explicit statement or at I/O operations; – node to node synchronization at remote communications. • Nodes are connected by a 3 D network, each node can efficently transfer data with its first, second and third neighbour. => well suited for homogeneous problems with short range interactions. Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Data Decomposition(1) • Application discretized D-dim lattice domain decomposed onto a 3 -dim processor mesh (maybe D != 3) => Each node has a subset of the lattice sites in its own memory • In other cases no decomposition is done, simulation is done in parallel without communications just to have a better statistics (FARM). Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Data Decomposition(2) x 00 01 For each lattice site and on each node in parallel the program performs an “evolutionary step”. Short-range interactions => first neighbour inter-node communication. y 10 11 Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Programming Languages(1) TAO dedicated parallel language • Fortran-like base syntax. • Dynamic Language: the (experienced) programmer can freely extend syntax with new statements, data types and operators => libraries configure the language for specific application domains (LQCD, Spin Glass, …) • Allows writing of high efficency codes by exposing the features of the hardware architecture (registers, prefetch queues, cache) Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Programming Languages(2) C 99 language • Few extensions to the standard language. • Eases the porting of applications and standard libraries. • Allows writing of high efficency codes by exposing the features of the hardware architecture (registers, prefetch queues, cache) Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Parallel Language Constructs(1) few parallel language constructs (same in C 99 and TAO): • Conditioned execution on a subset of nodes based on local to node conditions (where) • Boolean operators for promotion of local to global conditions to be used in flow control statements (any, all, none). • Communications between nodes in the 3 D mesh expressed as variable assignment, directions specified by mean of magic constants in the source address (X_PLUS, X_MINUS, …, Z_MINUS). Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Parallel Language Constructs(2) where statement - conditional execution on a mesh subset. where (x>=y) max_xy=x min_xy=y elsewhere max_xy=y min_xy=x endwhere Alessandro Lonardo - 18/12/06

ape. NEXT Programming Model Parallel Language Constructs(3) Inter-node communications: integer i real u[1024] register real rd 1, rd 2, rd 3, rloc. . . loads u[i] from node rd 1 = u[i+Z_PLUS] rd 2 = u[i+Y_PLUS+Z_PLUS] rloc = u[i+X_PLUS+X_MINUS] rd 3 = u[i+X_PLUS+Y_MINUS+Z_PLUS] Alessandro Lonardo - 18/12/06 [x, y, z+1] into rd 1

ape. NEXT Programming Model Parallel Language Constructs(4) any()/all()/none() boolean operators !!evaluation of mesh size along X !!with systolic algorithm sum_ix=1 sum_r[0]=node_abs_x sum_r[0]=sum_r[X_PLUS] !!internode communication while(any(sum_r[0]!=node_abs_x)) sum_ix=sum_ix+1 sum_r[0]=sum_r[X_PLUS] endwhile Alessandro Lonardo - 18/12/06

example 2 D application kernel C function T datain[LVOL], dataout[LVOL]; // LVOL is node local volume // precalculate neighbourhood tables int neighp[LVOL, 2], neighm[LVOL, 2]; . . . void kernel_fun() { register T res, d, dp 0, dp 1, dm 0, dm 1; for(i=0; i<LVOL; ++i) { // i is a linearized index d = datain[i]; // always local access dp 0 = datain[neighp[i, 0]]; // local or remote access dp 1 = datain[neighp[i, 1]]; dm 0 = datain[neighm[i, 0]]; dm 1 = datain[neighm[i, 1]]; res = calc(d, dp 0, dp 1, dm 0, dm 1); // big & inline dataout[i] = res; } } Alessandro Lonardo - 18/12/06

example 2 D application kernel domain decomposition x domain decomposition of the datain[] array y datain[] local domain of nodexy datain[LVOL] boundary of local domain Alessandro Lonardo - 18/12/06

example 2 D application kernel First Neighbour Systolic Communication x 00 10 y dm 0 = datain[neighp[i, 0]]; neighp[i, 0]: local displacement +X_MINUS neighp[i, 0]: local displacement 01 11 Alessandro Lonardo - 18/12/06

Example: Monte Carlo Pi Calculation • Estimate Pi by throwing darts at a unit square • Calculate percentage that fall in the unit circle – Area of square = r 2 = 1 – Area of circle quadrant = ¼ * p r 2 = p/4 • Randomly throw darts at x, y positions • If x 2 + y 2 < 1, then point is inside circle • Compute ratio: – # points inside / # points total – p = 4*ratio • Replicate the calculation on N nodes in parallel to have better statistics r =1 Alessandro Lonardo - 18/12/06

Example: Monte Carlo Pi Calculation C+Open. MP Code #include <stdio. h> <math. h> <stdlib. h> "omp. h" srand(FIRST_SEED); for(i=0; i<max_threads; i++) /*scorrelo i seeds*/ { seeds[i] = rand(); inline int hit(){ printf("seed%d=%dn", i, seeds[i]); double x = (double) rand() / (double) RAND_MAX; } double y = (double) rand() / (double) RAND_MAX; #pragma omp parallel private(i, seeds_index ) if ((x*x + y*y) <= 1. 0) return(1); shared(seeds, hits, trials) else return(0); { } seeds_index = omp_get_thread_num(); srand(seeds[seeds_index]); #define FIRST_SEED 3374 int main(int argc, char **argv) { #pragma omp for reduction(+: hits) int i, hits = 0, trials = 0; for (i=0; i < trials; i++) int seeds_index = 0; hits += hit(); const int max_threads = omp_get_max_threads(); unsigned int seeds[max_threads]; } double pi; pi = 4. 0*(double)hits/(double)trials; printf("MAX_THREADS = %dn", max_threads); printf("PI estimated to %. 10 gn", pi); return 0; if (argc != 2) } trials = 1000000; else trials = atoi(argv[1]); Alessandro Lonardo - 18/12/06

Example: Monte Carlo Pi Calculation ape. NEXT C Code #include #include <stdio. h> <math. h> <stdlib. h> <sysvars. h> <topology. h> if (argc != 2) trials = 1000000; else trials = atoi(argv[1]); srand(FIRST_SEED); inline int hit(){ double x = (double) rand() / (double) RAND_MAX; double y = (double) rand() / (double) RAND_MAX; if ((x*x + y*y) <= 1. 0) return(1); else return(0); } int main(int argc, char **argv) { int i, hits = 0, trials = 0; int seeds_index = 0; const int max_threads = *_mem_imachine_size_x_p * *_mem_imachine_size_y_p * *_mem_imachine_size_z_p; const node_index = *_mem_inode_abs_id_p; unsigned int seeds[max_threads]; double pi; for(i=0; i<max_threads; i++) { seeds[i] = rand(); printf("seed%d=%dn", i, seeds[i]); } srand(seeds[node_index]); for (i=0; i < trials; i++) hits += hit(); hits = global_sum(hits); trials *= max_threads; pi = 4. 0*(double)hits/(double)trials; printf("PI estimated to %. 10 gn", pi); return 0; } printf("MAX_THREADS = %dn", max_threads); Alessandro Lonardo - 18/12/06

Example: Monte Carlo Pi Calculation Results – Intel P 4 Dual Core lonardo@marlin>env OMP_NUM_THREADS=16. /monte_pi-gcc. o MAX_THREADS = 16 seed 0=1396293760 seed 1=1488115307 seed 2=1303873515 seed 3=37393359 seed 4=824846176 seed 5=1138759395 seed 6=1184683763 seed 7=1884735975 seed 8=443160774 seed 9=326610858 seed 10=878347714 seed 11=501308535 seed 12=1066424433 seed 13=1420631951 seed 14=391631339 seed 15=1730610200 PI estimated to 3. 14108 lonardo@marlin> Alessandro Lonardo - 18/12/06

Example: Monte Carlo Pi Calculation Results - ape. NEXT Board (16 Nodes) lonardo@ant>nrun -hib -board 033 -minit 0 monte-api. mem MAX_THREADS = 16 seed 0=6556077425992558173 seed 1=4923530068770806084 seed 2=4637196908100545377 seed 3=6221712952809700854 seed 4=279065984179923185 seed 5=7751953660738243840 seed 6=7614450982016732205 seed 7=1120288809807653798 seed 8=4640801604175907269 seed 9=4885633457180056444 seed 10=905770433927994553 seed 11=1598073754810041858 seed 12=7232028785291230425 seed 13=6726612558212505416 seed 14=3567338195430110971 seed 15=5194800804163472670 PI estimated to 3. 13989775 lonardo@ant> Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain rtc tao compiler • Retargetable Tao Compiler: produce an intermediate pseudo-assembly file which is further translated into assembly for APEmille or ape. NEXT. • Based on Zz dynamic parser. • Relies on a separate module for assembly code optimizations. • Stable, production quality compiler Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain nlcc c compiler • lcc 4. 2 compiler port on ape. NEXT architecture. • few optimizations. • c 99 + ape. NEXT syntax extensions • Low bug reports rate. Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain ngcc c compiler • Porting of GNU C compiler (GCC) for ape. NEXT architecture • Based on gcc version 4. 1 • Optimization passes performed on the compiler’s internal representation of code (tree-SSA, RTL) • Source language: C 99 and GNU Extensions to C 99, ape. NEXT extensions for parallel programming • Possibility to integrate frontends to other source languages (C++, Fortran, TAO) • Target language: ape. NEXT user level assembly (SASM) Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain ngcc status: • Single node C compiler – DONE • Vector data types and arithmetics – ALMOST DONE • Exploitation of native complex types and arithmetics – TO DO • Remote memory accesses implementation – DONE • Prefetch instructions – ALMOST DONE • Cache handling – TO DO • Where(), any(), all(), none() constructs – TO DO • libc adaptation – JUST STARTED • Work in progress Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain mpp macro-assembler • translates a “user-friendly” assembly into a microassembly representation • macro expansion. • label analisys. • emission of masm-instructions for cache handling. Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain sofan micro-assemby optimizer • based on the salto (INRIA) optimization toolkit • Transforms the micro-assembly code in order to perform a series of optimizations, such as: – mul-add fusion – Dead code removal – Copy propagation – Address generation optimization – Intruction pre-scheduling Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain shaker microcode scheduler • generation of optimized microcode to exploit the Pipelined Very Long Instruction Word Processor Architecture – scheduling – Register renaming – Register allocation – Microcode compression – Optional generation of executable for the functional simulator Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain shaker microcode scheduler • generation of microcode patterns, texec = tmax DEVICES 0 1 1 0 1 2 CYLES 1 3 2 2 3 3 3 2 4 4 1 shake up 2 2 4 5 5 3 3 4 4 5 tsu 4 5 5 • “shake up” phase: try to schedule each pattern earlier as possible respecting: – dependencies between instructions – device occupation at each cycle texec = tsu 5 tmax 1 5 5 Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain shaker microcode scheduler 0 DEVICES CYCLES 1 tsu • “shake down” phase: try to schedule each pattern later as possible respecting: 2 1 1 3 2 2 3 3 3 2 4 4 4 5 5 tsd shake down 2 3 4 1 3 2 2 3 3 2 5 5 5 1 1 tsu 5 4 4 5 4 – dependencies between instructions – device occupation at each cycle texec = tsu- tsd • Tipically tmax / texec ~ 10 in computing intensive code sections Alessandro Lonardo - 18/12/06

ape. NEXT Compilation Chain sf functional simulator • micro-assembly. Instruction level simulator. • Support for single and multinode simulations (1 x 1 x 1, 2 x 2 x 2, 4 x 2 x 2). • Fast simulation (multithreaded) • no cycle accurate. • bit exact arithmetic (microcode scheduling may give differences). Alessandro Lonardo - 18/12/06

ape. NEXT Execution Environment OS distributed architecture(1) I 2 C: bootstrap, exception handling, debugging (1. 5 MB/s) 7 th. Link: • Program loading • I/O operations 1 channel per unit Alessandro Lonardo - 18/12/06 200 MB/s per channel

ape. NEXT Execution Environment OS distributed architecture(2) • Master – resides on the front-end linux PC – user interface (shell commands) – Partitioning – dispatch I/O request to the slaves • Slave – Resides on the blade PCs – Handles communication with ape. NEXT on I 2 C and 7 th. Link PCI boards • tiny kernel of routines embedded in the ape. NEXT program – loader – I/O (routing of data to and from the interface node) – System services (time counters, etc) Alessandro Lonardo - 18/12/06

ape. NEXT Execution Environment • programs can be loaded and executed on a machine partition: – node (1 x 1 x 1) – board = 16 nodes (4 x 2 x 2) – unit = 4 boards (4 x 2 x 8) – crate = 4 units (4 x 8 x 8) – rack = 2 crates (8 x 8 x 8) • Partition is reserved until the program execution finishes (no multitasking!) – Single process – No virtual memory Alessandro Lonardo - 18/12/06

Batch system • Torque/Open. PBS • today fifo-Scheduling, implementing a users group quota based scheduler. • queues: – rack – crate – unit Alessandro Lonardo - 18/12/06

Batch System Job Submission • nsub: wrapper of the qsub command Usage: nsub [OPTIONS] script Submits a ape. NEXT job where OPTIONS are: -a date_time -c conf -m host_name -g group_name -o logfile -V -v -h Declares the time after which the job is eligible for execution the format is: [[[[CC]YY]MM]DD]hhmm[. SS] chooses among available ape. NEXT configurations conf=board|unit[01][0 -3]|crate[01] |rack(default=crate) requests a particular host overrides user group overrides logfile name dumps version information be verbose shows this help Alessandro Lonardo - 18/12/06

Batch System Job submission example lonardo@theboss>nsub -c crate 7 h_test. sh 15942. theboss. ape lonardo@theboss>qstat -an 1 theboss. ape: Job ID ----------15813. theboss. ape 15860. theboss. ape 15877. theboss. ape 15880. theboss. ape 15896. theboss. ape 15900. theboss. ape 15906. theboss. ape 15918. theboss. ape 15926. theboss. ape 15927. theboss. ape 15928. theboss. ape 15930. theboss. ape 15932. theboss. ape 15933. theboss. ape 15934. theboss. ape 15935. theboss. ape 15936. theboss. ape 15937. theboss. ape 15939. theboss. ape 15940. theboss. ape 15941. theboss. ape 15942. theboss. ape Username -------orifici simula zeidlew delia frezzott simula delia lacagnin cosmai delia devitiis orifici devitiis cosmai devitiis delia lonardo Queue -------crate unit rack crate unit crate crate unit crate Req'd Elap Jobname Sess. ID NDS TSK Memory Time S Time ------ --- ----- - ----stoc 2 1826 1 --- 24: 00 R 21: 52 run_tdilu 29170 1 --- 24: 00 R 14: 34 mu. 056. cjo 5925 1 --- 24: 00 R 11: 38 run. sh 6386 1 --- 24: 00 R 10: 59 rum 0175. sh 32291 1 --- 24: 00 R 08: 47 RUN_Rack 5. 18099 1 --- 24: 00 R 07: 56 run 0175. sh 1072 1 --- 24: 00 R 06: 49 RUN_Rack 2. 15890 1 --- 24: 00 R 04: 30 run 1_tdilu 4409 1 --- 24: 00 R 02: 47 run 0200. sh 3596 1 --- 24: 00 R 02: 47 run. 5. 7. sh 4772 1 --- 24: 00 R 02: 36 run. 5. 6. sh 2787 1 --- 24: 00 R 02: 34 b 5. 450_n 0. 2994 1 --- 24: 00 R 02: 02 rum 0200. sh 4216 1 --- 24: 00 R 01: 53 run 0225. sh 4552 1 --- 24: 00 R 01: 24 theboss. sh 28504 1 --- 24: 00 R 01: 22 stoc 28592 1 --- 24: 00 R 00: 57 theboss. sh 18065 1 --- 24: 00 R 00: 57 b 5. 450_n 1. 5845 1 --- 24: 00 R 00: 54 theboss. sh 18472 1 --- 24: 00 R 00: 22 rum 0225. sh 5290 1 --- 24: 00 R 00: 14 7 h_test. sh 11226 1 --- 24: 00 R -- Alessandro Lonardo - 18/12/06 rack 10/1 rack 4/1 rack 8/0 rack 8/1 rack 7/5 rack 5/0 rack 7/0 rack 2/0 rack 4/0 rack 7/4 rack 1/0 rack 9/1 rack 7/2 rack 7/1 rack 3/0 rack 3/1 rack 6/0 rack 1/1 rack 6/1 rack 7/3 rack 10/0