MPI in u Clinux on Microblaze Neelima Balakrishnan

MPI in u. Clinux on Microblaze Neelima Balakrishnan Khang Tran 05/01/2006

Project Proposal o Port u. Clinux to work on Microblaze o Add MPI implementation on top of u. Clinux o Configure NAS parallel benchmarks and port them to work on RAMP

What is Microblaze? o o Soft core processor, implemented using general logic primitives 32 -bit Harvard RISC architecture Supported in the Xilinx Spartan and Virtex series of FPGAs Customizability of the core makes it challenging while opening up vistas for kernel configurations

Components o o u. Clinux - kernel v 2. 4 MPICH 2 - portable, high performance implementation of the entire MPI-2 standard n n Communication via different channels sockets, shared memory, etc. MPI port for Microblaze communication is over FSL

Components (contd. ) o NASPB v 2. 4 - MPI-based source code implementations written and distributed by NAS n n 5 kernels 3 pseudo-applications

Porting u. Clinux to Microblaze o o o Done by Dr. John Williams - Embedded Systems group, University of Queensland in Brisbane, Australia Part of their reconfigurable computing research program. The work on this is still going on http: //www. itee. uq. edu. au/~jwilliams/mblazeuclinux

Challenge in porting u. Clinux to Microblaze o Linux derivative for microprocessors that lack a memory management unit (MMU) n n No memory protection No virtual memory For most user applications, the fork() system call is unavailable malloc() function call needs to be modified

MPI implementation o o MPI – Message Passing Interface Standard API used to create parallel applications Designed primarily to support the SPMD (single program multiple data) model Advantage over older message passing libraries n n Portability Fast as each implementation is optimized for the hardware it runs on

Interactions between Application and MPI Other processors ……………. Communication Channel MPI process manager MPI interface Initiating application MPI process manager MPI interface Application on other processors

NAS parallel benchmarks o o Set of 8 programs intended to aid in evaluating the performance of parallel supercomputers Derived from computational fluid dynamics (CFD) applications, n n o 5 kernels 3 pseudo-applications Used NPB 2. 4 version – MPI-based source code implementation

Phases o o o Studied the u. Clinux and found the initial port done for Microblaze Latest kernel (2. 4) and distribution from u. Clinux. org Successful compilation for Microblaze architecture MPI - MPICH 2 out of many versions of MPI Investigated the MPICH 2 implementation available from Argonne National Laboratory Encountered challenges in porting MPI onto u. Clinux

Challenges in porting MPI to u. Clinux o o o Use of fork and a complex state machine Default process manager for unix platforms is MPD written in Python and uses a wrapper to call fork Simple fork->vfork is not possible as the function is called deep inside other functions and require a lot of stack unwinding Alternate Approaches Port SMPD, written in C n It will involve a complex state machine and stack unwinding after the fork Use pthreads n Might involve a lot of reworking of code as the current implementation is not using pthreads n Need to ensure thread safety

NAS Parallel Benchmark o o o Used NAS PB v 2. 4 Compiled and executed it on a desktop and Millennium Cluster Obtained information about n n n MOPS Type of operation Execution time Number of nodes involved Number of processes and iterations

NAS PB simulation result (Millennium cluster, Class A)

Simulation result (cont. )

Estimated statistics for the floating point group o 4 test benches use floating point op heavily are: BT, CG, MG, and SP n n n Very few fp comparison ops in all BT (Block Tridiagonal) all fp ops are add, subtract, and multiply. About 5% of all ops is division CG (Conjugate Gradient) has the highest % of ops that is sqrt, 30%. Add, mult is about 60%, divide is about 10%. MG (Multigrid) about 5% is sqrt, 20% is division. The rest is add, subtract, and multiply For SP (Scalar Pentadiagonal) almost all ops are add, 10% is division

Floating Point Operation Frequency

Most frequently used MPI functions in NASPB v 2. 4

Observations about NASPB o NASPB suite – 6 out of 8 benchmarks are predictive of parallel performance n EP – little/negligible communication between processors. n IS – high communication overhead.

Project status o o o Compiled u. Clinux and put it on Microblaze Worked on the porting of MPI but not completed Compiled and executed NASPB on desktop and Millennium (which currently uses 8 computing nodes)