Hybrid Redux CUDA MPI CUDA MPI Hybrid Why
![Hybrid Redux: CUDA / MPI Hybrid Redux: CUDA / MPI](https://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-1.jpg)
Hybrid Redux: CUDA / MPI
![CUDA / MPI Hybrid – Why? � Harness � 16 � You more hardware CUDA / MPI Hybrid – Why? � Harness � 16 � You more hardware](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-2.jpg)
CUDA / MPI Hybrid – Why? � Harness � 16 � You more hardware CUDA GPUs > 1! have a legacy MPI code that you’d like to accelerate. 2
![CUDA / MPI – Hardware? � In a cluster environment, a typical configuration is CUDA / MPI – Hardware? � In a cluster environment, a typical configuration is](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-3.jpg)
CUDA / MPI – Hardware? � In a cluster environment, a typical configuration is one of: � Tesla � 1 S 1050 cards attached to (some) nodes. GPU for each node. � Tesla S 1070 server node (4 GPUs) connected to two different host nodes via PCI-E. � 2 GPUs per node. � Sooner’s CUDA nodes are like the latter � Those nodes are used when you submit a job to the queue “cuda”. � You 3 can also attach multiple cards to a workstation.
![CUDA / MPI – Approach � CUDA 1. 2. 3. will likely be: Doing CUDA / MPI – Approach � CUDA 1. 2. 3. will likely be: Doing](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-4.jpg)
CUDA / MPI – Approach � CUDA 1. 2. 3. will likely be: Doing most of the computational heavy lifting Dictating your algorithmic pattern Dictating your parallel layout � i. e. which nodes have how many cards � Therefore: 1. 2. 4 We want to design the CUDA portions first, and We want to do most of the compute in CUDA, and use MPI to move work and results around when needed.
![Writing and Testing � Use MPI for what CUDA can’t do/does poorly � communication! Writing and Testing � Use MPI for what CUDA can’t do/does poorly � communication!](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-5.jpg)
Writing and Testing � Use MPI for what CUDA can’t do/does poorly � communication! � Basic Organization: � Get data to node/CUDA card � Compute � Get result out of node/CUDA card � Focus � Be able to “turn off” either MPI or CUDA for testing � Keep this 5 testing on these three spots to pinpoint bugs a serial version of the CUDA code around just for
![Compiling � Basically, � These � Dirty we need to use both mpicc and Compiling � Basically, � These � Dirty we need to use both mpicc and](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-6.jpg)
Compiling � Basically, � These � Dirty we need to use both mpicc and nvcc are both just wrappers for other compilers trick – wrap mpicc with nvcc: nvcc -arch sm_13 --compiler-bindir mpicc driver. c kernel. cu � add 6 -g and -G to add debug info for gdb
![Running CUDA / MPI � Run no more than one MPI process per GPU. Running CUDA / MPI � Run no more than one MPI process per GPU.](http://slidetodoc.com/presentation_image_h2/947df0f0c67cc6414addca88f6a6456a/image-7.jpg)
Running CUDA / MPI � Run no more than one MPI process per GPU. � On sooner, this means each cuda node should be running no more than 2 MPI processes. � On Sooner, these cards can be reserved by using: #BSUB -R "select[cuda > 0]“ #BSUB -R "rusage[cuda=2]“ in your. bsub file. 7
- Slides: 7