CS 732 Advance Machine Learning Usman Roshan Department
- Slides: 8
CS 732: Advance Machine Learning Usman Roshan Department of Computer Science NJIT
Parallel computing • Why in an advance machine learning course? • Some machine learning programs take a long time to finish. For example large neural networks and kernel methods. • Dataset sizes are getting larger. While linear classification and regression programs are generally very fast they can be slow on large datasets.
Examples • Dot product evaluation • Gradient descent algorithms • Cross-validation – Evaluating many folds in parallel – Parameter estimation • http: //www. nvidia. com/object/data-scienceanalytics-database. html
Parallel computing • Multi-core programming – Open. MP: ideal for running same program on different inputs – MPI: master slave setup that allows message passing • Graphics Processing Units: – Equipped with hundred to thousand cores – Designed for running in parallel hundreds of short functions called threads
GPU programming • Memory has four types with different sizes and access times – – Global: largest, ranges from 3 to 6 GB, slow access time Local: same as global but specific to a thread Shared: on-chip, fastest, and limited to threads in a block Constant: cached global memory and accessible by all threads • Coalescent memory access is key to fast GPU programs. Main idea is that consecutive threads access consecutive memory locations.
GPU programming • Designed for running in parallel hundreds of short functions called threads • Threads are organized into blocks which are in turn organized into grids • Ideal for running the same function on millions of different inputs
Languages • CUDA: – C-like language introduced by NVIDIA – CUDA programs run only on NVIDIA GPUs • Open. CL: – Open. CL programs run on all GPUs – Same as C – Requires no special compiler except for opencl header and object files (both easily available)
CUDA • We will compile and run a program for determining interacting SNPs in a genomewide association study • Location: http: //www. cs. njit. edu/usman/Chi 8
