A Performance Study of UCX over Infini Band

A Performance Study of UCX over Infini. Band Nikela Papadopoulou – National Technical University of Athens Lena Oden – Forschungszentrum Jülich Pavan Balaji – Argonne National Laboratory 11/22/2020 CCGRID’ 17

Motivation UCX is a new open-source communication middleware • Used by MPICH (developed at ANL), Open. MPI, Open. SHMEM • Has a two-level API design in attempt to meet software demands for HPC systems • Offers portability and programming ease Communication software demands on HPC systems • Scalability – minimal memory requirements • Performance – minimal instruction counts/cache activity • Programming ease - high-level abstractions • Portability – support for multiple communication devices 11/22/2020 CCGRID’ 17 1

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 2

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 3

UCX as communication middleware High level-communication software MPI, Open. SHMEM, UPC. . . Communication middleware UCX, Portals, OFI, . . . Low-level communication software IB Verbs, TCP, Cray u. GNI, . . . Communication devices Infini. Band, Ethernet, Cray Aries, . . . 11/22/2020 CCGRID’ 17 4

UCX High-level API: Protocols UCP UCX design Transport selection, Wireup, Fragmentation, Software protocols for operations that are not (necessarily) supported by hardware Initialization/ Wireup Tagmatching RMA Atomics Services 11/22/2020 IB Verbs TCP NVIDIA CUDA CCGRID’ 17 Cray u. GNI Shared memory SYSV POSIX KNEM CMA UCS Low-level API: Transports UCT Hardware-supported operations: Active Messages, RMA, Atomics Memory management Data structures 5

Infini. Band support in UCX over Infini. Band Verbs UCX with Accelerated Verbs UCX UCP UCT User APIs OS UCP Driver API Verbs API HCA Driver (MLX 4/MLX 5) Kernel-level Verbs Driver API OS Hardware Verbs API HCA Driver (MLX 5) Kernel-level Verbs Hardware Infini. Band HCA • UCT with Accelerated Verbs (AVerbs) available for the Mellanox MLX 5 driver 11/22/2020 CCGRID’ 17 6

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 7

UCX as middleware over Infini. Band UCP UCT Verbs Functionality High-level abstractions Generic API for native functions Native functions Portability Any hardware, multiple devices Any hardware with API implementation, single device Infini. Band only Performance Overhead due to additional APIs and high-level abstractions Overhead due to additional API Closest to hardware Scalability Extra memory requirements due to additional API and highlevel abstractions Extra memory requirements due to additional API Memory requirements relative to transport 11/22/2020 CCGRID’ 17 8

Measuring overheads of UCP over UCT over (A)Verbs • Comparison of UCP, UCT and Verbs/AVerbs: Latency and Instructions • Benchmark: RDMA write with remote completion • Transport: Infini. Band RC/x. RC status = ucp_put(ep, buffer, size, remote_address, remote_key); //Put (RDMA Write operation) if (status != UCS_OK) exit(ERROR); ucp_ep_flush(ep); //Flush operation for remote completion • UCP over RC (x. RC) • size <= 98 B (size <= 220 B): short protocol • 98 B < size <= 8 KB (220 B <size <= 8 KB): buffered copy protocol • size > 8 KB: fragmented buffered copy protocol • Identical protocol selection in our UCT and Verbs/AVerbs benchmarks 11/22/2020 CCGRID’ 17 9

Experimental Setup Hardware (JLSE resources) • Two Intel Xeon E 5 -2699 CPUs • Mellanox Infini. Band EDR HCA – Connect X-4 • Mellanox FDR-10 switch Software • UCX v 1. 0 (7/13/2016) – configured to use Infini. Band only • Intel C++ compiler 16. 0. 0 • -O 3 optimization, interprocedural optimization, inlining, AVX/SSE 4. 2 • Intel Software Development Emulator v 7. 48 11/22/2020 CCGRID’ 17 10

RDMA Write latency over Infini. Band RC Infini. Band x. RC (accelerated) • UCP slower than UCT, UCT slower than Verbs/AVerbs • Small UCT-to-Verbs overhead • Larger UCP-to-UCT overhead especially for short messages with RC 11/22/2020 CCGRID’ 17 11 Lower is better!

RDMA Write instructions over Infini. Band • Instruction breakdown in UCP for Put and Flush • UCP and UCT consume more instructions on Flush for short messages • Verbs/AVerbs consume more instructions on Put for short messages • From Verbs/AVerbs to UCT to UCP: • Put instructions increase slightly • Flush instructions increase significantly • UCX adds overheads over the native network functions 11/22/2020 CCGRID’ 17 12

Objectives • Analyze UCP core operations • Identify unnecessary overheads in UCP • Close the performance gap between UCP and UCT and native Infini. Band performance • Without sacrificing any of the UCP functionality / portability! 11/22/2020 CCGRID’ 17 13

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 14

RDMA operations in UCP Blocking RDMA functions: ucp_put, ucp_get • Select UCT interface • Select transfer protocol (short, bcopy, fragmented bcopy) • Call UCT function Non-blocking RDMA functions: ucp_put_nbi, ucp_get_nbi • Select UCT interface • Select transfer protocol • Call UCT function or push communication request • Requests created when no resources are available or for long messages • Overhead added with request creation/queue insertion 11/22/2020 CCGRID’ 17 15

Analyzing UCP overheads: ucp_put over Infini. Band RC/x. RC 8 B - x. RC 1 KB - x. RC ucp_put calls: • UCT function for communication • (A)Verbs function • Request creation • Bookkeeping • Function calls (15 -20) • Function pointers! • Checks to select transfer (6) • Memory copies • RESOLVE_RKEY_RMA (35) 11/22/2020 CCGRID’ 17 16

System UCT UCP Background: UCX communication context ucp_context Context ucp_key Interface s uct_mm_posix_iface Memory Domains uct_mm_md Transports Device s 11/22/2020 Registered Memory uct_mm_key uct_ib_rc_iface uct_ib_ud_iface uct_ib_md Registered Memory uct_ib_key UD POSIX RC Shared Memory CCGRID’ 17 Infini. Band 17

ucp_worker ucp_ep Endpoints uct_worker UCT • UCP selects the fastest available interface for RMA • RC/x. RC for Infini. Band RMA AMO Endpoints UCP The RESOLVE_RKEY_RMA function AM Wireup Domains/Interfaces uct_ep_mm _posix MM/POSI X uct_ep_ib_u d IB/UD uct_ep_ib_rc IB/RC 11/22/2020 CCGRID’ 17 • RESOLVE_RKEY_RMA performs UCP-to-UCT translations for: • endpoint • remote memory key • transport configuration • UCP-to-UCT remote key resolution occurs every time with an RDMA operation • Even for message fragments 18

Optimizing RDMA functions in UCP • Objective: Reduce the overhead of RESOLVE_RKEY_RMA • Optimization: Perform UCP-to-UCT translations within the ucp_rkey_unpack function • • Called at connection establishment Out of critical path for communication Necessary information stored within the UCP remote key data structure More details in the paper 11/22/2020 CCGRID’ 17 19

Optimizing RDMA functions in UCP Lower is better! 27 of 35 instructions of RESOLVE_RKEY_RMA function eliminated • Small impact on latency (<1%) 11/22/2020 CCGRID’ 17 20

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 21

Progress/Flush operations in UCP Progress function: ucp_worker_progress • Progress all communication operations of the UCP/UCT worker Flush functions: ucp_worker_flush, ucp_ep_flush • Complete all outstanding operations on the worker or the endpoint on the local side (through progress) 11/22/2020 CCGRID’ 17 22

Analyzing UCP overheads: ucp_ep_flush over Infini. Band RC/x. RC Flush after 8 B Put - x. RC UD interface is progressed not in UCT RX queue is polled – not in Verbs 11/22/2020 x. UD interface is progressed – not in UCT CCGRID’ 17 23

Optimizing the UCP progress engine (1) • Infini. Band interface progress polls both the Infini. Band send (TX) and receive (RX) queues for requests • RDMA operations create requests on the TX queue only • The RX queue must be polled by the progress engine for active messages • Objective: Eliminate unnecessary polling • The TX queue polling can be avoided if no messages are sent over an interface • Works for any interface • Optimization: Avoid unnecessary TX polling • Condition: If no send requests are pending • More details in the paper 11/22/2020 CCGRID’ 17 24

Optimizing the UCP progress engine (1) Lower is better! (x)UD TX polling avoided! • Significant improvement for RC • 12% instruction decrease for 8 Bmessage • Small impact on latency for x. RC • Polling is faster with AVerbs • 4. 8% instruction decrease for 8 Bmessage 11/22/2020 CCGRID’ 17 25

Background: Connection establishment/Wireup in UCP RMA UCP ucp_worker AMO ucp_ep_stu b ucp_ep AM Wireup ✉ � UCT uct_worker IB/UD uct_ep_ib_u d IB/RC uct_ep_ib_rc 11/22/2020 Domains/Interfaces CCGRID’ 17 26

Optimizing the UCP progress engine (2) • Progress on the worker progresses both the (x)RC and the (x)UD interface • RDMA operations take place over (x)RC! • The UD interface remains open for wireup • Always used for wireup over Infini. Band • Objective: Eliminate unnecessary interface progress • Upon UCP endpoint creation, a stub endpoint initiates endpoint connection • All created endpoints are connected after a few calls to the progress engine • Scalability bug in UCX! • If all UCP endpoints are connected, the (x)UD interface is no longer useful • Optimization: Eliminate (x)UD progress • Condition: If no endpoints are connected for the interface • More details in the paper 11/22/2020 CCGRID’ 17 27

Optimizing the UCP progress engine (2) Lower is better! (x)UD progress avoided! • Significant decrease in latency for RC • 29. 8% instruction decrease for 8 Bmessage • Small decrease in latency for x. RC • Polling is faster with AVerbs • 26. 6% instruction decrease for 8 Bmessage 11/22/2020 CCGRID’ 17 28

Outline UCX design UCX as middleware over Infini. Band Overheads and optimizations in UCP RDMA functions Overheads and optimizations in UCP progress functions Evaluation of optimizations in UCX Discussion and outlook 11/22/2020 CCGRID’ 17 29

Evaluating optimizations on UCP RDMA Write/Read • Bandwidth/Message rate measurement (RC and x. RC): • Issue 256 non-blocking RDMA operations (Write/Put or Read/Get) • TX queue length of 256 • Flush the endpoint • Evaluation: 1. UCP 2. UCP with Optimized RESOLVE_RKEY_RMA + Optimized TX polling 3. UCP with Optimized RESOLVE_RKEY_RMA + Optimized UD progress 11/22/2020 CCGRID’ 17 30

Evaluation with UCP RDMA Write over RC Peak BW at 8 KB Peak Message Rate at 8 B Higher is better! • Put optimization is good for mediumsided messages • More instructions spent on Put than on Flush • On bcopy, RESOLVE_RKEY_RMA called twice • Avoiding UD progress impacts the message rate for short messages 11/22/2020 CCGRID’ 17 31

Evaluation with UCP RDMA Write over x. RC Peak Message Rate at 16 B Peak BW at 8 KB Higher is better! • Small impact of Put optimization • More instructions spent on Flush than on Put • Small impact of avoiding TX polling • For short messages • Fewer instructions over x. RC • High impact of avoiding UD progress • Progress is called more often on Flush over x. RC 11/22/2020 CCGRID’ 17 32

Evaluation with UCP RDMA Read • All optimizations have smaller impact on UCP RDMA Read • More time spent over the network • Memory copies/cache effects more important than instruction count • Avoiding progress on the (x)UD interface has a small impact on performance (up to 3. 5% for x. RC) • More instructions spent on Flush than on Get operations 11/22/2020 CCGRID’ 17 33

Discussion and Outlook • We can close the performance gap between UCP and UCT without sacrificing the benefits of the high-level abstractions of UCP • We helped reducing some overhead • Optimizations for RDMA functions • Eliminate overheads of UCP-to-UCT translations • Avoid function pointers from UCP to UCT • Requires major redesign • Optimizations for progress functions • Avoid progress over interfaces that are not in use • Poll queues for multiple requests at a time • A scalability bug: Connection Establishment / Wireup • Current UCP design forces all endpoints to connect, even if never used 11/22/2020 CCGRID’ 17 34

Thank you! Questions? nikela@cslab. ece. ntua. gr, l. oden@fz-juelich. de, balaji@anl. gov Acknowledgement US Department of Energy, Office of Science Joint Laboratory for System Evaluation (JLSE), Argonne National Laboratory IKY fellowships of excellence for postgraduate studies in Greece – SIEMENS program PMRS group, MCS Division, Argonne National Laboratory Gail Pieper 11/22/2020 CCGRID’ 17 35

Backup 11/22/2020 CCGRID’ 17 36

UCP UCX design: Communication Entities Worker ucp_worker Progress Engine Worker uct_worker UCT Progress Engine 11/22/2020 Worker • Core communication entity • Same semantics in UCP/UCT Progress engine • to progress communication (progress) • to order communication operations (fence) • to complete communication operations (flush) CCGRID’ 17 37

UCX design: Communication Entities Worker Endpoint Connection Establishment 11/22/2020 CCGRID’ 17 38

ucp_worker Worker ucp_worker Endpoint s ucp_ep Worker uct_worker Endpoint s uct_ep_mm _posix Endpoint s uct_ep_ib_u d uct_ep_ib_rc 11/22/2020 UCP Worker UCT UCP UCX design: Communication Entities uct_ep_mm _posix uct_ep_ib_u d uct_ep_ib_rc CCGRID’ 17 39

UCT UCP UCX design: Communication Primitives Tag matching Remote Memory Access Atomic Memory Operations • Single function for any message size • Blocking and non-blocking operations Active Messages Remote Memory Access Atomic Memory Operations • Multiple functions for various transfer methods (short, buffered copy, zero-copy) • Functions for each transport depending on the hardware capabilities • Non-blocking operations 11/22/2020 CCGRID’ 17 40

UCX design: Connection Establishment 1. A worker creates a UCP endpoint 2. The worker selects appropriate UCT interfaces • • For remote memory access (RMA) For atomic memory operations (AMO) For active messages (AM) For wireup 3. If an interface is connectionless, the corresponding UCT endpoint is created and connected immediately 4. If an interface is connection-oriented (P 2 P), the corresponding UCT endpoint is created but not connected 5. If connection-oriented interfaces exist, UCP creates a stub endpoint • The stub endpoint uses the UCT endpoint over the wireup interface to send wireup requests for all interfaces • The stub endpoint is destroyed when all UCT endpoints are connected 11/22/2020 CCGRID’ 17 41