Uncovering Performance and Interoperability Issues in the OFED

Uncovering Performance and Interoperability Issues in the OFED Stack March 2008 Dennis Tolstenko Sonoma Workshop Presentation

Agenda ØAgent Overview ØFunctionality ØUses ØDemo 2

RDMA Agent Overview Ø The RDMA Agent is a simple application that resides on a host running OFED. Ø Requires no hardware and is installed via an rpm Ø Runs as a daemon or can be manually started/stopped Ø Uses an out of band control interface which allows centralized management and control of the following: Ø Ø Discovery of RDMA Hosts Establishing RDMA DATA Channels Generating RDMA Reads, Writes, and Sends Gathering Performance results, validating interoperability and detecting errors. 3

RDMA OFA Agent Diagram Client Application TCP/IP Control Channel RDMA Test Agent OFA Software Stack RDMA Host RDMA Data Channel 4

Control Interface Ø The out of band control interface is implemented using a simple protocol named PULP. Ø PULP fills the role of the Application layer. Ø It allows a client to define a set of operations which can be presented to the control component of the DUT. Ø It is a “command-response” protocol. 5

Control Channel Ø Control Channel is the conduit for PULP requests and responses. Ø The Control Channel is instantiated via a TCP/IP connection Ø Multiple control channels can be established for parallel processing. Ø The application supports unsolicited message passing. 6

Agent Discovery Ø RDMA Agents are discoverable via a UDP broadcast messages. Ø The Discover Response contains the following information: Ø Host Information: Ø Host Name and IP address Ø Host OS and Kernel Ø OFED version Ø A list of RDMA Interfaces: Ø IP Address Ø Interface Name Ø Status of the Link Ø Vendor name Ø SW Version Ø HW Version 7

RDMA Traffic Ø The RDMA Agent allows the user to define & execute the following RDMA Operations: Ø RDMA READ Ø RDMA Write Ø RDMA SEND Ø Each RDMA Operation has the following fields: Ø Size – Number of bytes to transfer Ø Count – Number of RDMA operations to send Ø Delay – ms between each operation Ø Features Ø Run RDMA Operations serially (allows for ordering) Ø Run RDMA Operations in parallel on one or more Data Channels Ø Traffic shaping can be achieved through Size, Count and Delay 8

Gathering Results Ø Each RDMA Completions contains: Ø Ø Ø Ø RDMA Operation Type (Send, Read, Write) Total bytes Transferred Number of RDMA executions Time taken to complete Completion Status Error Number Vendor Error Number Ø Calculated values Ø Application Throughput Ø Rate Of Operations 9

Agent Highlights Ø Centralized management and control Ø Fabric Discovery & RDMA Device information Ø Able to define and generate RDMA Read, Write and Sends. Ø Accurate performance measurement 10

RDMA Agent Solutions Ø Simplify interoperability testing Ø Stress test RDMA Devices with the OFED stack Ø Measure true performance at the Application Layer Ø Provide a mechanism to test IB Verbs Ø Active Compliance Testing 11

XANStorm Overview Ø XANStorm is the first application to use the RDMA OFA Agent. Ø Easy to use intuitive GUI Ø Works for both Infini. Band i. WARP Ø Works on Linux or Windows 12

XANStorm Key Features Ø Fabric Discovery and Topology Configuration Ø Interoperability Testing Ø Performance Measuring Ø Stress Testing Ø Automated Regression Testing Ø Clear, Easy to Read Results Ø No Additional Hardware Needed 13

Discovery Ø Discover all nodes on the cluster (IB and i. WARP) Ø Provides a detailed information on each RDMA Host. Ø Create single or multiple connections between two or more nodes in the fabric Ø Manual control available for custom configuration 14

RDMA Operations Ø Define Sequence of RDMA Operations Ø Testing options Ø RDMA Type Ø Data Size Ø # of RDMA Operations to execute Ø timing (traffic shaping) of operations Ø direction of traffic flow between nodes Ø number of connections between nodes 15

Comprehensive Results Ø Results from running the “Performance Seq” Ø Results are gathered & presented in real time 16

Agent and XANStorm Information Ø See an online presentation 24/7 Øwww. lampreynetworks. com/xanstorm_demo Ø Request Demo product Øhttp: //www. lampreynetworks. com/Content/i. W ARP/i. WARP_RDMAgenerator. html Ø LNI sales contacts Ø(603) 868 -8411 Øsales@lampreynetworks. com 17