OFED Status and Process November 2007 Tziporet Koren

OFED - Status and Process November 2007 Tziporet Koren http: //openfabrics. org/

Agenda Ø OFED 1. 3 Status Ø OFED Development Process Ø OFED Future Discussion http: //openfabrics. org/ Mellanox Technologies 2

Open. Fabrics Software Stack http: //openfabrics. org/ Mellanox Technologies 3

OFED – Update Since Sonoma Ø OFED 1. 2 was released on 22 -Jun ØKernel base: 2. 6. 20 Ø OFED 1. 2. 5 was released on 16 -Aug ØKernel base 2. 6. 22 ØConnect. X support Ø OFED 1. 2. 5. 3 was released on 14 -Nov ØRDS in GA for Oracle 11 g RAC Ø OFED 1. 3 – in middle of release phase ØKernel base 2. 6. 24 http: //openfabrics. org/ Mellanox Technologies 4

OFED 1. 3 Components OFA Add on Ø HCA/NIC Drivers development Ø Mellanox, QLogic, IBM, Chelsio, Net. Effect Ø Bonding module Ø Core: verbs, mad, SMA, CMA, SA cache Ø IPo. IB Ø SDP Ø SRP, i. SER Ø SRP Target Ø RDS Ø Qlogic_VNIC Ø UDAPL Ø OSM Ø Diagnostic tools Ø NFS-RDMA – still open http: //openfabrics. org/ Ø MPI Components Ø MVAPICH Ø Open MPI Ø MVAPICH 2 Ø Benchmark tests Tested Ø Proprietary MPIs: Intel, HP, with Scali Ø Proprietary SMs: Cisco, Voltaire, Qlogic Mellanox Technologies 5

Main New Features - General Ø Base kernel: 2. 6. 24 Ø Install ØMinimize integration effort into OS distribution ØUser level split into separately-installable RPMs ØBased on feedback from Novell and Redhat Ø Package ØSources arrangement for the end user ØAdded the make-dist script Ø Reduce compilation warnings of backports http: //openfabrics. org/ Mellanox Technologies 6

General Features – Cont. Ø Qo. S Ø OSM, CM & CMA, ULPs: SRP, SDP, i. SER, IPo. IB Ø Updated SA cache Ø Integrated with sa_query module Ø Use of SA events (GID up/down) to provide additional synchronization Ø User-space control for dynamic enable/disable and update Ø Extended RC (XRC) Support Ø New verbs to enable XRC Ø New modules Ø NES (Net. Effect driver) Ø SRP Target: Based on Mellanox IBGD SRPT and interfaces with SCST scsi target middle level driver http: //openfabrics. org/ Mellanox Technologies 7

Main New Features – ULPs Ø IPo. IB Ø NAPI Ø Stateless offloads – for Datagram mode Ø Small message performance improvements Ø RDS: RDMA with Zero copy (API version 3) Ø SDP: Ø Asynch IO Ø Send Zero Copy Ø u. DAPL 2. 0 Ø In addition to u. DAPL 1. 0 http: //openfabrics. org/ Mellanox Technologies 8

Main New Features – Management Ø Open. SM Ø Quality of Service manager Ø Qo. S implementation of the IBTA annex Ø Qo. S policy parser and support for SA (PR, MPR) Ø Scalability/stability improvements Ø Performance improvements of the routing algorithms (min-hops, up/down, lash) Ø Optimized port and switch tables update policy Ø “Native” daemon mode Ø Performance manager Ø Fetches and stores performance counters Ø Open. SM integrated, work in any Open. SM state (standby, disabled) Ø Plug-in interface Ø Diagnostic tools enhancements Ø IBSIM – IB fabric simulator (new) Ø Emulates user_mad layer Ø useful for Open. SM and management tools testing http: //openfabrics. org/ Mellanox Technologies 9

Main New Features - MPI Ø MVAPICH 1. 0 Ø Ø Ø Ø Asynchronous progress Flexible message coalescing APM support Enhanced mpirun_rsh for scalable launching UD-based design Optimized and tuned collectives Support for Connect. X Support for QLogic/PSM Ø Open MPI 1. 2. 5 Ø Connect. X support Ø MVAPICH 2 1. 0 Ø Ø Ø Ø Message Coalescing Hot-spot avoidance Application-initiated systems-level checkpointing APM support Multi-rail support for i. WARP On-demand connection management for i. WARP and u. DAPL RDMA Read Blocking http: //openfabrics. org/ Mellanox Technologies 10

OFED 1. 3 Release Status Ø Features freeze: Oct 10, 2007 Ø Alpha: Oct 15, 2007– kernel base: 2. 6. 23 Ø Beta: Nov 20, 2007– kernel base: 2. 6. 24 -rc 2 Ø RC 1: Dec 4, 2007 Ø RC 2: Dec 18, 2007 Ø RC 3: Jan 8, 2008 Ø RC 4: Jan 22, 2008 Ø Release: Jan 29, 2008 Note: The delay mainly caused by the move to kernel 2. 6. 24 http: //openfabrics. org/ Mellanox Technologies 11

OFED 1. 3 OS Matrix Ø kernel. org: kernel 2. 6. 23 and 2. 6. 24 Ø Novell Ø SLES 10 SP 1 (up 1) Ø Redhat Ø RHEL 4 (up 4 and up 5) Ø RHEL 5 (up 1) Ø Free distros Ø Ø Partial support only (no QA) Open Su. SE 10. 2 Fedore C 6 Ubuntu 6. 06 (with RPM package installed) Ø Open: OSVs plans to pick up OFED 1. 3, or even 1. 2. 5 http: //openfabrics. org/ Mellanox Technologies 12

Agenda Ø OFED 1. 3 Status Ø OFED Development Process Ø OFED Future Discussion http: //openfabrics. org/ Mellanox Technologies 13

OFED – Development Process Ø Start – release definition Ø Ø Agree on release features Agree on OS matrix and kernel base Decide on target schedule Release definition is placed on the Wiki https: //wiki. openfabrics. org/tikiindex. php? page=OFED+Development+Wiki Ø Note Ø In all OFED releases there were changes in the above that caused release slips Ø Tracking meetings Ø Bi-weekly meetings during the development phase Ø Weekly meetings on the release phase Ø Meeting summaries are placed on the Wiki https: //wiki. openfabrics. org/tiki-index. php? page=Meeting+Minutes Ø Bug tracking in bugzilla Ø https: //bugs. openfabrics. org/ http: //openfabrics. org/ Mellanox Technologies 14

Kernel Development Ø Have one kernel git tree that is maintained by Vlad Ø git: //git. openfabrics. org/ofed_1_3/linux-2. 6. git Ø Cloned from Linux git tree Ø Add-on components (e. g. , SDP) are added to the tree Ø New features and bug fixes are added as patches under the fixes directory Ø Patch convention name: <module>_<number>_<text>. patch (e. g. , ipoib_0010_cq_coalescing. patch) Ø Supporting distros and previous kernels: Ø Backport directory for each kernel OFED can compile on Ø Add-on directory for headers from Linux Ø The make-dist script applies first the “fixes” patches and then the appropriate backport patches http: //openfabrics. org/ Mellanox Technologies 15

User Space Development Ø Each maintainer has a separate git tree on the server Ø Examples git: //git. openfabrics. org/~sashak/management. git: //git. openfabrics. org/~shefty/libibcm. git Ø Each package has its own RPM SPEC file Ø All packages should also be released to the ofa download site Ø http: //www. openfabrics. org/downloads. htm Ø Approaching a release – a branch with the release name is opened Ø Userspace component fixes must be committed -no patching scheme is used http: //openfabrics. org/ Mellanox Technologies 16

External Packages Development Ø External packages are provided as source RPMs Ø Each external package has a directory on the ofa server ØExample – the bonding package: /home/monis/public_html/ofed_1_3/ Ølatest. txt file points to the package that should be used ØEx. : ib-bonding-0. 9. 0 -19. src. rpm http: //openfabrics. org/ Mellanox Technologies 17

OFED Daily Build Ø An OFED package is created everyday at 6 am PST Ø http: //www. openfabrics. org/builds/ofed-1. 3/ Ø This package can be used by all to test the daily changes Ø This package includes Ø Source RPMs for kernel, all user space components, external modules Ø OFED scripts (install, etc. ) Ø OFED documents Ø Development phase: the build uses the master branch of all git trees Ø Release phase: the build is based on the release branch http: //openfabrics. org/ Mellanox Technologies 18

OFED Release Phase Ø Starts after the alpha release Ø Close tracking of components status Ø Alpha, beta and RC are done according to the package maturity Ø Bugs severity tracking which bugs must be fixed for the release Ø Update of all documents http: //openfabrics. org/ Mellanox Technologies 19

Agenda Ø OFED 1. 3 Status Ø OFED Development Process Ø OFED Future Discussion http: //openfabrics. org/ Mellanox Technologies 20

OFED Goals Ø Enterprise Working Group within Open Fabrics Alliance (OFA) Ø OFED has become a brand name in the industry today Ø Collaborative effort to test & release OFA software Ø Broader test participation Ø Components – Kernel & User Space Ø Interoperability Ø Add-on components for vendors to differentiate above OFA Reduces deployment complexity and cost http: //openfabrics. org/ Mellanox Technologies 21

Reasons to Continue OFED Ø Industry synchronization point Ø Set of kernel modules, libraries and applications that are tested together Ø Comprehensive tests by many vendors Ø OSVs rely on our qualification work Ø Faster new feature development and deployment Ø New drivers for new IHV devices Ø ULP updates for new ISV software packages, e. g. , Oracle 11 g Ø Enables support of new features in existing OSes Ø Ex. : Bonding to work on Redhat EL 4 up 5, SLES 10, etc http: //openfabrics. org/ Mellanox Technologies 22

OFED-based and OS Distro-based Ø OFED-based and OS Distro-based distribution of OFA software – one or both Ø Originally, some people intended OFED to be a short-term distribution until mainstream Linux distributions caught up and were able to distribute IB/i. WARP support Ø Today there is synergy between OFED and OS vendors as they rely on the OFED quall and support benefits delivered by commercial IB/i. WARP suppliers Ø Are OSV’s ready to take more quall and support responsibilities at this time? Ø Discussion … http: //openfabrics. org/ Mellanox Technologies 23