i SER on Infini Band and SCTP Problem
i. SER on Infini. Band (and SCTP)
Problem Statement • Currently defined IB Storage I/O protocol – SRP (SCSI RDMA Protocol) – SRP does not have a discovery or management protocol – SRP does not have a wide following • The RDMA Consortium voted overwhelmingly to create i. SER instead of porting SRP to IP • Missing the new function of i. SCSI & i. SER – Immediate Data – Unsolicited Data • Version 2 is at Level 0 & has not been updated for 1. 5 years • SCTP is not defined for i. SER
Reason for i. SER over IB or SCTP • Would like to have the same basic Storage Protocol across all RDMA Networks – – Easer to train staff Easer to create bridging products Motivate storage industry into an i. SCSI/i. SER mentality May help the acceptance of i. SCSI/i. SER on IP networks • Desire for a common Discovery and Management protocol across i. SCSI, i. SER/i. WARP, and IB – Want the same Management and discovery process and Software to handle IP networks and IB networks
Similarities i. WARP IB • • Local STags L_Key Remote STags R_Key RDMA Send. SE RDMA Send. Inv. SE (New) RDMA Read/Write Shared RQs (New) ZBTOs ZBVA (New)
Proposed New Logical Structure +-------------------+ | SCSI | +-------------------+ | i. SCSI | DI ------> +-------------------+ | i. SER | +-------------------+ | RDMAP | +------------+------+ | DDP | | +-------+-----+ Infini. Band | | MPA | | (RC) | +-------+ SCTP | | | TCP | | | +-------+------------+ Example of i. SCSI/i. SER Layering in Full Feature Mode
Clarify the Term i. WARP • Update the i. SER Draft – Use term i. WARP to mean either TCP or SCTP implementations – Use the term i. WARP/TCP to mean i. WARP over a TCP/IP base – Use the term i. WARP/SCTP to mean i. WARP over an SCTP base
Clarify the Term RDMAP • Update the i. SER Draft: – Use the term RDMAP to mean any RDMA protocol over i. WARP, Infini. Band, or any other carrier of RDMA Protocols – Use the term RDMAP/i. WARP to mean an implementation using i. WARP – Use the term RDMAP/IB to mean the implementation using Infini. Band – Etc.
Things to be addressed for i. SER on IB or SCTP 1. Defining, Addressing and Discovery of IB Storage Nodes 2. Handling of Login (SCTP or IB) 3. Selection of one path to storage vrs others 4. Handling older IB networks – (Network equipment with Pre 1. 2 Architecture)
I. Addressing • Background: IB has IP addressing – Part of IP-over-IB (IPo. IB) – Proposal for Mapping Port to IB Service. ID • IETF IPS WG should validate that: – i. SCSI and i. SER Discovery and Management can operate with IB via IPo. IB – If validated, may not even require normative changes to draft • IBTA (Infini. Band Trade Assoc. ) standardize Port to Service. ID mapping
II. Login • SCTP and IB need a way to send the i. SCSI Login PDUs • SCTP and IB need a way to transit to full i. SER mode • IETF IPS WG discussion needed to ascertain the best way to do this – i. SCSI assumes that TCP/IP streaming is used – But i. SER does not care, as long as it can transit into Full RDMA mode – i. SER Spec needs language to permit this • No need to define details, just language to permit • Leave details up to implementations – May have examples in Appendix, or separate informational drafts
III. Path Selection • A target could have several types of portal groups – i. SCSI, i. SER/TCP, i. SER/SCTP, IB, … – Some Host Systems may prefer one type vrs others • Can leave this completely up to implementation – Therefore not an IETF IPS issue (except informational) – For IB let IBTA standardize connection approach • Preference for direct Endport connection • Preference for i. SER Gateway vrs IPo. IB And/Or • Can add TPG type information to: – Send. Targets, SLP, i. SNS – Would be an IETF IPS issue
IV. Handling older RDMA Networks • May be an IETF IPS Workgroup issue Or • May be out of scope as a compatibility Hack However: • Some applications have requested to have these features • VA Based TO • Explicit Invalidates only • Toleration Language and Hello Flags permit both
Reference • http: //www. haifa. il. ibm. com/satran/ips/ i. SER-in-an-IB-network-V 9. pdf
Backups
I. Defining, Addressing and Discovery • IB nodes are addressed via a GID (Global ID) • With IP-over-IB (IPo. IB) all nodes have Normal IP addresses • IP Addresses are converted to GIDs via ARP – Returned like MAC Address • Therefore, Send. Targets, SLP and i. SNS can continue to function in the same way • Send. Targets, SLP, and i. SNS can all use normal TCP/IP via IPo. IB
II. Handling of Login • i. SCSI Login depends on the value of Max. Recv. Data. Segment. Length = 8192 • i. SCSI Login & Login Reply is basically a half duplex process • IB (and SCTP) can send Login PDUs to Target with <= 8 K data – IB Node will work with RC connections using “RDMA Sends” – No issue of Flow Control (it is half duplex) & Expecting buffer can be queued Max 8192 + i. SCSI header – Transition to i. SER mode is not something special in IB • Therefore, words are being proposed for the Login to be done in IB with Sends (or normal SCTP messages) – i. SCSI Login PDUs remain unchanged
III. Selection of Paths to Storage • In an IB environment it is useful to have a way to select an IB Storage Endpoint in preference to – An IB to: an i. SCSI or i. SER/i. WARP Gateway, or – An i. SCSI TCP IPo. IB Gateway to IP Network • And a way to select an IB to: i. SCSI or i. SER/i. WARP Gateway in preference to – An i. SCSI TCP IPo. IB Gateway to IP Network • This is done via IB defined connection process – Being address in the IBTA – Not an IETF issue
IV. Handling Older IB Networks (ZBTO vrs VABTOs) • Some IB Networks will not support ZBTO – They require a VA (VABTOs) • By using a previously reserved bit in the Hello/Hello. Reply message Initiators can request VABTOs – Can treat the Actual STag and VABTO as a Virtual STag (96 bits instead of 32 bits) in i. SER Headers (only)
IV. Handling Older IB Networks (Missing Auto HW Invalidate) • Some IB Network Nodes can not issue Send. Inv. SE type messages – Can just get by with Send. SE type message – i. SER requires Initiator side invalidates • Some IB Networks Nodes can not receive Send. Inv. SE and then Automatically Invalidate STags – Initiator tells Target by using previously reserved bit in Hello Message
- Slides: 19