MODULE 6 IP SAN AND FCOE EMC Proven

MODULE – 6 IP SAN AND FCOE EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 1

Module 6: IP SAN and FCo. E Upon completion of this module, you should be able to: • Describe IP SAN protocols, components, and topology • Describe FCo. E protocol, components, and topology EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 2

Module 6: IP SAN and FCo. E Lesson 1: IP SAN During this lesson the following topics are covered: • Drivers for IP SAN • IP SAN Protocols: i. SCSI and FCIP • Components, topologies, and protocol stack for i. SCSI and FCIP EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 3

Drivers for IP SAN • Traditional SAN enables the transfer of block I/O over Fibre. Channel and provides high performance and scalability 4 These advantages of FC SAN come with the additional cost of buying FC components, such as FC HBA and switches. • Organizations typically have an existing Internet Protocol (IP)-based infrastructure, which could be leveraged (use to maximum advantage) for storage networking. 4 Advancements in technology have enabled IP to be used for transporting block I/O over the IP network. EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 4

Drivers for IP SAN (contd. ) • IP SAN transports block-level data over IP network • IP is being positioned as a storage networking option because: 4 Existing network infrastructure can be leveraged 4 Reduced cost compared to investing in new FC SAN hardware and software 4 Many long-distance disaster recovery solutions already leverage IP-based network 4 Many robust and mature security options are available for IP network EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 5

IP SAN Protocol: i. SCSI • IP based protocol that is used to connect host and storage 4 A transport layer protocol that describes how Small Computer System Interface (SCSI) packets should be transported over a TCP/IP network. • Carries block-level data over IP-based network • Widely adopted for connecting servers to storage because it is relatively inexpensive and easy to implement EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 6

i. SCSI Implementation i. SCSI encapsulates SCSI commands and data into an IP packet and transports them using TCP/IP i. SCSI Gateway Server Storage Array IP FC Port i. SCSI HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. i. SCSI Port i. SCSI Implementation Storage Array Module 6: IP SAN and FCo. E 7

Components of i. SCSI initiator i. SCSI target • Hardware or software residing on a computer that handles communications with an i. SCSI array (storage devices) • Software-based i. SCSI initiator, such as Microsoft's free i. SCSI initiator - uses the existing network card (NIC) • Hardware-based i. SCSI initiator – such as i. SCSI HBA (Host Bus Adapter) • Does not consume CPU resources to handle i. SCSI commands • The i. SCSI-enabled device to which you will be storing data • Storage array with i. SCSI port • i. SCSI gateway – enables communication with FC storage array • Common deployment scenarios for an i. SCSI target include: Storage array, disk drives EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. IP network • Interconnected Ethernet switches and/or routers Module 6: IP SAN and FCo. E

i. SCSI Host Connectivity Options Standard NIC with software i. SCSI initiator • Simple, least expensive • NIC provides network interface • Software initiator provides i. SCSI functionality • Requires host CPU cycles for i. SCSI and TCP/IP processing • Encapsulation of SCSI into IP packets & decapsulation are carried out by host CPU • Additional overhead on the host CPU • Heavy I/O load -> host CPU become bottleneck TCP Offload Engine (TOE) NIC with software i. SCSI initiator • Moves TCP processing load off the host CPU onto the NIC card • Software initiator provides i. SCSI functionality • Requires host CPU cycles for i. SCSI processing EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. i. SCSI HBA • Offloads both i. SCSI and TCP/IP processing from host CPU • Simplest option for boot from SAN via i. SCSI Module 6: IP SAN and FCo. E 9

i. SCSI Topologies (1) Native i. SCSI Connectivity (2) Bridged i. SCSI Connectivity EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. (3) Combining FCP and Native i. SCSI Connectivity Module 6: IP SAN and FCo. E 10

i. SCSI Topologies: (1) Native i. SCSI • i. SCSI initiators are either directly attached to storage array or connected through IP network 4 No FC component • Storage array has i. SCSI port • Each i. SCSI port on the array is configured with an IP address and port number. IP Server i. SCSI HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Storage Array i. SCSI Port Module 6: IP SAN and FCo. E 11

i. SCSI Topologies: (1) Native i. SCSI (contd. ) • The array has one or more i. SCSI ports configured with an IP • • address and connected to a standard Ethernet switch. After an initiator is logged on to the network, it can access the available LUNs on the storage array. A single array port can service multiple hosts or initiators as long as the array port can handle the amount of storage traffic that the hosts generate. IP Server Initiator e. g: i. SCSI HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Storage Array i. SCSI Port Module 6: IP SAN and FCo. E 12

i. SCSI Topologies: (2) Bridged i. SCSI • i. SCSI gateway is used to enable communication between i. SCSI • host and FC storage i. SCSI gateway works as bridge between FC and IP network 4 Converts IP packets to FC frames and vice versa • i. SCSI initiator is configured with gateway’s IP address as its • target i. SCSI gateway is configured as FC initiator to storage array i. SCSI Gateway IP Server i. SCSI HBA FC SAN Server Storage Array FC HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FC Port Module 6: IP SAN and FCo. E 13

i. SCSI Topologies: (2) Bridged i. SCSI (contd. ) • In the figure illustrates an i. SCSI host connectivity to an FC storage • • • array In this case, the array does not have any i. SCSI ports An external device, called a gateway or a multiprotocol router, must be used to facilitate the communication between the i. SCSI host and FC storage Bridge devices contain both FC and Ethernet ports to facilitate the communication between the FC andi. SCSIIPGateway environments IP Server i. SCSI HBA FC SAN Server Storage Array FC HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FC Port Module 6: IP SAN and FCo. E 14

i. SCSI Topologies: (3) Combining FC and Native i. SCSI Connectivity • Array provides both FC and i. SCSI ports 4 Enable i. SCSI and FC connectivity in the same environment 4 No bridge devices needed i. SCSI Port IP Server i. SCSI HBA FC SAN Server FC HBA EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Storage Array FC Port Module 6: IP SAN and FCo. E 15

i. SCSI Protocol Stack • SCSI is the command protocol that works at the application layer of the Open System Interconnection (OSI) model • Initiators and targets use SCSI commands and responses to talk to each other • Commands, data, and status messages are encapsulated into TCP/IP and transmitted across the network between the initiators and targets • i. SCSI is the session-layer protocol that initiates a reliable session between devices that recognize SCSI commands and TCP/IP • Responsible for handling login, authentication, target discovery, and session management Figure: i. SCSI protocol layers and the encapsulation order of the SCSI commands for their delivery through a physical carrier OSI Model i. SCSI Initiator i. SCSI Target Layer 7 Application SCSI Commands and Data SCSI Layer 5 Session i. SCSI Login and Discovery i. SCSI Layer 4 Transport TCP Windows and Segments TCP Layer 3 Network IP Packets IP Layer 2 Data Link Ethernet Frames Ethernet Interconnect Ethernet IP EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. TCP i. SCSI Data Module 6: IP SAN and FCo. E 16

i. SCSI Protocol Stack (contd. ) • TCP is used with i. SCSI at the transport layer to provide reliable transmission • TCP controls message flow, windowing, error recovery, and retransmission. • Rely upon the network layer of the OSI model to provide global addressing and connectivity. • The Layer 2 protocols at the data link layer of this model enable node-tonode communication through a physical network. Figure: i. SCSI protocol layers and the encapsulation order of the SCSI commands for their delivery through a physical carrier OSI Model i. SCSI Initiator i. SCSI Target Layer 7 Application SCSI Commands and Data SCSI Layer 5 Session i. SCSI Login and Discovery i. SCSI Layer 4 Transport TCP Windows and Segments TCP Layer 3 Network IP Packets IP Layer 2 Data Link Ethernet Frames Ethernet Interconnect Ethernet IP EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. TCP i. SCSI Data Module 6: IP SAN and FCo. E 17

i. SCSI Protocol Stack (contd. ) OSI Model i. SCSI Initiator i. SCSI Target Layer 7 Application SCSI Commands and Data SCSI Layer 5 Session i. SCSI Login and Discovery i. SCSI Layer 4 Transport TCP Windows and Segments TCP Layer 3 Network IP Packets IP Layer 2 Data Link Ethernet Frames Ethernet Interconnect Ethernet IP TCP EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. i. SCSI Data Module 6: IP SAN and FCo. E 18

i. SCSI Discovery • For i. SCSI communication, initiator must discover location and • name of target on a network i. SCSI discovery takes place in two ways: Send. Targets discovery • Initiator is manually configured with the target’s network portal • Initiator issues Send. Targets command; target responds with required parameters EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Internet Storage Name Service (i. SNS) • Initiators and targets register themselves with i. SNS server • Initiator can query i. SNS server for a list of available targets Module 6: IP SAN and FCo. E 19

i. SCSI Name • i. SCSI name is a unique i. SCSI identifier that is used to identify • initiators and targets within an i. SCSI network Two common types of i. SCSI names are: 4 IQN: i. SCSI Qualified Name 8 To use IQN, the company must own a registered domain name 8 A date is included in the name to avoid potential conflicts caused by the transfer of domain names 8 iqn. 2008 -02. com. example: optional_string § Example: iqn. 1992 -05. com. emc: apm 000339013630000 -10 8 The optional_string can be the name of the host, serial number, asset number or any other storage identifier 4 EUI: Extended Unique Identifier 8 Use the WWN (World Wide Name) 8 eui. 0300732 A 32598 D 26 EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 20

IP SAN Protocol: FCIP is an IP-based protocol that is used to connect distributed FC SAN islands • Combines advantages of Fibre Channel and IP • Enables distributed FC SAN islands to be interconnected over the existing IP-based networks • Creates virtual FC links over existing IP network that is used to transport FC data between different FC SANs • Encapsulates FC frames onto IP packet • Extensively used in disaster recovery implementations in which data is duplicated to the storage located at a remote site EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 21

FCIP Topology • FCIP gateway is connected to each fabric via a standard FC connection • FCIP gateway at one end of the IP network encapsulates the FC frames into IP packets • FCIP gateway at the other end removes the IP wrapper and sends the FC data to the layer 2 fabric • Fabric treats these gateways as layer 2 fabric switches • IP address is assigned to the port on the gateway, which is connected to an IP network. • After the IP connectivity is established, the nodes in the two independent fabrics can communicate with other Servers Server FC SAN FCIP Gateway FC SAN IP Storage Array EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Storage Array Module 6: IP SAN and FCo. E 22

FCIP Protocol Stack Application SCSI Commands, Data, and Status FC Frame FCP (SCSI over FC) FCIP TCP FC to IP Encapsulation IP Physical Media EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. • Applications generate SCSI commands and data • Upper layer protocol SCSI includes the SCSI driver program that executes the read-and-write commands • Fibre. Channel Protocol (FCP) layer enables the FC frames to run natively within a SAN fabric environment • FCIP layer encapsulates the Fibre. Channel frames onto the IP payload and passes them to the TCP layer • TCP and IP are used for transporting the encapsulated information across Ethernet, wireless, or other media that support the TCP/IP traffic Module 6: IP SAN and FCo. E 23

FCIP Frame • Encapsulation of FC frame on to IP packet could cause the IP • • packet to be fragmented When an IP packet is fragmented, the required parts of the header must be copied by all fragments When a TCP packet is segmented, normal TCP operations are responsible for receiving and re-sequencing the data prior to passing it on to the FC processing portion of the device FC Frame SOF FC Header SCSI Data CRC EOF FCIP Encapsulation FCIP Frame IP Header TCP Header EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FCIP Header IP Payload Module 6: IP SAN and FCo. E 24

Module 6: IP SAN and FCo. E Lesson 2: Fibre Channel over Ethernet (FCo. E) During this lesson the following topics are covered: • Drivers for FCo. E • Components of FCo. E network • FCo. E frame mapping • Converged Enhanced Ethernet (CEE) EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 25

Drivers for FCo. E • Data centers have multiple networks to handle various types of I/O • • • traffic 4 Ethernet network for TCP/IP communication: used for client-server communication, data backup, infrastructure management communication, and so on. 4 FC network for FC communication: used for moving block-level data between storage and servers. To support multiple networks, servers in a data center are equipped with multiple redundant physical network interfaces—for example, multiple Ethernet and FC cards/adapters. To enable the communication, different types of networking switches and physical cabling infrastructure are implemented in data centers. The need for two different kinds of physical network infrastructure increases the overall cost and complexity of data center operation. EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 26

Drivers for FCo. E (contd. ) • FCo. E is a protocol that transports FC data over Ethernet network • (Converged Enhanced Ethernet) FCo. E is being positioned as a storage networking option Reduces cost because: Enables • consolidation Enables consolidation (merge) of FC SAN traffic and Ethernet traffic onto a common Ethernet infrastructure • Effective sharing of high -bandwidth links Reduces the number of adapters, switch ports, and cables • Lower capital expenditure EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. and eases data center management • Lower Total Cost of Ownership (TCO) • Potentially lower administration cost, with convergence of LAN and SAN Reduced power and cooling requirement • Reduces power and cooling cost, and floor space Module 6: IP SAN and FCo. E 27

Data Center Infrastructure – Before Using FCo. E Servers • LAN and SAN are two • • different networks. Storage resources are accessible using HBAs and network resources that are accessible using NICs by hosts. In a data center, a server is configured with 2 to 4 NIC cards and redundant HBA cards. Servers Server FC Switches Storage Array EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. IP Switches LAN Storage Array Module 6: IP SAN and FCo. E 28

Data Center Infrastructure – Before Using FCo. E (contd. ) Servers • If the data center has • • hundreds of servers, it would require a large number of adapters, cables, and switches. Complex environment, difficult to manage and scale. The cost of power, cooling, and floor space further adds to the challenge Server FC Switches Storage Array EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. IP Switches LAN Storage Array Module 6: IP SAN and FCo. E 29

Data Center Infrastructure – After Using FCo. E • I/O consolidation with FCo. E using FCo. E Server switches and Converged Network Adapters (CNAs) • CNA replaces both HBAs and NICs in the server and consolidates both the IP and FC traffic • The FCo. E switch passes Fibre Channel traffic to the SAN, and the Ethernet traffic to an attached Ethernet network. Servers Server FCo. E Switches FC Switches Storage Array EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. LAN Storage Array Module 6: IP SAN and FCo. E 30

Data Center Infrastructure – After Using FCo. E • The cable requirements Server from host to FCo. E switches can be reduced, which in turn reduces the cooling costs, management requirements, and the overall operational cost. Servers Server FCo. E Switches FC Switches Storage Array EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. LAN Storage Array Module 6: IP SAN and FCo. E 31

Components of an FCo. E Network (1) Converged Network Adapter (CNA) (2) Cable EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. (3) FCo. E switch Module 6: IP SAN and FCo. E 32

Components of an FCo. E Network: (1) Converged Network Adapter (CNA) • Provides functionality of both – a standard NIC and an FC HBA 4 Eliminates the need to deploy separate adapters and cables for FC and Ethernet communications • Contains separate modules for 10 Gigabit Ethernet, FC, and FCo. E Application Specific Integrated Circuits (ASICs) 4 FCo. E ASIC encapsulates FC frames into Ethernet frames EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 33

Components of an FCo. E Network: (2) Cable • Two options are available for FCo. E cabling 4 Copper based Twinax cable 4 Standard fiber optical cable Twinax Cable Fiber Optical Cable Suitable for shorter distances (up to 10 meters) Can run over longer distances Requires less power and are less expensive than fiber optical cable Relatively more expensive than Twinax cables Uses Small Form Factor Pluggable Plus (SFP+) connector EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 34

Components of an FCo. E Network: (3) FCo. E Switch • Provides both Ethernet and FC switch functionalities • Consists of Fibre Channel Forwarder (FCF), Ethernet bridge, and set of Ethernet ports and FC ports (optional) 4 FCF encapsulates and de-encapsulates FC frames EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FC Port Fibre Channel Forwarder (FCF) Ethernet Bridge Ethernet Port Module 6: IP SAN and FCo. E 35

Components of an FCo. E Network: (3) FCo. E Switch (contd. ) • Forwards frames based on Ethertype 4 If the Ethertype of the frame is FCo. E, the switch recognizes that the frame contains an FC payload and forwards it to the FCF. 8 FC is extracted from the FCo. E frame and transmitted to FC SAN over the FC ports. 4 If the Ethertype is not FCo. E, the switch handles the traffic as usual Ethernet traffic and forwards it over the Ethernet ports. EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FC Port Fibre Channel Forwarder (FCF) Ethernet Bridge Ethernet Port Module 6: IP SAN and FCo. E 36

FCo. E – Frame Structure Normal Ethernet Frame: Ether Type = FCo. E Control Information: Version and Ordered Sets (SOF and EOF) 24 Bytes 16 Bytes FCS EOF Fibre Channel Payload CRC FC Header FCo. E Header Ethernet Header Same as a Physical Fibre Channel Frame 4 Bytes Up to 2112 Bytes (MAC Addresses) + 4 Bytes (802. 10 TAG) EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. 1 Byte(EOF) + 3 Bytes (Padding) Module 6: IP SAN and FCo. E 37

FCo. E Frame Mapping • The FCo. E protocol specification replaces the FC-0 and FC-1 layers of the FC stack with Ethernet • This provides the capability to carry the FC-2 to the FC-4 layer over the Ethernet layer. OSI Stack 7 - Application FCo. E Protocol Stack 6 - Presentation 5 - Session FC Layers 4 - Transport 3 - Network 2 - Data Link 1 - Physical FC - 4 Protocol map FC - 3 Services FC - 2 Framing FCo. E Mapping IEEE 802. 1 q Layers FC Protocol Stack 2 - MAC 1 - Physical EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. FC - 1 Data enc/dec FC - 0 Physical Ethernet Module 6: IP SAN and FCo. E 38

Converged Enhanced Ethernet (CEE) • Conventional Ethernet is lossy in nature, which means that • frames might be dropped or lost during transmission Converged Enhanced Ethernet (CEE) provides lossless Ethernet 4 Eliminates the lossy nature of Ethernet • Lossless Ethernet requires following functionalities: Priority-based flow control (PFC) 2. Enhanced transmission selection (ETS) 3. Congestion notification (CN) 4. Data center bridging exchange protocol(DCBX) 1. EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 39

Lossless Ethernet : (1) Priority-Based Flow Control (PFC) • Traditional FC manages congestion through the use of a link-level, credit-based flow control that guarantees no loss of frames. Typical Ethernet, coupled with TCP/IP, uses a packet drop flow control mechanism. The packet drop flow control is not lossless • PFC provides a link level flow • • control mechanism Creates eight virtual links on a single physical link Uses PAUSE capability of Ethernet for each virtual link EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. • A virtual link can be paused and restarted independently • PAUSE mechanism is based on user priorities or classes of service Module 6: IP SAN and FCo. E 40

Lossless Ethernet : (2) Enhanced Transmission Selection (ETS) • Allocates bandwidth to different traffic classes such as LAN, SAN, • and Inter Process Communication (IPC) Provides available bandwidth to other classes of traffic when a particular class of traffic does not use its allocated bandwidth EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 41

Lossless Ethernet : (3) Congestion Notification (CN) • Provides a mechanism for detecting congestion and notifying the source 4 Enables a switch to send a signal to other ports that need to stop or slow down their transmissions Rate limiting to avoid packet loss FCo. E Switch Host (Node A) Congestion Notification Message Congestion EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Storage Array (Node B) Module 6: IP SAN and FCo. E 42

Lossless Ethernet : (4) Data Center Bridging Exchange Protocol (DCBX) • Enables Converged Enhanced Ethernet (CEE) devices to convey and configure their features with other CEE devices in the network 4 Allows a switch to distribute configuration values to attached adapters • Ensures consistent configuration across network EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 43

Module 6: Summary Key points covered in this module: • IP SAN protocols, their components, and topologies • FCo. E protocol, its components, and topology EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCo. E 44