Quick Start Guide Virtual Port Channel v PC

Quick Start Guide Virtual Port Channel (v. PC) Architecture & Solutions Group US Public Sector Advanced Services Mark Stinnette, CCIE Data Center #39151 Date 11 June 2013 Version 1. 6. 2 © 2013 Cisco and/or its affiliates. All rights reserved. 1

This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center technologies with end-to-end configurations for several commonly deployed architectures. This presentation will provide end-to-end configurations mapped directly to commonly deployed data center architecture topologies. In this cookbook style; quick start guide; configurations are broken down in an animated step by step process to a complete end-toend good clean configuration based on Cisco best practices and strong recommendations. Each QSG will contain set the stage content, technology component definitions, recommended best practices, and more importantly different scenario data center topologies mapped directly to complete end-to-end configurations. This QSG is geared for network engineers, network operators, and data center architects to allow them to quickly and effectively deploy these technologies in their data center infrastructure based on proven commonly deployed designs. © 2013 Cisco and/or its affiliates. All rights reserved. 2

© 2013 Cisco and/or its affiliates. All rights reserved. 3

Double-Sided v. PC © 2013 Cisco and/or its affiliates. All rights reserved. 4

© 2013 Cisco and/or its affiliates. All rights reserved. 5

© 2013 Cisco and/or its affiliates. All rights reserved. 6

v. PC Peer-Keepalive Link v. PC Peer-Link Dedicated Layer 3 Infrastructure v. PC Peer Device v. PC Member Port v. PC Domain Orphan Port v. PC VLAN : : VLAN(s) carried over the v. PC peer-link and used to communicate via a v. PC; As soon as a VLAN is defined on v. PC peer-link it becomes a v. PC VLAN non-v. PC VLAN : : VLAN(s) that is not part of any v. PC and not present on the v. PC Peer-Link © 2013 Cisco and/or its affiliates. All rights reserved. 7

v. PC is a virtualization technology that presents paired or two Nexus devices as a unique Layer 2 logical node to the access layer devices or endpoints. v. PC belongs to Multichassis Ether. Channel [MCEC] family of technology. A virtual port channel (v. PC) allows links that are physically connected to two different Cisco Nexus 7000 or 5000 Series devices to appear as a single port channel to a third device. The third device can be a switch, server, firewall, load balancer or any other networking device that supports link aggregation technology. v. PC provides the following technical benefits: • Eliminates Spanning Tree Protocol (STP) blocked ports • Uses all available uplink bandwidth; Layer 2 hashing algorithm • Allows dual-homed servers to operate in active-active mode • Provides fast convergence upon link or device failure • Offers dual active/active FHRP (default gateways) for servers • Each peer device in the v. PC domain runs its own control plane, and both devices work independently Using v. PC; you gain immediate operational and architectural advantages: • Simplifies network design • Build highly resilient and robust Layer 2 network • Enables seamless virtual machine mobility and server high-availability clusters • Scales available Layer 2 bandwidth, increasing bisectional bandwidth • Grows the size of the Layer 2 network • v. PC feature is included in the base NX-OS software license v. PC also leverages native split horizon/loop management provided by port-channeling technology; meaning a packet entering a port-channel cannot immediately exit that same port-channel. © 2013 Cisco and/or its affiliates. All rights reserved. 8

© 2013 Cisco and/or its affiliates. All rights reserved. 9

Feature Benefit Overview v. PC auto-recovery (reload restore) Increase High. Availability (1) Provides a backup mechanism in case a v. PC peer-link failure followed by a v. PC primary peer device failure; (2) Both v. PC peer devices reload or DC power outage; but only one v. PC peer comes up - this allows one v. PC device to assume STP / v. PC primary role and bring up all local v. PCs in case (auto-recovery reload-delay) v. PC Peer-Gateway Service Continuity Allows a v. PC switch to act as the active gateway for packets addressed to the peer router MAC (ie. NAS) v. PC orphan-ports suspend Increase High. Availability When v. PC peer-links go down, v. PC secondary shuts down all the v. PC member ports as well as orphan ports. It avoids single attached devices like FW, LB or NIC teamed devices when isolated during v. PC peer-link failure v. PC ARP SYNC Improve Convergence Time Improve Convergence for Layer 3 flows after v. PC peer-link is UP or recovers from a failure v. PC Peer-Switch Improve Convergence Time Virtualize both v. PC peer devices so they appear as a unique STP root bridge v. PC Role & System Priority Service Continuity Manually set v. PC system priority to ensure v. PC peer devices are the primary devices on LACP. Manually set the v. PC role as primary and secondary – deterministic v. PC Peer-keepalive Increase High. Availability Option 1 : : use SUP mgmt int on dedicated OOB Option 2 : : use separate L 3 Po in dedicated VRF v. PC Delay Restore Service Continuity Delays v. PCs member links from bring up on the recovering v. PC peer device. This allows for the Layer 3 routing protocols to converge before allowing any traffic on v. PC member links; resulting in a more graceful restoration and zero packet loss during the recovery phase. (This feature is enabled by default – 30 seconds) © 2013 Cisco and/or its affiliates. All rights reserved. 10

Option 1 Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKAL VRF) Use separate line cards & 1 Gig ports are enough, else you burn 10 Gig interfaces Option 2 Use Mgmt 0 interfaces off Supervisors to dedicated routable OOB network + use management VRF Peer-Keepalive traffic along with management traffic Option 3 As a last resort, route the peer-keepalive traffic over the Layer 3 infrastructure + use default VRF © 2013 Cisco and/or its affiliates. All rights reserved. 11

Option 1 Use Mgmt 0 interfaces to dedicated routable OOB network + use management VRF Peer-Keepalive traffic along with management traffic Option 2 (Nexus 5000 with L 3 module) Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKAL VRF) Use separate interfaces & will burn 10 Gig interfaces Option 2 (Nexus 5000 without L 3 module) Dedicated link(s) in a separate Layer 2 Port-Channel , have the peer-keepalive peer across to the SVI’s, manually prune those VLANs off the peer-link making those VLANs (non-v. PC VLANs), only trunk the peer-keepalive VLAN across this Layer 2 Port-Channel Due to ISSU checks via the show spanning issue-impact, ISSU will fail, workaround is to disable STP on this dedicated Layer 2 Port-Channel via the spanning-tree port type edge trunk command, assuming you have the global command spanning-tree port type edge bpdufilter default enabled Will burn 10 Gig interfaces © 2013 Cisco and/or its affiliates. All rights reserved. 12

feature lacp vlan 1 – 200 vrf context PKAL 7 K-1 7 K-2 Po 1 interface port-channel 1 vrf member PKAL ip address […. ]/30 1/1 2/1 3/1 4/1 Po 2 interface e 1/1 , e 2/1 channel-group 1 mode active vrf context PKAL interface port-channel 1 vrf member PKAL ip address […. ]/30 interface e 1/1 , e 2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport mode trunk interface e 3/1 , e 4/1 channel-group 2 force mode active 5 K-1 5 K-2 Create a dedicated VRF for the v. PC peer-keepalive link (best practice) Step 1 : : turn on LACP feature Step 2 : : define your vlans Step 3 : : build peer-keepalive Step 4 : : build L 2 port channel for peer-link © 2013 Cisco and/or its affiliates. All rights reserved. Building a v. PC peer-link, follow these guidelines : : (1) Must have Peer-keepalive link up first; ensure the peerlink member ports are 10 Gig interfaces (2) Use a minimum of two 10 Gig ports (M 1 up to 8 member ports & F 1/F 2 up to 16 member ports) (3) Use at least two different line cards to increase high availability of peer-link 13

(Optional Config) – when using v. PC peer-switch in a ‘hybrid’ environment use the spanning-tree pseudo-information to load balance VLANs across the 2 peer devices feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default no spanning-tree loopguard default 7 K-1 7 K-2 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default no spanning-tree loopguard default spanning-tree vlan 1 -200 priority 0 spanning-tree pseudo-information vlan 1 -200 root priority 4096 vlan 1 -100 designated priority 8192 vlan 101 -200 designated priority 16384 spanning-tree vlan 1 -200 priority 0 spanning-tree pseudo-information vlan 1 -200 root priority 4096 vlan 1 -100 designated priority 16384 vlan 101 -200 designated priority 8192 vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination […. ] source […. ] vrf PKAL peer-switch peer-gateway auto-recovery reload-delay restore 30 ip arp synchronize vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination […. ] source […. ] vrf PKAL peer-switch peer-gateway auto-recovery reload-delay restore 30 ip arp synchronize Hard set the Nexus 7 K on the left v. PC role primary and Nexus 7 K on the right v. PC role secondary (deterministic) Make the Nexus 7 Ks control LACP establishment for all port-channels; (lowest) vpc domain id + system priority Setup the peer-keepalive; use the correct VRF accordingly © 2013 Cisco and/or its affiliates. All rights reserved. 5 K-1 5 K-2 Step 1 : : turn on vpc feature Step 2 : : configure spanning-tree defaults Step 3 : : configure spanning-tree vlan root priorities Step 4 : : configure vpc domain (per best practices) Enable peer-switch; when activated both v. PC peer devices must have the same STP priority set for all v. PC VLANs – making them appear as a unique STP root bridge Enable peer-gateway, auto-recovery, delay restore, and ip arp synchronize (per best practice) … see Strong Recommendations & Key Notes sections 14

feature lacp feature vpc vlan 1 – 200 7 K-1 7 K-2 vrf context PKAL interface port-channel 1 vrf member PKAL ip address […. ]/30 peer-link interface e 1/1 , e 2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type network vpc peer-link 5 K-1 5 K-2 interface e 3/1 , e 4/1 channel-group 2 force mode active Step 1 : : enable v. PC peer-link on the L 2 port channel Always perform VLAN pruning on v. PC peer-link with the allowed list of v. PC VLANs; v. PC VLANs must also be pruned on the v. PC member port s as well © 2013 Cisco and/or its affiliates. All rights reserved. Bridge Assurance is enabled by default when configuring v. PC peer-link (spanning-tree port type network); Do NOT disable it on the v. PC peer-link 15

7 K-2 7 K-1 peer-keepalive link Use Mgmt 0 interfaces to dedicated routable OOB network + use management VRF (configured during initial device setup); includes Peer-Keepalive traffic along with management traffic feature lacp vlan 1 – 200 vrf context management ip route 0. 0/0 […. ] interface mgmt 0 ip address […. ]/24 interface port-channel 1 switchport mode trunk interface e 1/1 - 2 channel-group 2 force mode active © 2013 Cisco and/or its affiliates. All rights reserved. 1/1 1/2 mgmt 0 5 K-1 5 K-2 OOB Step 1 : : turn on LACP feature Step 2 : : define your vlans Step 3 : : build L 2 port channel for peer-link vrf context management ip route 0. 0/0 […. ] interface mgmt 0 ip address […. ]/24 interface port-channel 1 switchport mode trunk interface e 1/1 - 2 channel-group 2 force mode active 16

Always use a different domain ID in a double-sided v. PC topology; once configured, both peer devices use the v. PC domain ID to automatically assign a unique v. PC system MAC address; which is used as part of the LACP protocol vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination […. ] source […. ] vrf management peer-switch peer-gateway auto-recovery reload-delay restore 30 ip arp synchronize 7 K-2 7 K-1 vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination […. ] source […. ] vrf management peer-switch peer-gateway auto-recovery reload-delay restore 30 ip arp synchronize feature lacp feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default no spanning-tree loopguard default vpc domain 10 role priority 1 system-priority 8096 peer-keepalive destination […. ] source […. ] vrf management auto-recovery reload-delay restore 30 ip arp synchronize © 2013 Cisco and/or its affiliates. All rights reserved. 5 K-1 5 K-2 OOB Manually set v. PC system priority to ensure v. PC peer devices are the primary devices on LACP at the aggregation layer or not the primary devices on LACP at the access layer Step 1 : : turn on vpc feature Step 2 : : configure spanning-tree defaults Step 3 : : configure vpc domain (per best practices) vpc domain 10 role priority 2 system-priority 8096 peer-keepalive destination […. ] source […. ] vrf management auto-recovery reload-delay restore 30 ip arp synchronize 17

7 K-2 7 K-1 feature lacp feature vpc vlan 1 – 200 peer-link interface port-channel 1 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type network vpc peer-link 5 K-1 interface port-channel 1 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type network vpc peer-link 5 K-2 OOB interface e 1/1 - 2 channel-group 2 force mode active Step 1 : : enable v. PC peer-link on the L 2 port channel Always perform VLAN pruning on v. PC peer-link with the allowed list of v. PC VLANs; v. PC VLANs must also be pruned on the v. PC member port s as well © 2013 Cisco and/or its affiliates. All rights reserved. Bridge Assurance is enabled by default when configuring v. PC peer-link (spanning-tree port type network); Do NOT disable it on the v. PC peer-link 18

Configure v. PC member port as spanning-tree port type normal Keep Spanning Tree protocol root function on the aggregation layer of the network; For each v. PC peer device, configure root guard on ports connected to access devices interface port-channel 10 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal spanning-tree guard root vpc 10 7 K-2 7 K-1 interface e 1/13 , e 2/13 channel-group 10 force mode active interface port-channel 10 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal spanning-tree guard root vpc 10 interface e 1/13 , e 2/13 channel-group 10 force mode active 1/13 2/13 port-channel load-balance src-dst ip-l 4 port-vlan v. PC 10 interface port-channel 10 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal vpc 10 interface e 1/9 , e 1/10 channel-group 10 force mode active port-channel load-balance src-dst ip-l 4 port-vlan 1/9 1/10 5 K-1 1/10 5 K-2 Step 1 : : enable v. PC on the member ports Step 2 : : enable spanning-tree port configurations Step 3 : : change port channel load balancing method The configuration of the v. PC member port must match on both v. PC peer devices. If there is a inconsistency, a VLAN or the entire port channel may suspend (depending on type-1 or type-2 consistency check for the v. PC member port). Use the same v. PC ID as the port channel ID for ease of configuration, monitoring, and troubleshooting © 2013 Cisco and/or its affiliates. All rights reserved. 1/9 interface port-channel 10 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal vpc 10 interface e 1/9 , e 1/10 channel-group 10 force mode active port-channel load-balance src-dst ip-l 4 port-vlan Use source-destination, IP, L 4 port and VLAN as fields for the port channel load balancing hashing algorithm; this improves fair usage of all member ports forming in the port channel 19

interface port-channel 20 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal spanning-tree port guard root vpc 20 7 K-2 7 K-1 interface e 3/13 channel-group 20 force mode active 3/13 3/14 3/13 interface port-channel 30 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type edge trunk vpc 30 interface port-channel 20 switchport mode trunk switchport trunk allowed vlan 1 -200 3/14 interface port-channel 30 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type edge trunk vpc 30 v. PC 20 interface e 3/14 channel-group 30 force mode active interface port-channel 20 switchport mode trunk switchport trunk allowed vlan 1 -200 spanning-tree port type normal spanning-tree port guard root vpc 20 1/25 1/26 v. PC 30 interface e 3/14 channel-group 30 force mode active Step 1 : : enable v. PC on the member ports + enable spanning-tree port configurations accordingly interface e 1/25 , e 1/26 channel-group 20 force mode active © 2013 Cisco and/or its affiliates. All rights reserved. 20

feature lacp feature fex Notice in the 5 k/2 k Ev. PC topology you DON’T need the v. PC command under the port channel towards the server fex 100 pinning max-links 1 fex 199 pinning max-links 1 7 K-2 7 K-1 interface port-channel 100 switchport mode fex-fabric vpc 100 fex associate 100 interface port-channel 199 switchport mode fex-fabric vpc 199 fex associate 199 interface e 1/28 channel-group 100 interface e 1/29 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk interface e 100/1/1 , e 199/1/1 channel-group 1000 force mode active © 2013 Cisco and/or its affiliates. All rights reserved. feature lacp feature fex v. PC 10 5 K-1 5 K-2 1/28 1/29 FEX 199 interface e 1/29 channel-group 199 1/28 FEX 100/1/1 1/29 interface e 1/28 channel-group 100 v. PC 199 v. PC 100 Po 1000 interface port-channel 199 switchport mode fex-fabric vpc 199 fex associate 199/1/1 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk interface e 100/1/1 , e 199/1/1 channel-group 1000 force mode active 21

install feature-set fex Default VDC Only feature lacp feature-set fex Straight-Through Topology (only supported topology between 7 k & 2 k FEX) 7 K-2 7 K-1 install feature-set fex feature lacp feature-set fex 199 pinning max-links 1 interface port-channel 199 switchport mode fex-fabric fex associate 199 5/28 6/28 interface e 5/28, e 6/28 switchport mode fex-fabric fex associate 199 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk vpc 1000 5/28 6/28 Po 199 FEX 199 199/1/1 v. PC 1000 interface e 199/1/1 channel-group 1000 force mode active Default VDC Only interface e 5/28, e 6/28 switchport mode fex-fabric fex associate 199 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk vpc 1000 interface e 199/1/1 channel-group 1000 force mode active Notice in the 7 k/2 k Straight-through topology you need the v. PC command under the port channel towards the server FET is an optical transceiver that provides a highly cost-effective solution for connecting FEX to its parent switch (7 k, 5 k, 6 k). Note that FET can only be used to connect Fabric links between the Fabric Extender and the parent switch; FET-10 G must be connected to another FET-10 G) © 2013 Cisco and/or its affiliates. All rights reserved. 22

feature interface-vlan feature hsrp interface port-channel 80 switchport mode trunk switchport trunk allowed vlan 100, 200 spanning-tree port type edge trunk vpc 80 interface e 6/13 channel-group 80 force mode active 7 K-2 7 K-1 interface e 6/13 channel-group 80 force mode active 6/13 v. PC 80 interface vlan 200 ip address 20. 20. 5/24 no ip redirect hsrp 200 preempt priority 110 ip 20. 20. 254 0/0 See VMDC Architecture for more virtual firewall configuration use cases and best practices 0/1 ASA-5585 -X interface vlan 200 ip address 20. 20. 6/24 no ip redirect hsrp 200 preempt ip 20. 20. 254 ip route 10. 10. 10. 0/24 20. 20. 20. 1 Subnet 10. 10. 0 /24 is serviced by the ASA in this example interface port-channel 80 switchport mode trunk switchport trunk allowed vlan 100, 200 spanning-tree port type edge trunk vpc 80 interface Gigabit. Ethernet 0/0, Ge 0/1 channel-group 80 mode active no nameif no secruity-level no ip address interface port-channel 80 port-channel load-balance vlan-src-dst-ip no nameif no secruity-level no ip address © 2013 Cisco and/or its affiliates. All rights reserved. interface port-channel 80. 100 vlan 100 nameif inside secruity-level 99 ip address 10. 10. 1 255. 0 standby 10. 10. 2 interface port-channel 80. 200 vlan 200 nameif outside secruity-level 1 ip address 20. 20. 1 255. 0 standby 20. 20. 2 route outside 0. 0 20. 20. 254 23

Separate Layer 3 (routed traffic) and Layer 2 (bridged traffic) infrastructure. Use dedicated Layer 3 point-to-point link between the v. PC peer devices for backup path to core Use a dedicated Layer 2 port-channel trunk for non-v. PC VLAN and create dedicated VLAN/SVI to established a Layer 3 relationship (note those VLANS are not on the peer-link – manually pruned off) CAN’T Dynamically route over a v. PC – road mapped in version 7. x Firewalls attached in a v. PC; use static routing 1. ASA static route to HSRP on Nexus 2. Nexus static route to ASA VIP © 2013 Cisco and/or its affiliates. All rights reserved. Firewalls attached in a VRF sandwich; separate v. PC attachment 24

featue lacp feature ospf feature interface-vlan feature hsrp vlan 1 – 200 7 K-1 interface loopback 0 ip address […. ]/32 3/32 7 K-2 1/32 2/32 3/32 router ospf 1 router-id […. ] log-adjacency-changes detail auto-cost reference-bandwidth 100 Gbps v. PC 10 interface port-channel 5 ip address […. ]/30 ip router ospf 1 area 0. 0 ip ospf network point-to-point interface e 1/32, e 2/32 channel-group 5 force mode active interface vlan 100 ip address [10. 10. 2]/24 no ip redirects ip router ospf 1 area 0. 0. 0. 10 ip ospf passive-interface hsrp 100 preempt priority 110 ip [10. 10. 1] © 2013 Cisco and/or its affiliates. All rights reserved. interface loopback 0 ip address […. ]/32 router ospf 1 router-id […. ] log-adjacency-changes detail auto-cost reference-bandwidth 100 Gbps interface port-channel 5 ip address […. ]/30 ip router ospf 1 area 0. 0 ip ospf network point-to-point interface e 3/32 ip address […. ]/30 ip router ospf 1 area 0. 0 ip ospf network point-to-point vlan 1 – 200 interface e 1/32, e 2/32 channel-group 5 force mode active 5 K-1 5 K-2 Use dedicated Layer 3 point-to-point link between the v. PC peer devices for backup path to core Define the SVI associated with HSRP as passive routing interface in order to avoid forming routing adjacency over v. PC peer-link Define v. PC primary peer device as the active HSRP instance and v. PC secondary peer device as the standby HSRP (from control plane standpoint) for ease of operations Disable ip redirect (no ip redirect) on the interface VLAN where HSRP is configured interface e 3/32 ip address […. ]/30 ip router ospf 1 area 0. 0 ip ospf network point-to-point interface vlan 100 ip address [10. 10. 3]/24 no ip redirects ip router ospf 1 area 0. 0. 0. 10 ip ospf passive-interface hsrp 100 preempt ip [10. 10. 1] 25

Failure 1 : : Peer-Keepalive fails Failure 2 : : Peer-Link fails on Aggregation Failure 3 : : Peer-Link fails on Access Layer Failure 4 : : Peer-Keepalive fails + Peer-Link fails (Split Brain) nothing happens – no traffic loss Role Primary Role Operational Secondary Primary v. PC member ports are shut down and all the v. PC VLAN interfaces (SVIs) are shut down; meaning no more L 3 advertisements – all this happens on the secondary v. PC peer device When PKL link fails and PL fails ( in this order ) , you have a dual active situation , while both links are down , the primary v. PC peer device remains primary and your secondary v. PC device becomes operational primary In a v. PC environment only operational primary switch behaves as STP root and processes BPDU and your secondary switch do not process BPDUs ( this is regardless of whichever switch is configured as STP root ) Existing flows continue to be forwarded as before the failure; but new flows learning are impaired and uncertain forwarding (or broken state) for new flows will be observed. So when the links comes back up, the originally primary switch will see that, there is an existing operational primary switch (originally secondary) which is behaving like an STP root and processing BPDUs If the originally primary switch tries to reclaim the primary role at this point, that would mean more convergence time while operational root role is being switched, hence we do not try to reclaim the vpc primary (and acting STP root ) role back to avoid more convergence times © 2013 Cisco and/or its affiliates. All rights reserved. Role Primary Role Secondary Black hole traffic to single attached devices connected to v. PC Peer device with secondary role 26

Failure 1 : : Peer-Link fails on FEX Parent Switch at Access Layer Failure 2 : : Single FEX Fails or Power Loss Role Primary Role Secondary No traffic loss – Only the v. PC members are shut down northbound facing the Aggregation Layer and the NIF interfaces are lost on the FEX facing the secondary v. PC peer device – all traffic will be forward from both FEXs to the primary v. PC peer device Single Attached hosts connected to the FEX are unaffected Role Primary Role Secondary 5 K Parent switches have lost communication to the failed FEX; resulting all host traffic will forward out the secondary FEX. Minimal to no traffic loss when hosts are dual attached in LACP; Active / Standby NIC teaming will failover to the secondary FEX Black hole traffic to devices connected to single FEX Active / Standby NIC teaming will failover to the secondary FEX © 2013 Cisco and/or its affiliates. All rights reserved. 27

Without Bridge Assurance prevents a spanning-tree domain from failing in an “open” state. When a port configured for Bridge Assurance stops receiving BPDU’s, the port transitions into a “blocking” state as opposed to remaining in a “forwarding” state. This “closed” state reduces the likelihood for mis-configured devices from creating STP loops. ‘spanning-tree bridge assurance ’ is enabled by default for all ‘network’ port types Specifies bi-directional transmission of BPDUs on all ports of type “network”. With Bridge Assurance Protects against unidirectional links and peer switch software issues Provides IGP like hello-dead timer behavior for Spanning Tree In all versions of NX-OS, available in IOS on the Catalyst 6500 beginning 12. 2(33) SXI Recommended in STP topologies Not recommend in v. PC topologies; only on the peer-link (default) © 2013 Cisco and/or its affiliates. All rights reserved. 28

There are two types of consistency checks : Type-1 : : Puts peer device or interface into a suspended state to prevent invalid packet forwarding behavior. With v. PC Graceful Consistency check, suspension occurs only on the secondary peer device. Type-2 : : Peer device or Interface still forward traffic; however they are subject to undesired packet forwarding behavior. Type 1 and Type 2 consistency check apply both for global configuration and for v. PC interface configuration. show vpc consistency-parameters global – (displays global type-1 consistency parameters) Parameter Name Value Spanning Tree Protocol (STP) mode RPVST or MST STP Enable/Disable state per VLAN Yes / No STP region configuration for MST Region name, revision, instance to VLAN mapping STP global settings Bridge Assurance settings Port type settings Loop guard settings BPDU filter settings MST simulate PVST enable / disable show vpc consistency-parameters interface port-channel [id] – (displays interface type-1 consistency parameters) Parameter Name Value Port channel LACP mode ON, ACTIVE, PASSIVE Link speed & duplex per port channel Speed in mpbs & Half / Full duplex Switchport mode per port channel Trunk / Access, native VLAN STP interface settings Port type setting Loop Guard Root Guard MST Simulate PVS Enable / Disable MTU per port channel Maximum transmission Unit (MTU) value © 2013 Cisco and/or its affiliates. All rights reserved. 29

© 2013 Cisco and/or its affiliates. All rights reserved. 30

If any of the v. PC Type-2 parameters listed in the table below are not configured identically on both v. PC peer devices, the inconsistent configuration cause undesirable behavior in the traffic flow Type-2 consistency check parameters Parameter Name Value MAC aging timers MAC aging timer for a particular VLAN should be the same on both v. PC peer devices Static MAC entries in a particular VLAN should be applied on both v. PC peer devices VLAN interface (switch virtual interface [SVI]) Each peer device must have a VLAN interface configured for the same VLAN on both ends, and this VLAN interface must be in the same operational state ACL Configuration and parameters ACL configuration should be identical on both v. PC peer devices Qo. S Configuration and parameters Qo. S configuration should be identical on both v. PC peer devices STP interface settings BPDU filter, Link type (auto, point-to-point, shared), Cost, Port-priority, STP interface setting should be identical on both v. PC peer devices VLAN Database You must create all VLANs on both the primary and secondary v. PC peer devices, or the VLAN will be suspended. Those VLANs configured on only one peer device do not pass traffic using the v. PC or v. PC peer-link Port security NAC, Dynamic ARP Inspection, IP source guard, port security must be identical on both v. PC peer devices Cisco Trust. Sec configuration should be identical on both v. PC peer devices DHCP snooping configuration should be identical on both v. PC peer devices IGMP snooping configuration should be identical on both v. PC peer devices HSRP configuration should be identical on both v. PC peer devices PIM configuration should be identical on both v. PC peer devices GLBPconfiguration should be identical on both v. PC peer devices All routing protocol configurations Routing configuration should be consistent on both v. PC peer devices © 2013 Cisco and/or its affiliates. All rights reserved. 31

• Always use different domain ID in double-sided v. PC topology • Operations perspective, define v. PC primary on the left Nexus and v. PC secondary on the right Nexus (role priority) • When configuring large number of VLANs in a v. PC environment, use the range command (vlan x-z) vs. individually configuring one at a time • Create a dedicated VRF for the v. PC peer-keepalive link (ie. vrf context PKAL) • When building a v. PC peer-link, follow these guidelines • Must have Peer-keepalive link up first; ensure the peer-link member ports are 10 Gig interfaces • Use a minimum of two 10 Gig ports (M 1 up to 8 member ports & F 1/F 2 up to 16 member ports) • Use at least two different line cards to increase high availability of peer-link • Use dedicated mode 10 Gig ports with M 1 32 line card vs. shared mode ports • Split v. PC and non-v. PC VLANs on different interswitch port channels • Don’t insert any device between v. PC peers; a peer-link is a point-to-point link • Any v. PC VLAN allowed on the v. PC member port MUST be allowed on the v. PC peer-link • Always perform VLAN pruning on v. PC peer-link with allowed list of v. PC VLAN; v. PC VLAN must have been pruned on the v. PC member port previously • If the M 1 32 is used for both the v. PC peer-link and L 3 uplinks to L 3 Core, use v. PC object tracking feature • When building a v. PC member port, follow these guidelines • The configuration of the v. PC member port must match on both v. PC peer devices • If there is a inconsistency, a VLAN or the entire port channel may suspend (depending on type-1 or type-2 consistency check for the v. PC member port) • Use the same v. PC ID as port channel ID for ease of configuration, monitoring, and troubleshooting • With the M 1 Series line card : : there can be up to 8 active ports bundled – resulting a 16 -way port channel to be built for the whole v. PC • With the F 1/F 2 Series line card : : there can be up to 16 active ports bundled – resulting a 32 -way port channel to be built for the whole v. PC • Do not mix different port types (M 1, F 2) in the same v. PC member port; this is not allowed by the software • Both sides of the v. PC member ports must be of the same port type © 2013 Cisco and/or its affiliates. All rights reserved. 32

• The v. PC peer-keepalive link carries periodic heartbeat (UDP 3200) between v. PC peer devices. It is used at the boot up of the v. PC systems to guarantee both peer devices are up before forming v. PC domain and also when v. PC peer-link fails to down state; in the latter case, v. PC peer-keepalive link is leveraged to detect split brain scenario (both v. PC peer devices are active–active) [when v. PC peer-link is down, there is no more real time synchronization between the 2 peer devices so v. PC systems must react to this active-active situation; this is done by shutting down v. PC member ports on secondary peer device]. • The v. PC peer-keepalive link is a pre-requisite for the v. PC domain to form initially (ie. prior to the v. PC peer-link configuration + if peer-link is initial up before peer-keepalive is up) • v. PC has 3 timers; hold-timeout (default 3 sec), timeout (default 5 s), hello interval (default 1 s). The hold-timeout starts once the v. PC peer link goes to a down state; during this time period the secondary v. PC peer will ignore any peer-keepalive hello messages. During the timeout period, the secondary v. PC peer device will look for v. PC peer-keepalive hello messages from the primary v. PC peer device. If a single hello is received, the secondary v. PC peer concludes that there must be a dual active scenario and therefore will disable all its v. PC member ports (that is, all port-channels that carry the keyword vpc). Command line configuration to modify v. PC timers is (under v. PC domain configuration context): peerkeepalive destination ipaddress [source ipaddress | hold-timeout secs | interval msecs {timeout secs}] The default values are ok in most situations. v. PC peer link is down ! Keepalive Hold Timeout Keepalive Timeout © 2013 Cisco and/or its affiliates. All rights reserved. 33

• Always enable v. PC peer-gateway in the v. PC domain (on both peer devices), even if there is no end device using this feature (devices that don’t perform standard ARP request for their default IP gateway), there is no side effect enabling it • (Corner Case) always use v. PC peer-gateway exclude-vlan when a transit VLAN (over v. PC peer-link) is used in the v. PC domain, this is applicable only for mixed chassis mode (M 1/F 1) with peer-link on F 1 ints; note only static routing supported • Always enable v. PC ARP sync on both v. PC peers; performs a bulk ARP sync, improves convergence time for L 3 flows • Always enable v. PC delay restore on both v. PC peer devices and tune the timer according based on the network profile • Always enable v. PC graceful type-1 check on both v. PC peer devices; graceful consistency-check; (enabled by default) • Always enable v. PC auto-recovery on both v. PC peer devices • Always enable v. PC auto-recovery reload-delay on both v. PC peer devices (note the v. PC auto-recovery reload-delay deprecates the previous feature called v. PC reload restore) • Use v. PC orphan port suspend when single-attached devices connected to a v. PC domain need to be disconnected from the network when v. PC peer-link fails • Always use a different domain ID in a double-sided v. PC topology; once configured, both peer devices use the v. PC domain ID to automatically assign a unique v. PC system MAC address; which is used as part of the LACP protocol • v. PC role is non-preemptive so v. PC operational role is the most relevant of the information per table below © 2013 Cisco and/or its affiliates. All rights reserved. 34

• • • • Use LACP protocol when connecting access devices to v. PC domain (channel-group [x] mode active Use LACP when available for graceful failover and misconfiguration protection LACP mode active on both sides of the port channel If access device does not support LACP, use manual bundling (channel-group [x] mode on) If the downstream access switch is a Cisco Nexus device, enable LACP graceful-convergence (its on by default) If the downstream access switch is NOT is a Cisco Nexus device, disable LACP graceful-convergence Use source-destination, IP, L 4 port and VLAN as fields for the port channel load balancing hashing algorithm; this improves fair usage of all member ports forming in the port channel When possible, always dual-attach access devices to a v. PC domain using a port channel When connecting a single-attached access device to a v. PC domain using a v. PC VLAN, always connect it to the v. PC primary peer device; reason is when if the v. PC peer-link fails down any single attached device connected to the secondary peer device (and using v. PC VLAN) will become completely isolated with the rest of the network Single Attached Recommendations (descending order of priority): • Connect access device to an intermediate switch which is dual-attached to a v. PC domain • Connect single-attached device to a v. PC domain using non-v. PC VLAN (must also create an inter-switch link between the 2 peer devices to transport non-v. PC VLAN • Connect single-attached device to a v. PC domain using v. PC VLAN and leveraging v. PC peer-link In a double-sided v. PC topology, all interconnect links between the 2 v. PC domains MUST belong to the same v. PC ID; all links form a unique v. PC (on both sides of the 2 v. PC domains) LACP port suspend : : By default, LACP sets a port to the suspended state if it does not receive an LACP PDU from the peer (ie a server or host). In some cases, although this feature helps in preventing loops created due to misconfigurations, it can cause servers to fail to boot up because they require LACP to logically bring up the port. You can put a port into an individual state by using the lacp suspend-individual command. On the Nexus 5000 this feature is disabled (no lacp suspend-individual) for servers connecting via LACP; on the Nexus 7000 this feature is enabled by default (lacp suspend-individual) © 2013 Cisco and/or its affiliates. All rights reserved. 35

• Recommended Spanning Tree Protocol Configuration with v. PC • Spanning Tree protocol must remain enabled for all VLANs (even if all access devices are v. PC attached to the v. PC domain); Do NOT disable spanning-tree protocol • Use MST with v. PC if you need to build a large L 2 domain; Plan ahead to avoid future configuration changes that can trigger v. PC type-1 consistency failure • Implement consistent STP mode in the same L 2 domain, ensure that all switch in your L 2 domain are running with Rapid-PVST+ (default) or MST to avoid slow Spanning Tree convergence (30 seconds or more) • Perform VLAN pruning on v. PC member ports to reduce internal resource consumption • Keep Spanning Tree protocol root function on the aggregation layer of the network (aggregation v. PC domain) • For each v. PC peer device, configure root guard on ports connected to access devices • Bridge Assurance is enabled by default when configuring v. PC peer-link (spanning-tree port type network); Do NOT disable it on the v. PC peer-link • It is not necessary to enable Bridge Assurance on the v. PC (members ports in the v. PC) – configure v. PC member port as spanning-tree port type normal • Configure port fast (edge or edge trunk port type) on the host facing interfaces to avoid slow Spanning Tree protocol convergence (30 seconds or more) when port transitions to an up state • Configure BPDU guard on host facing interfaces to block any BPDU sent from the host (access switch port receiving the BPDU will be put in errdisable mode) – enable BPDU guard globally • Always define the v. PC domain as the STP root for all VLANs in that domain (configure aggregation v. PC peer devices as the STP root primary and STP root secondary) – enforce this rule with root guard on v. PC peer device ports connected to another L 2 switch • IF the v. PC peer-switch is activated, both v. PC peer devices MUST have the SAME spanning tree configuration (same priority for all v. PC VLANs) – recommendation to activate v. PC peer-switch in the environment • Do not enable Loop guard on v. PC (disabled by default) • When using v. PC peer-switch in a hybrid environment use the spanning-tree pseudo-information to load balance VLANs across the 2 peer devices • Enable UDLD in normal mode on v. PC peer-link and v. PC member ports © 2013 Cisco and/or its affiliates. All rights reserved. 36

• Layer 3 and v. PC Guidelines and Recommendations • Use separate Layer 3 link(s) to connect to L 3 devices (like a router or firewall in routed mode) to a v. PC domain; use individual Layer 3 links for routed traffic and a separate Layer 2 port-channel for bridged traffic if both routed and bridged traffic are required • Always build a Layer 3 backup routed path for the v. PC domain in order to increase network resilience and availability; use an OSPF point-to-point adjacency (or equivalent L 3 protocol) between the 2 v. PC peer devices to establish a L 3 backup path to the core in case of uplink failure • Do NOT use a Layer 2 v. PC to attach Layer 3 devices to a v. PC domain unless the Layer 3 device can statically route to the HSRP address configured on the v. PC peer devices • You can’t dynamically route over a v. PC • Layer 3 backup routing path options (descending order of preference) • Use dedicated Layer 3 point-to-point link between the v. PC peer devices for backup path to core • Use a dedicated Layer 2 port-channel trunk for non-v. PC VLAN and create dedicated VLAN/SVI to established a Layer 3 relationship (note those VLANS are not on the peer-link) • HSRP / VRRP Guidelines and Recommendations • When running HSRP/VRRP in active-active mode (data plane standpoint), aggressive timers can be relaxed; use the default HSRP/VRRP timers • Define the SVI associated with HSRP/VRRP as passive routing interface in order to avoid forming routing adjacency over v. PC peer-link • Define v. PC primary peer device as the active HSRP/VRRP instance and v. PC secondary peer device as the standby HSRP/VRRP (from control plane standpoint) for ease of operations • Disable ip redirect (no ip redirect) on the interface VLAN where HSRP/VRRP is configured. • Do NOT use HSRP/VRRP object tracking in a v. PC domain © 2013 Cisco and/or its affiliates. All rights reserved. 37

• Recommendations for Multilayer v. PC for DCI Solution • Use different v. PC domain-id for each v. PC domain (DC 1: v. PC domain for aggregation, v. PC for DCI. DC 2: v. PC domain for aggregation, v. PC for DCI) • For each data center, interconnect the aggregation v. PC domain to the DCI v. PC domain using av. PC (double-sided topology) • Interconnect the 2 data centers using a v. PC (v. PC between DCI v. PC domain in site 1 and site 2) • Enable BPDU filter on the v. PC used for DCI (under the port-channel configuration, activate the following command: spanning-tree bpdufilter enable) to avoid BPDU propagation • Configure the v. PC used for DCI as spanning-tree port type edge (i. e port fast) to fasten port state forwarding mode when port is operationally up • Remember by default v. PC peer-link runs in spanning-tree port type network i. e bridge assurance is activated on the link • Configure root guard on aggregation v. PC domain (more exactly on v. PC between this v. PC domain and DCI v. PC domain). STP root must remain on aggregation v. PC domain on each side of the data center • No loop must exist outside the v. PC domains. • Do not use Layer 3 peering between data centers (in other words, there is no Layer 3 over v. PC). • Do not use bridge assurance for interconnect v. PC (DCI v. PC) – use spanning-tree port type edge trunk • Use M 1 ports for DCI v. PC if flows between the 2 data centers need to be encrypted using 802. 1 ae MACsec © 2013 Cisco and/or its affiliates. All rights reserved. 38

• Best Practices for Network Services / Appliances and v. PC • Configure v. PC to the inside and outside interfaces for ASA firewalls – use spanning-tree port type edge trunk • If needed, use multiple VRF instances for the inside interfaces – intra data center nets (see VMDC architecture) • Be aware of the following Layer 3 over v. PC design caveat • Use dedicated Layer 2 port-channel for the service appliances state and keepalive VLANs (recommend don’t use the v. PC peer-link) • Recommended the ASA port channel hashing algorithm and the Nexus v. PC hashing algorithm are the same • Connect ASA in routed mode to a v. PC – must use static routing • ASA static route to HSRP on Nexus • Nexus static route to ASA VIP • If Connected ASA in routed mode and use dynamic routing • Single attach ASA to v. PC domain • Create separate non-v. PC interswitch link • Peer with non-v. PC VLAN/SVIs SLB attached via v. PC SLB attached via Po with orphan port suspend Firewall attached via v. PC & Static Routes Firewall attached via non-v. PC Po & Dynamic Routing Bandwidth reduced during certain failure scenarios © 2013 Cisco and/or its affiliates. All rights reserved. Bandwidth maintained during certain failure scenarios 39

External (public) Great External Resource Nexus v. PC best practices design guide http: //www. cisco. com/en/US/docs/switches/datacenter/sw/design/vpc_best_practices_design_guide. pdf Nexus 7000/6000/5000 Configuration Guides http: //www. cisco. com/en/US/products/ps 9402/products_installation_and_configuration_guides_list. html http: //www. cisco. com/en/US/products/ps 9670/products_installation_and_configuration_guides_list. html http: //www. cisco. com/en/US/partner/products/ps 12806/products_installation_and_configuration_guides_list. html Nexus 5000 Enhanced v. PC Configuration Guide http: //www. cisco. com/en/US/docs/switches/datacenter/nexus 5000/sw/mkt_ops_guides/513_n 1_1/n 5 k_enhanced_vpc. html © 2013 Cisco and/or its affiliates. All rights reserved. 40