Chapter 7 Troubleshooting Case Study TINC Garbage Disposal

Chapter 7: Troubleshooting Case Study: TINC Garbage Disposal CCNP TSHOOT: Maintaining and Troubleshooting IP Networks TSHOOT v 7 Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 1

Chapter 7 Objectives § TINC Garbage Disposal Trouble Ticket 1 • Troubleshooting BGP Neighbor Relationships • Troubleshooting Port Security • Troubleshooting VLANs § TINC Garbage Disposal Trouble Ticket 2 • Troubleshooting GW 1’s OSPF Neighbor Relation Problem with Router R 1 • Troubleshooting OSPF Adjacency • Troubleshooting SSH and Telnet • Troubleshooting HSRP Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 2

Chapter 7 Objectives § TINC Garbage Disposal Trouble Ticket 3 • • Troubleshooting Erroneous Routing Information Troubleshooting Multiple Masters within a VRRP Troubleshooting Ether. Channel § TINC Garbage Disposal Trouble Ticket 4 • Troubleshooting FHRPs • The Cisco IOS DHCP Snooping Feature Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 3

TINC Garbage Disposal Scenario Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All

TINC Garbage Disposal Trouble Ticket 1 Chapter 7 © 2007 – 2016, Cisco Systems,

Troubleshooting BGP Neighbor Relationships § BGP is a TCP-based application. § Before BGP neighbor relationship occurs, they must complete a TCP threeway handshake. • If the isn’t a 3 -way handshake there will be no BGP relationship • Possible causes include: no ip connectivity or tcp port 179 being blocked § The BGP open message, includes the sender’s router ID, ASN, hold time, and BGP version, is sent from one neighbor to the other. If there are no configuration conflicts with the aforementioned parameters that are sent from one neighbor to the other, the neighbor relationship is established. • If the are any conflicts the BGP relationship will not occur • Possible cause include: wrong neighbor address (due to the use of loopback to identify the host), wrong number of hops configured for ebgp or incorrect ASN § Next, the neighbors do full exchange of best paths to all known and permitted prefixes. (Filters may prevent a router to send all known prefixes to certain neighbors. ) § After that, BGP neighbors only exchange hello messages unless there is a change; in that case, they send update (reachable and unreachable) messages to each other. § In case of any type of error, BGP neighbors send notification messages to each other, specifying the cause of the conflict. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 7

BGP Relationship States § The neighbor relationship between two BGP routers (speakers) can be in one of the following states: § Idle: The router is searching the IP routing table to find a path to reach the neighbor. § Active: The router is still trying (up to 16 retries) to complete a TCP three-way handshake. § Connect: The router found a route to the neighbor and has completed the TCP three-way handshake. § Open Sent: After completion of the TCP three-way handshake, the BGP open was message sent to the neighbor. § Open Confirm: The router received an open message from the neighbor who has agreed to the parameters for establishing a BGP session. § Established: Peering is established; update and hello messages will follow. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 8

BGP Neighbor troubleshoot § To troubleshoot a case where a BGP neighbor will not get established, use the following commands and techniques: • Use the show ip bgp summary command to display the state of all BGP peers. • Verify IP connectivity to the neighbor’s address using extended ping and traceroute. • Verify the existence of the IP route to the neighbor’s address with the show ip route command. • Discover additional information with the show ip bgp neighbor command. • Verify the BGP configuration with the command show run | section router bgp • Use the debug ip bgp command options to obtain more information and clues about the problem. • For example, debug ip bgp events displays the state transitions for peers. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 9

BGP Neighbor troubleshoot Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights

Troubleshooting Port Security § After you have set the maximum number of secure MAC addresses on a port, secure addresses can be inserted in the address table in one of the following ways: • You can statically configure all secure MAC addresses by using the switchport-security mac-address mac_address interface configuration command. • You can allow the port to dynamically configure secure MAC addresses with the MAC addresses of connected devices (also called sticky learning). switchport-security mac-address sticky • Note: If you enter a write memory or copy running-config startup-config command, port security with sticky MAC addresses saves dynamically learned MAC addresses in the startup-config file, and the port does not have to learn addresses from ingress traffic after boot or a restart. • You can statically configure a number of addresses and allow the rest to be dynamically configured. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 11

Troubleshooting Port Security § When configuring port security violation modes, note the following information: • protect: Drops packets with unknown source addresses until you remove a sufficient number of secure MAC addresses to drop below the maximum value • restrict: Drops packets with unknown source addresses until you remove a sufficient number of secure MAC addresses to drop below the maximum value and causes the security violation counter to increment • shutdown: Puts the interface into the error-disabled state immediately and sends an SNMP trap notification Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 12

Troubleshooting Port Security § If the show mac address-table command does not reveal a specific MAC address (that you expect) in the MAC address table, most likely the switch is blocking all the frames from that source. To troubleshoot this type of situation, use the following guidelines: • Verify the interface status with the show interface status command. The access interface connecting the host to the network must be in up/up state. • Check the MAC table for dynamic and static entries with the show mac addresstable command. You can distinguish entries according to the interface, VLAN, and type. • Check the port security overall status with the show port-security command. You can see on which interfaces port security is actually applied, what kind of violation mode is set for each interface, and check all the relevant counters. • Check the port security status on the interface with the show port-security interface ethernet x/y command. The command reveals some additional information about the port security status of the specific interface. • Check the port security configuration with the show running-config and show startup-config commands. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 13

Troubleshooting Port Security Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights

Troubleshooting VLANs § Use the following checklist during VLAN troubleshooting: • Verify the status of all the relevant interfaces with the show interface command. The interface must be in the up/up state. • Check the VLAN database with the show vlan command. This command enables you to verify which VLANs exist and to view port-to-VLAN mapping. Trunks are not listed because they do not belong to any particular VLAN. • Check trunk interfaces with the show interfaces trunk command the show interfaces switchport command. Using these commands, you can discover all interfaces that are configured as trunks and find out per trunk information about the configured trunk mode, encapsulation type, native VLAN, and allowed VLANs. • Check the MAC table for dynamic and static entries in a particular VLAN with the show mac-address-table command. This is the main command to verify Layer 2 forwarding. It shows you the MAC addresses that are learned by the switch and their corresponding port and VLAN associations. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 15

Troubleshooting Native VLANs § The native VLAN is used to carry untagged traffic across an 802. 1 Q trunk. The default native VLAN is set to VLAN 1, but can be changed on each trunk interface. • Cisco Discovery Protocol (CDP) monitors the native VLANs and displays a notification if a mismatch is detected. Mismatch messages are displayed every minute and they look like the following: • %CDP-4 -NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on Gigabit. Ethernet 1/0/25 (100), with sw 1 Gigabit. Ethernet 1/0/25 (300). • STP is also able to detect native VLAN mismatches and can block the affected VLAN on the interface. This will result in notifications such as: • %SPANTREE-2 -RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 300 on Gigabit. Ethernet 1/0/25 VLAN 100. • %SPANTREE-2 -BLOCK_PVID_PEER: Blocking Gigabit. Ethernet 1/0/25 on VLAN 0300. Inconsistent peer vlan. • %SPANTREE-2 -BLOCK_PVID_LOCAL: Blocking Gigabit. Ethernet 1/0/25 on VLAN 0100. Inconsistent local vlan. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 16

Troubleshooting Native VLANs § The native VLAN is typically untagged on 802. 1 Q trunks. Which can lead to a security vulnerability due to vlan hooping attack. • This can be prevented with the vlan dot 1 q tag native command, which explicitly tags the native VLAN as well. • For older switches, which do not support this feature, you can remove the native VLAN from the allowed VLAN on that trunk. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 17

TINC Garbage Disposal Trouble Ticket 2 Chapter 7 © 2007 – 2016, Cisco Systems,

Troubleshooting GW 1’s OSPF Neighbor Relation Problem with Router R 1 Chapter 7 ©

Troubleshoot OSPF Adjacency § Missing Neighbor • Verify the interface status with the show interface command. The interface to the neighbor must be in the up/up state. • Check the IP connectivity to the neighbor with the ping command. Ping the IP address of the neighbor and multicast IP address 224. 0. 0. 5, which is the destination address for Hello messages. • Check to see whether any access list is applied to the interface by using the show ip interface command. • Check the OSPF status on the interface with the show ip ospf interface command. The command reveals all the interfaces where OSPF is enabled. • Check whether the interface is configured as passive. You can use show ip ospf interface as well. Active OSPF interfaces should display a remaining time to the next Hello message. • Verify that the router ID of the routers is not the same. If the router receives a Hello packet with the same router ID, the router will ignore this Hello packet. Use the show ip ospf command to verify the OSPF router ID. § Verify that the Hello parameters match between neighbors. Several parameters must match to establish the full adjacency: • OSPF area number: You can verify this using the show ip ospf interface command. • OSPF area type (such as stub or NSSA): Use the show ip ospf command to discover the area type. • Subnet and subnet mask: Check the IP address and the mask with the show ip interface command. • OSPF Hello and Dead timers: The show ip ospf interface command reveals the OSPF timers. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 25

Troubleshooting SSH and Telnet § Common causes of SSH/Telnet problems include the following: • Access lists • If Telnet or SSH connections to a device are refused, check whether access lists are applied to vty lines or interfaces. • Telnet/SSH not enabled on the vty line • By default, all protocols are enabled on the vty lines. You can limit the protocols by using the transport input command. To check the enabled protocols, you can use the show line command. • Authentication • You should check the running configuration to find out the AAA mechanisms that are used. The debug radius and the debug tacacs commands are also useful for troubleshooting RADIUS and TACACS+ protocols. • All vty lines are busy • To check all the connections to the vty lines, use the show line command. To clear some of the vty lines, use the clear line command. • SSH version • Make sure that the correct SSH version and key size are configured. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 26

Troubleshooting SSH and Telnet Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All

Troubleshooting HSRP § The following are some possible configuration errors with respect to HSRP: • Error in HSRP group configuration leads to the duplicate IP address problem. • HSRP virtual IP addresses are configured differently, you will receive appropriate log messages. • Incorrect configuration of the HSRP authentication is also informed through log messages. • HSRP for IPv 4 has Versions 1 and 2. If there is a version mismatch, both routers will become active. This results in duplicate IP addresses, too. • Wrong HSRP group is configured on the peers, both peers become active. This will also manifest as a duplicate IP address problem. • HSRP Versions 1 and 2 are not compatible. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 28

Troubleshooting HSRP § You can solve most of the HSRP configuration problems by checking the output of the show standby command. § In the output, you can view the active IP and the MAC address, timers, the active router, and several other parameters. § HSRP messages are sent to multicast IP address 224. 0. 0. 2 and UDP port 1985 in Version 1 and multicast IP address 224. 0. 0. 10 and UDP port 1985 in Version 2. • These IP addresses and ports need to be permitted by access lists. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 29

TINC Garbage Disposal Trouble Ticket 3 Chapter 7 © 2007 – 2016, Cisco Systems,

Troubleshooting Erroneous Routing Information § Incorrect routing information can be sourced from • Illegitimate devices participating in the process • Legitimate devices sourcing incorrect information § Manual neighbor configuration and neighbor authentication help avoidance of unwanted peers. § If you want to have strict control over peering relations, disable the automatic peer discovery mechanisms. • For example, when the neighbor is configured manually, EIGRP messages are exchanged using unicast packets, and packets are accepted only from manually configured neighbors. This does not hold true for OSPF. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 31

Troubleshooting Erroneous Routing Information § Authentication options for currently used IPv 4 routing protocols: • OSPF: Authentication can be enabled for the whole OSPF area, or on an interfaceby-interface basis. • EIGRP and RIPv 2: Authentication is enabled in interface configuration mode. Cisco EIGRP makes use of a key chain for authentication. • BGP: Neighbor authentication is configured in the router configuration mode for each neighbor individually. It is possible to use the same pre -shared key for all neighbors or to configure a different key for each neighbor Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 32

Troubleshooting Erroneous Routing Information § To exclude segments from routing information exchange, in the router configuration mode make use of the passiveinterface command. § You can also issue the passive-interface default command treat all the interfaces that are configured to participate in the routing process as passive. § Interfaces that are supposed to exchange routing information should be explicitly configured to do so using the no passive-interface command Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 33

Troubleshooting Multiple Masters within a VRRP § When troubleshooting VRRP, consider the following configuration errors: • • • The VRRP group virtual IP address may be configured incorrectly. The VRRP group number may be configured incorrectly. The VRRP group members may have authentication discrepancies. The VRRP group members may have advertisement timer discrepancies. The VRRP messages may be erroneously blocked by ACLs. § The following commands can prove useful in diagnosing VRRP-related issues: • Use the show vrrp brief command to get a concise overview of the VRRP groups and their basic parameters. • Use the show vrrp interface command to view VRRP groups on a specific interface. • Use the debug vrrp all command to display debugging messages for VRRP errors, events, and state transitions. • Use the debug vrrp authentication , debug vrrp error , and debug vrrp state commands to see debug messages specifically related to MD 5 authentication, error conditions, and status transitions. • Use the debug vrrp packets and debug vrrp events commands to view summaryinformation about the sent and the received packets and VRRP events. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 34

Troubleshooting Multiple Masters within a VRRP Chapter 7 © 2007 – 2016, Cisco Systems,

Troubleshooting Ether. Channel § The following characteristics must match among the interfaces that are members of an Ether. Channel bundle: • Interface speed and duplex • Use the show interface command to verify the interface speed and duplex. • Interface trunking mode and related VLANs • Use the show interface command with the switchport option or show running-config interface type number command to verify trunking mode of an interface. • The following must match: mode (access or trunk), VLAN number (for access interfaces), native VLAN, allowed VLANs, and encapsulation (for trunk interfaces). • switchport (Layer 2 interface) or no switchport (Layer 3 interface) • All the physical interfaces must be acting in the same layer Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 37

Troubleshooting Ether. Channel § You can configure the switch interface using one of the following three channel-establishment options: • Manual • No protocol, statically configured channel • PAg. P • Port Aggregation Protocol, a Cisco proprietary link aggregation protocol • LACP • Link Aggregation Control Protocol (standards based) § Only the following combinations of port channel modes make the port channel form successfully: • LACP • Active-active and active-passive • PAg. P • Desirable-desirable and desirable-passive • With no protocol • On-on Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 38

Troubleshooting Ether. Channel § Etherchannel Verification Commands • • show etherchannel summary show etherchannel groupnumber detail show etherchannel load-balance show etherchannel traffic Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 39

TINC Garbage Disposal Trouble Ticket 4 Chapter 7 © 2007 – 2016, Cisco Systems,

Troubleshooting FHRPs § GLBP, in addition to HSRP, is a Cisco proprietary FHRP. § The advantages of GLBP over HSRP are its built-in support for load sharing and that up to four routers can participate in every GLBP group as active forwarders. § The following is a list of common GLBP misconfigurations: • • • Virtual IP address configured incorrectly on one or more of the GLBP group members. GLBP group number discrepancy among GLBP group members. Authentication method or key mismatch among GLBP group members. Preempt option missing or configured on one router only. Access lists or firewalls blocking GLBP messages • (GLBP messages are sent to the IP multicast address 224. 0. 0. 102, and GLBP’s UDP port number is 3222) § The following is a list of useful GLBP diagnostic commands: • show glbp brief • show glbp interface type number [brief] • debug glbp [packets | events | terse | error | all] Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 41

Troubleshooting FHRPs § The following are some significant facts about FHRPs: • HSRP and GLBP do not allow the group virtual IP address to be assigned to either of the routers participating in the group. VRRP, however, does allow you to assign the virtual IP address to one of the routers within the group. The router whose address is used as the virtual IP address will be the Master router for that VRRP group, even if another router has a higher priority. • VRRP is an IETF standard (RFC 5798); this makes VRRP the only FHRP suitable for multivendor environments. • HSRP and GLBP do not have the preempt option enabled by default; VRRP does. If you do not want the preempt option with VRRP, you can disable it. • GLBP’s main feature is that up to four routers belonging to the same group forward traffic and exercise load sharing for a single virtual IP address. • The default Hello and Hold timers for HSRP and GLBP are 3 seconds and 10 seconds; the default Hello and Hold timers for VRRP are 1 second and 3 seconds. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 42

The Cisco IOS DHCP Snooping Feature § DHCP snooping is a Layer 2 security feature that acts like a firewall between untrusted hosts and trusted DHCP servers. § The primary function of the DHCP snooping is to prevent rogue DHCP servers in the network. § DHCP snooping is enabled on switches on a per-VLAN basis. § Interfaces of a LAN switch are configured as trusted or untrusted. • Trusted interfaces allow all types of DHCP messages through, whereas only DISCOVER and REQUEST messages are allowed on untrusted interfaces. • Trusted interfaces are interfaces that connect to a DHCP server or are uplinks toward the DHCP server. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 43

The Cisco IOS DHCP Snooping Feature § With DHCP snooping enabled, the switch also builds a DHCP snooping binding database. • Each entry in the database includes the MAC address of the host, the leased IP address, the lease time, the binding type, the VLAN number, and interface information associated with the host. • The DHCP snooping binding database can be used by other security features such as dynamic ARP inspection (DAI). § DHCP snooping may also be used to limit the rate of DHCP messages (ADDRESS REQUEST). • This option is configured per switch interface. Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 44

The Cisco IOS DHCP Snooping Feature To configure DHCP snooping on a Cisco LAN switch, follow these steps: • Step 1. Enable DHCP snooping globally using the command ip dhcp snooping. • Step 2. Apply DHCP snooping to a particular VLAN using the command ip dhcp snooping vlan-number. • Step 3. Configure interfaces that connect to the DHCP server or lead toward the legitimate DHCP servers as trusted, using the ip dhcp snooping trust command. • Step 4. Optionally, you can enable rate limiting on the DHCP snooping untrusted interfaces. Rate is configured as DHCP packets per second. This is accomplished using the interface configuration mode command ip dhcp snooping limit rate. • Step 5. To verify the DHCP snooping configuration, use the show ip dhcp snooping command. You can display the DHCP snooping Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 45

Chapter 7 Summary § § § § Troubleshooting BGP Neighbor Relationships Troubleshooting Port Security Troubleshooting VLANs Troubleshooting GW 1’s OSPF Neighbor Relation Problem with Router R 1 Troubleshooting OSPF Adjacency Troubleshooting SSH and Telnet Troubleshooting HSRP Troubleshooting Erroneous Routing Information Troubleshooting Multiple Masters within a VRRP Troubleshooting Ether. Channel Troubleshooting FHRPs The Cisco IOS DHCP Snooping Feature Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 46

Chapter 7 Labs § Lab 7 -1 OSPF Opportunities Chapter 7 © 2007 –

Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public

Acknowledgment • Some of the texts and images are from Troubleshooting and Maintaining Cisco IP Networks (TSHOOT) Foundation Learning Guide by Amir Ranjbar (158720455 X) • Copyright © 2015 – 2016 Cisco Systems, Inc. • Special Thanks to Bruno Silva Chapter 7 © 2007 – 2016, Cisco Systems, Inc. All rights reserved. Cisco Public 49