Section 15 Tasks
- Read today’s lesson notes (below)
- Review yesterday’s lesson notes
- Complete today’s lab
- Read the ICND1 cram guide
- Spend 15 minutes on the subnetting.org website
We have covered much of the ICND1 troubleshooting requirements in previous lessons, particularly ACLs and IP addressing. Layers 1 and 2 cover a lot of possible issues and their causes, which will be the focus of today’s lesson.
LAN switching is a form of packet switching that is used in Local Area Networks (LANs). LAN switching is performed in hardware at the Data Link Layer. Because LAN switching is hardwarebased, it uses hardware addresses that are referred to as Media Access Control (MAC) addresses. The MAC addresses are then used by LAN switches to forward frames.
Today you will learn about the following:
- Troubleshooting at the Physical Layer
- VLAN, VTP, and trunking overview
- Troubleshooting VLANs
- Using the show vlan command
This module maps to the following CCNA syllabus requirements:
- Troubleshoot and resolve Layer 1 problems
- Framing
- CRC
- Runts
- Giants
- Dropped packets
- Late collisions
- Input / Output errors
- Troubleshoot and resolve VLAN problems
- Verify that VLANs are configured
- Verify that port membership is correct
- Verify that the IP address is configured
- Troubleshoot and resolve trunking problems on Cisco switches
- Verify that the trunk states are correct
- Verify that encapsulation is configured correctly
- Verify that VLANs are allowed
Troubleshooting at the Physical Layer
Cisco IOS switches support several commands that can be used to troubleshoot Layer 1, or at least suspected Layer 1, issues. However, in addition to being familiar with the software command suite, it is also important to have a solid understanding of physical indicators (i.e., LEDs) that can be used to troubleshoot link status or that indicate an error condition.
Troubleshooting Link Status Using Light Emitting Diodes (LEDs)
If you have physical access to the switch or switches, LEDs can be a useful troubleshooting tool. Different Cisco Catalyst switches provide different LED capabilities. Understanding the meaning of the LEDs is an integral part of Catalyst switch link status and system troubleshooting. Cisco Catalyst switches have front-panel LEDs that can be used to determine link status, as well as other variables such as system status.
Check Cisco documentation for the Catalyst 2960 switch model by Googling “Catalyst 2960 Switch Hardware Installation Guide.” The installation and configuration guides consist of many hundreds of pages of notes, advice, and technical information. It’s worth browsing through it but you shouldn’t be expected to know the contents of it for the CCNA exam beyond what is in the syllabus (which is covered in this guide).
Figure 15.1 – Cisco 2960 Switch LEDs. Image Copyright Cisco Systems
The PoE LED is found only on the Catalyst 2960 switch model.
System LED
The system LED indicates that the system is receiving power (or is not) and is functioning properly.
Table 15.1 below lists the LED colours and the status that they indicate:
Table 15.1 – System LEDs
RPS LED
The RPS LED is only present on switches featuring a redundant power supply. Table 15.2 below lists the LED colours and their meanings:
Table 15.2 – RPS LEDs
Port LEDs and Modes
Port LEDs give information about a group of ports or individual ports, as shown in Table 15.3 below:
Table 15.3 – Modes for Port LEDs
You can cycle through modes by pressing the Mode button until you reach the mode setting you require. This will change the meaning of the port LED colours, as shown in Table 15.4 below:
Table 15.4 – Mode Settings
In addition to understanding what the different LED colours mean, it is also important to have an understanding of what action to take to remedy the issue. For example, assume that you are troubleshooting a Catalyst 6500 Series switch and you notice that the status LEDs on the supervisor engine (or any switching modules) is red or off. In such cases, it is possible that the module might have shifted out of its slot, or, in the event of a new module, was not correctly inserted into the chassis. In this case, the recommended action is to reseat the module. In some cases, it also may be necessary to reboot the entire system.
While a link or port LED colour other than green typically indicates some kind of failure or other issue, it is important to remember that a green link light does not always mean that the cable is fully functional. For example, a single broken wire or one shut down port can cause the problem of one side showing a green link light while the other side does not. This could be because the cable encountered physical stress that caused it to be functional at a marginal level. In such cases, the CLI can be used to perform additional troubleshooting.
Troubleshooting Cable Issues
When troubleshooting cabling issues (Layer 1 troubleshooting), it can sometimes be very easy to find the problem because you can directly see and inspect the cable. However, sometimes cabling problems can be invisible, so you will have to engage in a systematic troubleshooting process to make sure the problem is really localised at Layer 1. A general recommendation is to properly test all cabling before engaging in a complex infrastructure implementation. Some common cabling problems include the following:
- Plugging in a cable but getting no connection
- Plugging in a cable and getting a connection but with very low throughput on that connection
- Everything is working normally but suddenly the connection goes away, and then comes back, and then goes away again (i.e., flapping)
- Intermittent connectivity, where it seems to work fine but the signal gets lost from time to time
Some of the recommended tests for these problems include:
- Verifying that the switch link light is on
- Verifying that the link light is not turning on and off intermittently
- Verifying that the cable is punched correctly
- Verifying that the cable is not physically damaged
- Verifying that the cable is not too long (this may cause signal degradation)
- Verifying that the cable connectors are not faulty (you might need to use other connectors)
- Verifying that the wires are pinned in the correct order (in the case of copper cables)
If you want to be sure that you are not dealing with a cabling issue, one of the simplest things to do is to replace the cable and run the same tests again. This is very easy to do and might help in immediately fixing the issue without investing much time and resources into the troubleshooting process.
NOTE: Sometimes even brand new cables can come with a defect, so do not assume that a new cable should function as expected.
Troubleshooting Module Issues
Most routers and switches used in an enterprise network offer copper port connectivity, but also dedicated ports that can be populated with different kind of transceivers. These transceivers are usually used for fibre connectivity, but there are also copper-compatible transceivers.
Fibre connections may run over very long distances, and generally those particular ports are modular and require a compatible SFP (small form-factor pluggable transceiver), like the one presented in Figure 15.2 below:
Figure 15.2 – SFP Module
Although they look similar, depending upon the type of connectivity used, the appropriate SFP module should be used based on several parameters, including:
- Type of media: optical fibre or copper
- Fibre type: single-mode or multi-mode fibre
- Bandwidth
- Wavelength
- Core size
- Modal bandwidth
- Operating distance
NOTE: When purchasing transceivers for your network, you should always check the compatibility between the device ports, type of module, and type of fibre used.
Transceivers can be plugged into and unplugged from the network device (e.g., switch, router, firewall, etc.) at any time without restarting the device. When there is no connection you will see no activity on the SFP modules, and this is one of the easiest issues to troubleshoot if you have access to the device.
On the other hand, you might plug in a fibre cable that will activate that port but the connectivity suffers from different issues (e.g., performance degradation or intermittent connectivity) or simply does not exist. In this case, there are several approaches you could take:
- Verify that the correct cable types have been used (multi-mode vs. single-mode)
depending upon the type of transceiver - Verify that the cable is not broken, using dedicated fibre optic testing tools
- Verify that the correct type of transceiver has been used
- Verify that the transceiver does not have hardware issues (swap it and test the connection
with another SFP) - Verify that the device port is configured with the correct parameters based on the type of
transceiver and cable used
To minimise connection downtime, you should monitor the ports populated with SFP modules in order to see possible errors that appear in the statistics. This can be done with standard monitoring tools, most often using SNMP.
Using the Command Line Interface to Troubleshoot Link Issues
Several Command Line Interface (CLI) commands can be used to troubleshoot Layer 1 issues on Cisco IOS Catalyst switches. Commonly used commands include the show interfaces, the show controllers, and the show interface [name] counters errors commands. In addition to knowing these commands, you also are required to be able to interpret accurately the output or information that these commands provide.
The show interfaces command is a powerful troubleshooting tool that provides a plethora of information, which includes the following:
- The administrative status of a switching port The port operational state
- The media type (for select switches and ports)
- Port input and output packets
- Port buffer failures and port errors
- Port input and output errors
- Port input and output queue drops
The output of the show interfaces command for a GigabitEthernet switch port is illustrated below:
Catalyst-3750-1#show interfaces GigabitEthernet3/0/1
GigabitEthernet0/1 is up, line protocol is down (notconnect)
Hardware is GigabitEthernet, address is 000f.2303.2db1 (bia 000f.2303.2db1)
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
Keepalive not set
Auto-duplex, Auto-speed, link type is auto, media type is unknown
input flow-control is off, output flow-control is desired
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of “show interface” counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Most Cisco Catalyst switch ports default to the notconnect state, as illustrated in the first line of the output printed by this command. However, a port can also transition to this state if a cable is removed from the port or is not correctly connected. This status is also reflected when the connected cable is faulty or when the other end of the cable is not connected to an active port or device (e.g., if a workstation connected to the switch port is powered off).
NOTE: When troubleshooting GigabitEthernet ports, this port status may also be a result of incorrect Gigabit Interface Converters (GBICs) being used between the two ends.
The first part of the output in the first line printed by this command (i.e., [interface] is up) refers to the Physical Layer status of the particular interface. The second part of the output (i.e., line protocol is down) indicates the Data Link Layer status of the interface. If this indicates an “up,” then it means that the interface can send and receive keepalives. Keep in mind that it is possible for the switch port to indicate that the Physical Layer is up while the Data Link Layer is down, for example, such as when the port is a SPAN destination port (for sniffer traffic) or if the local port is connected to a CatOS (older switch operating system) switch with its port disabled.
The Input queue indicates the actual number of frames dropped because the maximum queue size was exceeded. The flushes column counts Selective Packet Discard (SPD) drops on the Catalyst 6000 Series switches. SPD drops low-priority packets when the CPU is overloaded in order to save some processing capacity for high-priority packets. The flushes counter in the show interfaces command output increments as part of SPD, which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies only to processswitched traffic.
The total output drops indicates the number of packets dropped because the output queue is full. This is often seen when traffic from multiple inbound high-bandwidth links (e.g., GigabitEthernet links) is being switched to a single outbound lower-bandwidth link (e.g., a FastEthernet link). The output drops increment because the interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.
Some of the other interface-specific terms that can be analysed from the show interfaces output and can be very useful during Layers 1 and 2 troubleshooting are:
- Frame number: This field describes the number of packets received incorrectly having a
CRC error and a non-integer number of octets. This is usually the result of collisions due
to a malfunctioning Ethernet device (hardware fault). - CRC: This field indicates that the CRC (cyclic redundancy checksum) generated by the
sending device does not match the checksum calculated at the receiving device. This
usually indicates transmission problems on a LAN, collisions, or a system transmitting bad
data. - Runts: This field indicates the number of packets that are discarded due to being smaller
than the minimum packet size. On Ethernet segments, packets smaller than 64 bytes are
considered runts. - Giants: This field indicates the number of packets that are discarded due to being larger
than the maximum packet size. On Ethernet segments, packets larger than 1518 bytes are
considered giants. - Late collisions: Late collisions usually occur when Ethernet cables are too long or when
there are too many repeaters in the network. The number of collisions represents the
number of messages retransmitted due to an Ethernet collision. This is usually caused by
an overextended LAN. - Input errors: This field provides the total sum of runts, giants, CRC, overruns, and ignored
packets. - Output errors: This field provides the total sum of all errors that prevented the final
transmission of datagrams out of the interface.
In addition to the show interfaces command, the show interfaces [name] counters errors command can also be used to view interface errors and facilitate Layer 1 troubleshooting. The output that is printed by the show interfaces [name] counters errors command is as follows:
Catalyst-3750-1#show interfaces GigabitEthernet3/0/1 counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize Gi3/0/1 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Gi3/0/1 0 0 0 0 0 0 Port Giants Gi3/0/1 0
The following section describes some of the error fields included in the output of the show interfaces [name] counters errors command, and which issues or problems are indicated by non-zero values under these fields.
The Align-Err field reflects a count of the number of frames received that do not end with an even number of octets and that have a bad CRC. These errors are usually the result of a duplex mismatch or a physical problem, such as cabling, a bad port, or a bad network interface controller (NIC). When the cable is first connected to the port, some of these errors can occur. In addition, if there is a hub connected to the port, collisions between other devices on the hub can cause these errors.
The FCS-Err field reflects the number of valid-sized frames with Frame Check Sequence (FCS) errors but no framing errors. This is typically a physical issue, such as cabling, a bad port, or a bad NIC. Additionally, a non-zero value under this field could indicate a duplex mismatch.
A non-zero value in the Xmit-Err field is an indication that the internal send (Tx) buffer is full. This is commonly seen when traffic from multiple inbound high-bandwidth links (e.g., GigabitEthernet links) is being switched to a single outbound lower-bandwidth link (i.e., a FastEthernet link), for example.
The Rcv-Err field indicates the sum of all received errors. This counter is incremented when the interface receives an error such as a runt, a giant, or an FCS, for example.
The UnderSize field is incremented when the switch receives frames that are smaller than 64 bytes in length. This is commonly caused by a faulty sending device.
The various collision fields indicate collisions on the interface. This is common for half-duplex Ethernet, which is almost non-existent in modern networks. However, these counters should not increment for full-duplex links. In the event that non-zero values are present under these counters, this typically indicates a duplex mismatch issue. When a duplex mismatch is detected, the switch prints a message similar to the following on the console or in the log:
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not full duplex), with R2 FastEthernet0/0 (full duplex)
As will be described in the section pertaining to Spanning Tree Protocol (STP), duplex mismatches can cause STP loops in the switched network if a port is connected to another switch. These mismatches can be resolved by manually configuring the speed and the duplex of the switch ports.
The Carri-Sen (carrier sense) counter increments every time an Ethernet controller wants to send data on a half-duplex connection. The controller senses the wire and ensures that it is not busy before transmitting. A non-zero value under this field indicates that the interface is operating in half-duplex mode. This is normal for half-duplex.
Non-zero values can also be seen under the Runts field due to a duplex mismatch or because of other Physical Layer problems, such as a bad cable, port, or NIC on the attached device. Runts are received frames with a bad CRC that are smaller than the minimum IEEE 802.3 frame size, which is 64 bytes for Ethernet.
Finally, the Giants counter is incremented when frames are received that exceed the IEEE 802.3 maximum frame size, which is 1518 bytes for non-jumbo Ethernet, and that have a bad FCS. For ports or interfaces connected to a workstation, a non-zero value under this field is typically caused by a bad NIC on the connected device. However, for ports or interfaces that are connected to another switch (e.g., via a trunk link), this field will contain a non-zero value if 802.1Q encapsulation is used. With 802.1Q, the tagging mechanism implies a modification of the frame because the trunking device inserts a 4-byte tag and then re-computes the FCS.
Inserting a 4-byte tag into a frame that already has the maximum Ethernet size creates a 1522- byte frame that can be considered a baby giant frame by the receiving equipment. Therefore, while the switch will still process such frames, this counter will increment and contain a nonzero value. To resolve this issue, the 802.3 committee created a subgroup called 802.3ac to extend the maximum Ethernet size to 1522 bytes; however, it is not uncommon to see a nonzero value under this field when using 802.1Q trunking.
The show controllers ethernet-controller <interface> command can also be used to display traffic counter and error counter information similar to that printed by the show interfaces and show interfaces <name> counters errors commands. The output of the show controllers ethernet-controller <interface> command is shown below:
Catalyst-3750-1#show controllers ethernet-controller GigabitEthernet3/0/1 Transmit GigabitEthernet3/0/1 Receive 4069327795 Bytes 3301740741 Bytes 559424024 Unicast frames 376047608 Unicast frames 27784795 Multicast frames 1141946 Multicast frames 7281524 Broadcast frames 1281591 Broadcast frames 0 Too old frames 429934641 Unicast bytes 0 Deferred frames 226764843 Multicast bytes 0 MTU exceeded frames 137921433 Broadcast bytes 0 1 collision frames 0 Alignment errors 0 2 collision frames 0 FCS errors 0 3 collision frames 0 Oversize frames 0 4 collision frames 0 Undersize frames 0 5 collision frames 0 Collision fragments 0 6 collision frames 0 7 collision frames 257477 Minimum size frames 0 8 collision frames 259422986 65 to 127 byte frames 0 9 collision frames 51377167 128 to 255 byte frames 0 10 collision frames 41117556 256 to 511 byte frames 0 11 collision frames 2342527 512 to 1023 byte frames 0 12 collision frames 5843545 1024 to 1518 byte frames 0 13 collision frames 0 Overrun frames 0 14 collision frames 0 Pause frames 0 15 collision frames 0 Excessive collisions 0 Symbol error frames 0 Late collisions 0 Invalid frames, too large 0 VLAN discard frames 18109887 Valid frames, too large 0 Excess defer frames 0 Invalid frames, too small 264522 64 byte frames 0 Valid frames, too small 99898057 127 byte frames 76457337 255 byte frames 0 Too old frames 4927192 511 byte frames 0 Valid oversize frames 21176897 1023 byte frames 0 System FCS error frames 127643707 1518 byte frames 0 RxPortFifoFull drop frames 264122631 Too large frames 0 Good (1 coll) frames 0 Good (>1 coll) frames
NOTE: The output above will vary slightly depending upon the switch platform on which this command is executed. For example, Catalyst 3550 series switches also include a Discarded frames field, which shows the total number of frames whose transmission attempt is abandoned due to insufficient resources. A large number in this field typically indicates a network congestion issue. In the output above, you would look at the RxPortFifoFull drop frame field, which indicates the total number of frames received on an interface that are dropped because the ingress queue is full.
Troubleshooting Port Configuration
Each networking device can be configured in different ways. Most types of misconfigurations generate problems within the network, including:
- Poor throughput
- Lack of connectivity
A device can be connected to the network, have a signal, and be able to communicate to the Internet and to other devices but the performance might be low, in a consistent and easily reproducible way. This can manifest during normal operations, including file transfer or other types of communications with the rest of the network.
With major configuration issues, the issue might manifest as complete lack of connectivity, including no link lights on the specific device ports. Sometimes the link lights are on but you still lack any kind of connectivity. This shows that the signal is passing through the cable, which means that you don’t have a cabling issue but rather a port configuration issue on one port or the other. This requires problem investigation in the device’s configuration.
There are a number of different settings when configuring a port, including:
- Speed
- Duplex
- Encapsulation/VLAN.
Most of these parameters have to be synchronised on both sides of the link, either by manually configuring them or by enabling port autoconfiguration. If detected, this method will send negotiation packets on the link to each device to detect the capabilities on the other end device and commonly agree on the best possible parameters supported by both of them to ensure an optimal transmission. The problem is that sometimes autoconfiguration does not select the best parameters for your needs, so you should also verify this and manually configure the ports according to each specific case.
If you are performing manual configuration on each port, one of the first parameters you have to take care of is the interface speed. This has to be identical on both sides of a link. If you configure it incorrectly on one side, the link might not be operational. Another related setting is port duplex, which can be configured to be either half duplex or full duplex. You can configure a link with half duplex on one side and full duplex on the other side, and even though the link will come up, the throughput will be highly affected because each side is expecting to handle communication in a different way. This will result in collisions which will affect the transmission on that particular link. Make sure that both sides use the same duplex settings in order for the traffic to be sent as efficiently as possible.
If you are operating in an enterprise-level environment, you might need to use different VLANs to segment the traffic. Each switch must be properly configured in this regard so each switch port is assigned to the correct VLAN. If you are directly connecting ports configured to use different VLAN IDs, the communication will be broken at Layer 2, even though the Physical Layer shows no issues.
By examining all the port configuration options presented above and making sure that you have everything synchronised at both ends of a link, you can be assured that the connectivity and throughput of the configured devices will be optimised.
Troubleshooting VLANs and Trunking
In the previous section, we discussed the use of three CLI commands that can be used for troubleshooting Physical Layer issues. This section describes some common approaches to identifying and troubleshooting intra-VLAN connectivity issues. Some of the more common causes of intra-VLAN connectivity issues include the following:
- Duplex mismatches
- Bad NIC or cable
- Congestion
- Hardware issues
- Software issues
- Resource oversubscription
- Configuration issues
Duplex mismatches can result in very slow network performance and connectivity. While some improvements in auto-negotiation have been made, and the use of auto-negotiation is considered a valid practice, it is still possible for duplex mismatches to occur. As an example, when the NIC is set to 100/Full and the switch port is auto-negotiating, the NIC will retain its 100/Full setting, but the switch port will be set to 100/Half. Another example would be the inverse; that is, the NIC is set to auto-negotiate, while the switch port is set to 100/Full. In that case, the NIC would auto-negotiate to 100/Half, while the switch retained its static 100/Full configuration, resulting in a duplex mismatch.
It is therefore good practice to specify manually the speed and duplex settings for 10/100 Ethernet connections, where feasible, to avoid duplex mismatches with auto-negotiation. Duplex mismatches can affect not only users directly connected to the switch but also network traffic that traverses inter-switch links that have mismatched duplex settings. The port interface speed and duplex settings can be viewed using the show interfaces command.
NOTE: Because Catalyst switches support only full-duplex for 1Gbps links, this is not commonly an issue for GigabitEthernet connections.
Multiple counters in Cisco IOS software can be used to identify a potentially bad NIC or cabling issue. NIC or cabling issues can be identified by checking the values of certain counters in different show commands. For example, if the switch port counters show an incrementing number of frames with a bad CRC or with FCS errors, this can most likely be attributed either to a bad NIC on the workstation or machine or to a bad network cable.
Network congestion can also cause intermittent connectivity issues in the switched network. The first sign that your VLAN is overloaded is if the Rx or Tx buffers on a port are oversubscribed. Additionally, excessive frame drops on a port can also be an indication of network congestion. A common cause of network congestion is due to underestimating aggregate bandwidth requirements for backbone connections. In such cases, congestion issues can be resolved by configuring EtherChannels or by adding additional ports to existing EtherChannels. While network congestion is a common cause of connectivity issues, it is also important to know that the switch itself can experience congestion issues, which can have a similar impact on network performance.
Limited switch bandwidth can result in congestion issues, which can severely impact network performance. In LAN switching, bandwidth refers to the capacity of the switch fabric. Therefore, if the switch fabric is on 5Gbps and you attempt to push 7Gbps worth of traffic through the switch, the end result is packet loss and poor network performance. This is a common issue in oversubscribed platforms, where the aggregate capacity of all ports can exceed the total backplane capacity.
Hardware problems can also cause connectivity issues in the switched LAN. Examples of such issues include bad ports or bad switch modules. While you could troubleshoot such issues by looking at physical indicators such as LEDs, if possible, such issues are sometimes difficult to troubleshoot and diagnose. In most cases, you should seek the assistance of the Technical Assistance Centre (TAC) when you suspect potentially faulty hardware issues.
Software bugs are even more difficult to identify because they cause deviation, which is hard to troubleshoot. In the event that you suspect a software bug may be causing connectivity issues, you should contact the TAC with your findings. Additionally, if error messages are printed on the console or are in the logs, you can also use some of the online tools available from Cisco to implement a workaround or get a recommendation for a version of software in which the issue has been resolved and verified.
As with any other hardware device, switches have limited resources, such as physical memory. When these resources are oversubscribed, this can lead to severe performance issues. Issues such as high CPU utilisation can have a drastic impact on both switch and network performance.
Finally, as with any other technology, incorrect configurations may also cause connectivity issues, either directly or indirectly. For example, the poor placement of the Root Bridge may result in slow connectivity for users. Directly integrating or adding an incorrectly configured switch into the production network could result in an outright outage for some or all users. The following sections describe some common VLAN-related issues, their probable causes, and the actions that can be taken to remedy them.
Troubleshooting Dynamic VLAN Advertisements
Cisco Catalyst switches use VLAN Trunk Protocol (VTP) to propagate VLAN information dynamically throughout the switched domain. VTP is a Cisco proprietary Layer 2 messaging protocol that manages the addition, deletion, and renaming of VLANs for switches in the same VTP domain.
There are several reasons why a switch might not be able to receive any VLAN information dynamically when added to the VTP domain. Some common causes include the following:
- Layer 2 trunking misconfigurations
- Incorrect VTP configuration
- Configuration revision number
- Physical Layer issues
- Software or hardware issues or bugs
- Switch performance issues
In order for switches to exchange VLAN information using VTP, a trunk must be established between the switches. Cisco IOS switches support both ISL and 802.1Q trunking mechanisms. While some switches default to ISL, which is a Cisco proprietary trunking mechanism, the current Cisco IOS Catalyst switches default to 802.1Q. When provisioning trunking between switches, it is considered good practice to specify manually the trunking encapsulation protocol. This is accomplished using the switchport trunk encapsulation [isl|dot1q] interface configuration command when configuring the link as a trunk port.
There are several commands that you can use to troubleshoot trunk connectivity issues. You can use the show interfaces command to verify basic port operational and administrative status. Additionally, you can append the [trunk] or [errors] keyword to perform additional troubleshooting and verification. The show interfaces [name] counters trunk command can be used to view the number of frames transmitted and received on trunk ports.
The output of this command also includes encapsulation errors, which can be used to verify 802.1Q and ISL, and trunking encapsulation mismatches, as illustrated in the following output:
Cat-3550-1#show interfaces FastEthernet0/12 counters trunk Port TrunkFramesTx TrunkFramesRx WrongEncap Fa0/12 1696 32257 0
Referencing the output above, you can repeat the same command to ensure that both the Tx and Rx columns are incrementing and perform additional troubleshooting from there. For example, if the switch is not sending any frames, then the interface might not be configured as a trunk, or it might be down or disabled. If the Rx column is not incrementing, then it may be that the remote switch is not configured correctly.
Another command that can be used to troubleshoot possible Layer 2 trunk misconfigurations is the show interfaces [name] trunk command. The output of this command includes the trunking encapsulation protocol and mode, the native VLAN for 802.1Q, the VLANs that are allowed to traverse the trunk, the VLANs that are active in the VTP domain, and the VLANs that are pruned. A common issue with VLAN propagation is that the upstream switch has been configured to filter certain VLANs on the trunk link using the switchport trunk allowed vlan interface configuration command. The output of the show interfaces [name] trunk command is shown below:
Cat-3550-1#show interfaces trunk Port Mode Encapsulation Status Native vlan Fa0/12 desirable n-802.1q trunking 1 Fa0/13 desirable n-802.1q trunking 1 Fa0/14 desirable n-isl trunking 1 Fa0/15 desirable n-isl trunking 1 Port Vlans allowed on trunk Fa0/12 1-4094 Fa0/13 1-4094 Fa0/14 1-4094 Fa0/15 1-4094 Port Vlans allowed and active in management domain Fa0/12 1-4 Fa0/13 1-4 Fa0/14 1-4 Fa0/15 1-4 Port Vlans in spanning tree forwarding state and not pruned Fa0/12 1-4 Fa0/13 none Fa0/14 none Fa0/15 none
Another common trunking misconfiguration issue is native VLAN mismatches. When you are configuring 802.1Q trunks, the native VLAN must match on both sides of the trunk link; otherwise, the link will not work. If there is a native VLAN mismatch, then STP places the port in a port VLAN ID (PVID) inconsistent state and will not forward on the link. In such cases, an error message similar to the following will be printed on the console or in the log:
*Mar 1 03:16:43.935: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on FastEthernet0/11 VLAN2. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_PEER: Blocking FastEthernet0/11 on VLAN0001. Inconsistent peer vlan. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking FastEthernet0/11 on VLAN0002. Inconsistent local vlan. *Mar 1 03:16:43.935: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on FastEthernet0/12 VLAN2. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_PEER: Blocking FastEthernet0/12 on VLAN0001. Inconsistent peer vlan. *Mar 1 03:16:43.939: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking FastEthernet0/12 on VLAN0002. Inconsistent local vlan.
While STP troubleshooting will be described later in this guide, this inconsistent state could be validated using the show spanning-tree command, as illustrated below:
Cat-3550-1#show spanning-tree interface FastEthernet0/11 Vlan Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- ---------------- VLAN0001 Desg BKN* 19 128.11 P2p *PVID_Inc VLAN0002 Desg BKN* 19 128.11 P2p *PVID_Inc
If you have checked and validated that the trunk is indeed correctly configured and operational between the two switches, then the next step would be to validate VTP configuration parameters. These parameters include the VTP domain name, the correct VTP mode, and the VTP password, if one has been configured for the domain, using the show vtp status and show vtp password commands, respectively. The output of the show vtp status command is shown below:
Cat-3550-1#show vtp status VTP Version : running VTP2 Configuration Revision : 0 Maximum VLANs supported locally : 1005 Number of existing VLANs : 8 VTP Operating Mode : Server VTP Domain Name : TSHOOT VTP Pruning Mode : Enabled VTP V2 Mode : Enabled VTP Traps Generation : Disabled MD5 digest : 0x26 0x99 0xB7 0x93 0xBE 0xDA 0x76 0x9C ... [Truncated Output]
When using the show vtp status command, ensure that the switches are running the same version of VTP. By default, Catalyst switches run VTP version 1. A switch running VTP version 1 cannot participate in a VTP version 2 domain. If the switch is incapable of running VTP version 2, then all VTP version 2 switches should be configured to run version 1 instead using the vtp version global configuration command.
NOTE: I f you change the VTP vers ion on the server, then the change will be propagated automatically to client switches in the VTP domain.
VTP propagation is enabled for VTP client/server or server/server devices. If VTP is disabled on a switch (i.e., transparent mode), then the switch will not receive VLAN information dynamically via VTP. However, be mindful of the fact that with version 2, transparent mode switches will forward received VTP advertisements out of their trunk ports and act as VTP relays. This happens even if the VTP version is not the same. The VTP domain name should also be consistent on the switches.
Finally, the output of the show vtp status command also includes the MD5 hash used for authentication purposes. This hash, which is derived from the VTP domain name and password, should be consistent on all switches in the domain. If the VTP passwords or domain names are different on the switches, then the calculated MD5 will also be different. If the domain name or password is different, then the show vtp status command will indicate an MD5 digest checksum mismatch, as illustrated in the following output:
Cat-3550-1#show vtp status VTP Version : running VTP2 Configuration Revision : 0 Maximum VLANs supported locally : 1005 Number of existing VLANs : 8 VTP Operating Mode : Server VTP Domain Name : TSHOOT VTP Pruning Mode : Enabled VTP V2 Mode : Enabled VTP Traps Generation : Disabled MD5 Digest : 0x26 0x99 0xB7 0x93 0xBE 0xDA 0x76 0x9C *** MD5 digest checksum mismatch on trunk: Fa0/11 *** *** MD5 digest checksum mismatch on trunk: Fa0/12 *** ... [Truncated Output]
Finally, the configuration revision number can wreak havoc when using VTP. Switches use the configuration revision number to keep track of the most recent information in the VTP domain. Every switch in the domain stores the configuration revision number that it last heard from a VTP advertisement, and this number is incremented every time new information is received. When any switch in the VTP domain receives an advertisement message with a higher configuration revision number than its own, it will overwrite any stored VLAN information and synchronise its own stored VLAN information with the information received in the advertisement message.
Therefore, if you are wondering why the switch that you integrated into the VTP domain is not receiving any VLAN information, it may be that the same switch had a higher configuration revision number and caused all other switches to overwrite their local VLAN information and replace it with the information received in the advertisement message from the new switch. To avoid such situations, always ensure that the configuration revision number is set to 0 prior to integrating a new switch into the domain. This can be done by changing the VTP mode or changing the VTP domain name on the switch. The configuration revision number is included in the output of the show vtp status command.
Troubleshooting Loss of End-to-End Intra-VLAN Connectivity
There are several possible reasons for a loss of end-to-end connectivity within a VLAN. Some of the most common causes include the following:
- Physical Layer issues
- VTP pruning
- VLAN trunk filtering
- New switches
- Switch performance issues
- Network congestion
- Software or hardware issues or bugs
NOTE: For brevity, only trunking, VTP pruning, trunk filtering, and the integration of new switches into the domain will be described in this section. Software or hardware issues or bugs and switch performance issues are described throughout this guide. Physical Layer troubleshooting was described earlier in this module.
VTP pruning removes VLANs from the VLAN database of the local switch when no local ports are a part of that VLAN. VTP pruning increases the efficiency of trunks by eliminating unnecessary Broadcast, Multicast, and unknown traffic from being flooded across the network.
While VTP pruning is a desirable feature to implement, incorrect configuration or implementation can result in a loss of end-to-end VLAN connectivity. VTP pruning should be enabled only in client/server environments. Implementing pruning in a network that includes transparent mode switches may result in a loss of connectivity. If one or more switches in the network are in VTP transparent mode, you should either globally disable pruning for the entire domain or ensure that all VLANs on the trunk link(s) to the upstream transparent mode switch(es) are pruning ineligible (i.e., they are not pruned), using the switchport trunk pruning vlan interface configuration command under the applicable interfaces.
Verify Allowed VLANs and Trunk Status
In addition to VTP pruning, incorrectly filtering VLANs on switch trunk links can result in a loss of end-to-end VLAN connectivity. By default, all VLANs are allowed to traverse all trunk links; however, Cisco IOS software allows administrators to remove (or add) VLANs selectively to specific trunk links using the switchport trunk allowed vlan interface configuration command. You can use the show interfaces [name] trunk and the show interfaces [name] switchport commands to view pruned and restricted VLANs on trunk links. The output of the show interfaces [name] trunk command, which is the easiest way to verify the allowed VLANs on a trunk, is shown below:
Cat-3550-1#show interfaces trunk Port Mode Encapsulation Status Native vlan Fa0/1 on 802.1q trunking 1 Fa0/2 on 802.1q trunking 1 Port Vlans allowed on trunk Fa0/1 1,10,20,30,40,50 Fa0/2 1-99,201-4094 Port Vlans allowed and active in management domain Fa0/1 1,10,20,30,40,50 Fa0/2 1,10,20,30,40,50,60,70,80,90,254 Port Vlans in spanning tree forwarding state and not pruned Fa0/1 1,10,20,30,40,50 Fa0/2 1,40,50,60,70,80,90,254
You should also check that the correct VLANs are advertised on your trunk links. Inproper VLANs allowed on the link can lead to a lack of functionality or security issues. Also, you want to make sure that the same VLANs are allowed on both ends of a trunk.
NOTE: You should be very careful not to forget the [add] keyword when adding another VLAN(s) that should be allowed over a trunk link. For example, if you already have switchport trunk allowed vlan 10, 20 configured and you want to allow VLAN 30 as well, you need to enter the command switchport trunk allowed vlan add 30. If you simply configured switchport trunk allowed vlan 30, previously permitted VLANs 10 and 20 would be removed from the trunk, which would cause a break of communication for VLANs 10 and 20.
Another important piece of information that is offered by the show interface trunk command is the trunk status. This confirms whether the trunk is formed or not and has to be checked at both ends of the link. If the interface is not in “trunking” mode, one of the most important things that have to be verified is the mode of operation (on, auto, etc.) to see whether it allows forming a trunking state with the other end of the link.
Verify Encapsulation Type
Another important step in resolving trunking problems is verifying that the correct encapsulation is configured at both ends of a trunk link. Most Cisco switches allow both ISL and dot1Q encapsulation types. Althouth most modern network designs use dot1Q, there might be situations in which ISL is the preferred method. The encapsulation type is configured using the switchport trunk encapsulation <type> command. Some of the commands that can be used to verify the encapsulation types are:
- show interface trunk
- show interface <number> switchport
The output of the show interfaces [name] switchport command on a port that has been configured statically as an 802.1Q trunk link is shown below:
Cat-3550-2#show interfaces FastEthernet0/7 switchport Name: Fa0/7 Switchport: Enabled Administrative Mode: trunk Operational Mode: trunk Administrative Trunking Encapsulation: dot1q Operational Trunking Encapsulation: dot1q Negotiation of Trunking: On Access Mode VLAN: 1 (default) Trunking Native Mode VLAN: 1 (default) Administrative Native VLAN tagging: enabled Voice VLAN: none Administrative private-vlan host-association: none Administrative private-vlan mapping: none Administrative private-vlan trunk native VLAN: none Administrative private-vlan trunk native VLAN tagging: enabled Administrative private-vlan trunk encapsulation: dot1q Administrative private-vlan trunk normal VLANs: none Administrative private-vlan trunk associations: none Administrative private-vlan trunk mappings: none Operational private-vlan: none Trunking VLANs Enabled: 3,5,7 Pruning VLANs Enabled: 2-8 Capture Mode Disabled Capture VLANs Allowed: ALL Protected: false Unknown unicast blocked: disabled Unknown multicast blocked: disabled Appliance trust: none
As was described in the previous section, the integration of a new switch into the network can result in a loss of VLAN information in the management domain. This loss of VLAN information can result in a loss of connectivity between devices within the same VLAN. Ensure that the configuration revision number is reset prior to integrating a new switch into the LAN.
Using the “show vlan” Command
In addition to the commands that were described in the previous sections, there are additional Cisco IOS software commands that are useful for both verifying and troubleshooting VLAN configurations. One of the most commonly used VLAN verification and troubleshooting commands is the show vlan command. This command displays parameters for all VLANs within the administrative domain, as illustrated in the following output:
Cat-3550-1#show vlan VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Fa0/11, Fa0/12, Fa0/13, Fa0/14, Fa0/20, Fa0/21, Fa0/22, Fa0/23, Fa0/24 150 VLAN_150 active Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Fa0/9, Fa0/10 160 VLAN_160 active Fa0/15, Fa0/16, Fa0/17, Fa0/18, Fa0/19 170 VLAN_170 active Gi0/1, Gi0/2 1002 fddi-default active 1003 token-ring-default active 1004 fddinet-default active 1005 trnet-default active VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode ---- ----- ---------- ----- ------ ------ -------- ---- -------- 1 enet 100001 1500 - - - - - 150 enet 100150 1500 - - - - - 160 enet 100160 1500 - - - - - 170 enet 100170 1500 - - - - - 1002 fddi 101002 1500 - - - - - 1003 tr 101003 1500 - - - - - 1004 fdnet 101004 1500 - - - ieee - 1005 trnet 101005 1500 - - - ibm - Trans1 Trans2 ------ ------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Remote SPAN VLANs ------------------------------------------------------------------- Primary Secondary Type Ports ------- --------- ----------------- -------------------------------
This command prints all available VLANs, along with the ports that are assigned to each of the individual VLANs. Only access ports, regardless of whether they are up or down, will be included in the output of this command. Trunk links will not be included, as these belong to all VLANs. The show vlan command also provides information on RSPAN VLANs, as well as Private VLAN (PVLAN) configuration on the switch (this is a CCNP subject). The show vlan command can be used with additional keywords to provide information that is more specific. The following output displays the supported additional keywords that can be used with this command:
Cat-3550-1#show vlan ? brief VTP all VLAN status in brief id VTP VLAN status by VLAN id ifindex SNMP ifIndex name VTP VLAN status by VLAN name private-vlan Private VLAN information remote-span Remote SPAN VLANs summary VLAN summary information | Output modifiers <cr>
The brief field prints a brief status of all active VLANs. The output that is printed by this command is the same as the output above, with the only difference being that the last two sections will be omitted. The id field provides the same information as the show vlan command, but only for the specified VLAN, as shown in the following output:
Switch-1#show vlan id 150 VLAN Name Status Ports ---- -------------------------------- --------- -------------------- 150 VLAN_150 active Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Fa0/9, Fa0/10 VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode ---- ----- ---------- ----- ------ ------ -------- ---- -------- 150 enet 100150 1500 - - - - - Trans1 Trans2 ------ ------0 0 0 0 Remote SPAN VLAN ----------------Disabled Primary Secondary Type Ports ------- --------- ----------------- --------------------------------
Again, the VLAN name is included in the output, as are all of the access ports that belong to the VLAN. Trunk ports are not included in this output because they belong to all VLANs. Additional information also includes the VLAN MTU, RSPAN configuration (if applicable), and PVLAN configuration parameters (if applicable).
The name field allows the VLAN name to be specified instead of the ID. This command prints the same information as the show vlan id <number> command. The ifindex field displays the SNMP IfIndex for the VLAN (if applicable), while the private-vlan and remote-span fields print PVLAN and RSPAN configuration information, respectively. Finally, the summary field prints a summary of the number of VLANs that are active in the management domain. This includes standard and extended VLANs.
The show vlan command, with or without parameters, is the most useful command in the following aspects of the troubleshooting process:
- Identifying that VLANs are configured on the device
- Verifying port membership
Another useful VLAN troubleshooting command is the show vtp counters command. This command prints information on VTP packet statistics. The output of the show vtp counters command on a switch configured as a VTP server (default) is shown below:
Cat-3550-1#show vtp counters VTP statistics: Summary advertisements received : 15 Subset advertisements received : 10 Request advertisements received : 2 Summary advertisements transmitted : 19 Subset advertisements transmitted : 12 Request advertisements transmitted : 0 Number of config revision errors : 0 Number of config digest errors : 0 Number of V1 summary errors : 0 VTP pruning statistics: Trunk Join Transmitted Join Received Summary advts received from non-pruningcapable device ----- ---------------- -------------- ----------------------- Fa0/11 0 1 0 Fa0/12 0 1 0
The first six lines of the output printed by the show vtp counters command provide the statistics for the three types of VTP packets: advertisement requests, summary advertisements, and subset advertisements. These different messages will be described in the following section.
VTP advertisement requests are requests for configuration information. These messages are sent by VTP clients to VTP servers to request VLAN and VTP information they may be missing. A VTP advertisement request is sent out when the switch resets, the VTP domain name changes, or in the event that the switch has received a VTP summary advertisement frame with a higher configuration revision number than its own. VTP servers should show only the received counters incrementing, while any VTP clients should show only the transmitted counters incrementing.
VTP summary advertisements are sent out by servers every five minutes, by default. These types of messages are used to tell an adjacent switch of the current VTP domain name the configuration revision number and the status of the VLAN configuration, as well as other VTP information, which includes the time stamp, the MD5 hash, and the number of subset advertisements to follow. If these counters are incrementing on the server, then there is more than one switch acting or configured as a server in the domain.
VTP subset advertisements are sent out by VTP servers when a VLAN configuration changes, such as when a VLAN is added, suspended, changed, deleted, or other VLAN-specific parameters (e.g., VLAN MTU) have changed. One or more subset advertisements will be sent following the VTP summary advertisement. A subset advertisement contains a list of VLAN information. If there are several VLANs, more than one subset advertisement may be required in order to advertise all the VLANs.
The Number of config revision errors field shows the number of advertisements that the switch cannot accept because it received packets with the same configuration revision number but with a different MD5 hash value. This is common when changes are made to two or more server switches in the same domain at the same time and an intermediate switch receives these advertisements at the same time. This concept is illustrated in Figure 15.3 below, which illustrates a basic switched network:
Figure 15.3 – Troubleshooting Configuration Revision Number Errors
Figure 15.3 illustrates a basic network that incorporates redundancy and load sharing. It should be assumed that Sw1 and Sw2 are configured as servers, while Sw3 is configured as a client. Sw1 is the root for VLANs 10 and 30, while Sw2 is the root for VLANs 20 and 40. Assume that a simultaneous change is implemented on Sw1 and Sw2, adding VLAN 50 to Sw1 and VLAN 60 to Sw2. Both switches send out an advertisement following the change to the database.
The change is propagated throughout the domain, overwriting the previous databases of the other switches that receive this information. Assume that Sw5 receives the same information from neighbours at the same time and both advertisements contain the same configuration revision number. In such situations, the switch will not be able to accept either advertisement because they have the same configuration revision number but different MD5 hash values.
When this occurs, the switch increments the Number of config revision errors counter field and does not update its database. This situation can result in a loss of connectivity within one or more VLANs because VLAN information is not updated on the switch. To resolve this issue and ensure that the local database on the switch is updated, configure a dummy VLAN on one of the server switches, which results in another update with an incremented configuration revision number. This will overwrite the local database of all switches, allowing Sw5 to update its database as well. Keep in mind that this is not a common occurrence; however, it is possible, hence, the reason for this counter.
The Number of config digest errors counter field increments whenever the switch receives an advertisement with a different MD5 hash value than it calculated. This is the result of different VTP passwords configured on the switches. You can use the show vtp password command to verify that the configured VTP password is correct. It is also important to remember that the passwords may be the same, but hardware or software issues or bugs could be causing data corruption of VTP packets, resulting in these errors.
Finally, the VTP pruning statistics field will only ever contain non-zero values when pruning is enabled for the VTP domain. Pruning is enabled on servers and this configuration is propagated throughout the VTP domain. Servers will receive joins from clients when pruning has been enabled for the VTP domain.
Section 15 Questions
- What is the colour of the system LED under normal system operations?
- What is the colour of the RPS LED during a fault condition?
- You can cycle through modes by pressing the Mode button until you reach the mode setting you require. This changes the status of the port LED colours. True or false?
- What port speed is represented by a blinking green LED?
- If you want to be sure that you are not dealing with a cabling issue, one of the simplest things to do is to _______ the cable and run the same tests again.
- Which command is generally used to troubleshoot Layer 1 issues (besides show interfaces)?
- The _______ status is reflected when the connected cable is faulty or when the other end of the cable is not connected to an active port or device (e.g., if a workstation connected to the switch port is powered off).
- What are runts?
- The _______ command can also be used to view interface errors and facilitate Layer 1 troubleshooting.
- Which command prints a brief status of all active VLANs?
Section 15 Answers
- Green.
- Amber.
- True.
- 1000Mbps.
- Replace.
- The show controllers command.
- notconnect.
- Packets that are smaller than the minimum packet size (less than 64 bytes on Ethernet).
- show interfaces [name] counters errors.
- The show vlan brief command.
Section 15 Labs
Layer 1 Troubleshooting Lab
Test the relevant Layer 1 troubleshooting commands presented in this module on real devices:
- Examine switch system and port LED status for different scenarios, as described in the module
- Issue a show interface command and examine all the related information as per the description in this module
- Issue the same for show controllers and show interface counters errors commands
Layer 2 Troubleshooting Lab
Test the relevant Layer 2 troubleshooting commands presented in this module on real devices:
- Configure VTP between the switches and advertise some VLANs from the VTP server to the VTP client (see the VTP lab in Section 3)
- Configure a trunk between two switches and generate some traffic (ping)
- Test the show vlan command
- Test the show interface counters trunk command
- Test the show interface switchport command
- Test the show interface trunk command
- Test the show VTP status command
- Test the show VTP counter command