The last two weeks we had about 3 times 30 seconds of downtime caused by out switches. The problem is that a lot of packets are being discared. These packets are received on our core switch. We have configured two 6224's in a stack connected to multiple 3500's. They are connected via two cables and configured as port channel on both sides.
The regular port channel does not show any errors:
CoreSwitch#show interfaces counters port-channel 8
Ch InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
ch8 2214803607 11553709 144015 8191
Ch OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
ch8 12432260369 13633306 1196509 655365
Alignment Errors: ............................. 0
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Multiple Collision Frames: .................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Oversize Packets: ............................. 288931
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
AccessSwitch# show interfaces counters port-channel 1
Ch InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
ch1 966097929 93105839 49047587 831619282606
Ch OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
ch1 862523206 12199637 1535178 211770915270
FCS Errors: 0
Single Collision Frames: 0
Late Collisions: 0
Excessive Collisions: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0
However, looking at the interfaces more specific show something different:
CoreSwitch#show interfaces counters ethernet 1/g8
Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
1/g8 586862096 2984293 96750 4030
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
1/g8 6678995559 6414005 356659 246406
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 106394
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 0
LANStack2#show interfaces counters ethernet 2/g8
Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
2/g8 1652287248 8681279 47883 4190
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
2/g8 5849332071 7335827 843155 410456
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 183346
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 2167
Transmit Packets Discarded: ................... 0
Unit 2 discarded some frames! I see this behavior on all switches connected to the second unit.
Port configuration:
CoreSwitch#show running-config interface ethernet 1/g8
channel-group 8 mode auto
CoreSwitch#show running-config interface ethernet 2/g8
channel-group 8 mode auto
CoreSwitch#show running-config interface port-channel 8
service-policy in VOIP
spanning-tree guard root
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
switchport general allowed vlan add 1 tagged
AccessSwitch#
interface port-channel 1
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
exit
interface ethernet g3
switchport mode general
channel-group 1 mode auto
interface ethernet g4
switchport mode general
channel-group 1 mode auto
A soon as we lost connection I rapidly connected to the switch and did a show process cpu command to see what what going on. The cpu utilisation is higher than normale but not unacceptable high. Normally it's below 10% and now the 1 min timer raised to 32%. 18% was caused by the ipMapForwardingTask, which might indicate a buffer issue?
Any troubleshoot tips? Vlans are the same. The stranged thing is that only unit2 is discarding packets. Vlans are configured the same so that doens't explain the discards. We also notice a connection loss for a short time so that's not the issue.