intermittent ethernet connectivity issue on B4860QDS?

klauffenburger · ‎11-19-2015

Hello,

We are experiencing intermittent ethernet connectivity problems on multiple B4860QDS boards.

We are using SDK v1.8 out-of-the-box kernel and rootfs (core) images from SDK v1.8. Here is the RCW:

Reset Configuration Word (RCW):

00000000: 140e0018 0f001218 00000000 00000000

00000010: 54980000 9000a000 fc025000 a9000000

00000020: 01000000 00000000 00000000 0001b1f8

00000030: 00000000 14000020 00000000 00000011

The /etc/network/interfaces file is configured like this:

------------

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface

auto lo

iface lo inet loopback

auto fm1-mac5

iface fm1-mac5 inet dhcp

------------

We have at least three boards having this issue. On two of the boards, the fm1-mac5 port usually connects (initializes and obtains an IP address from the DHCP server) on boot-up more than 50% of the time. The third board almost never is able to connect. This is also true if the port is configured for static IP address via the /etc/network/interfaces file.

When the port doesn't connect, the TX packets stat for the port increments while the RX packets stat is stuck at zero, making it appear as though either outgoing packets are not making it onto the media or incoming packets are not getting received.

When the problem occurs, the port status looks like this right after the system boots:

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:1296 (1.2 KiB)

Memory:fe4e8000-fe4e8fff

Note that the port status shows UP, and there are transmit packets showing, but no receive packets.

If I subsequently try to disable the port using the ifconfig command, I get this:

root@b4860qds:~# ifdown -a

ifdown: interface fm1-mac5 not configured

But the port status still shows UP:

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:1296 (1.2 KiB)

Memory:fe4e8000-fe4e8fff

If I try to disable the port using the ip command, I get this:

root@b4860qds:~# ip link set dev fm1-mac5 down

cpu6/6: > WARNING (FM-Port) [CPU06, drivers/net/ethernet/freescale/fman/Peripherals/FM/Port/fm_port.c:3548 FM_PORT_Disable]:

cpu6/6: FM-0-port-1g-TX-4: BMI or QMI is Busy. Port forced downcpu6/6:

I'm not sure what this means. However, now the port status does show DOWN:

root@b4860qds:~# ifconfig

root@b4860qds:~# ip link

1: lo: <LOOPBACK> mtu 65536 qdisc noqueue state DOWN mode DEFAULT group default

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: fm1-mac5: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether 00:04:9f:00:a9:04 brd ff:ff:ff:ff:ff:ff

3: fm1-mac6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 00:04:9f:00:a9:05 brd ff:ff:ff:ff:ff:ff

4: fm1-mac9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 00:04:9f:00:a9:08 brd ff:ff:ff:ff:ff:ff

5: fm1-mac10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 00:04:9f:00:a9:09 brd ff:ff:ff:ff:ff:ff

6: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default

link/ipip 0.0.0.0 brd 0.0.0.0

7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default

link/sit 0.0.0.0 brd 0.0.0.0

Here is what happens if I then try to re-enable the port:

root@b4860qds:~# ifup -a

udhcpc (v1.22.1) started

Sending discover...

No lease, failing

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:12 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:2592 (2.5 KiB)

Memory:fe4e8000-fe4e8fff

If I set the port back to DOWN status using the ip command, and change the port configuration to static, so that the /etc/network/interfaces file looks like this:

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface

auto lo

iface lo inet loopback

auto fm1-mac5

iface fm1-mac5 inet static

address 192.168.5.200

netmask 255.255.255.0

gateway 192.168.5.1

...then set the port back up, I get the following:

root@b4860qds:~# ifup -a

ifup: interface lo already configured

IPv6: ADDRCONF(NETDEV_UP): fm1-mac5: link is not ready

root@b4860qds:~# IPv6: ADDRCONF(NETDEV_CHANGE): fm1-mac5: link becomes ready

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

inet addr:192.168.5.200 Bcast:192.168.5.255 Mask:255.255.255.0

inet6 addr: fe80::204:9fff:fe00:a904/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:20 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:3240 (3.1 KiB)

Memory:fe4e8000-fe4e8fff

The port appears to be UP, but a ping doesn't work:

root@b4860qds:~# ping 192.168.5.20

PING 192.168.5.20 (192.168.5.20) 56(84) bytes of data.

From 192.168.5.200 icmp_seq=1 Destination Host Unreachable

From 192.168.5.200 icmp_seq=2 Destination Host Unreachable

From 192.168.5.200 icmp_seq=3 Destination Host Unreachable

From 192.168.5.200 icmp_seq=4 Destination Host Unreachable

From 192.168.5.200 icmp_seq=5 Destination Host Unreachable

From 192.168.5.200 icmp_seq=6 Destination Host Unreachable

...and the port operational state appears to be the same as when DHCP was used (no packet reception):

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

inet addr:192.168.5.200 Bcast:192.168.5.255 Mask:255.255.255.0

inet6 addr: fe80::204:9fff:fe00:a904/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:29 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:3618 (3.5 KiB)

Memory:fe4e8000-fe4e8fff

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:8 errors:0 dropped:0 overruns:0 frame:0

TX packets:8 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:896 (896.0 B) TX bytes:896 (896.0 B)

root@b4860qds:~# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: fm1-mac5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 00:04:9f:00:a9:04 brd ff:ff:ff:ff:ff:ff

inet 192.168.5.200/24 brd 192.168.5.255 scope global fm1-mac5

valid_lft forever preferred_lft forever

inet6 fe80::204:9fff:fe00:a904/64 scope link

valid_lft forever preferred_lft forever

3: fm1-mac6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:04:9f:00:a9:05 brd ff:ff:ff:ff:ff:ff

4: fm1-mac9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:04:9f:00:a9:08 brd ff:ff:ff:ff:ff:ff

5: fm1-mac10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:04:9f:00:a9:09 brd ff:ff:ff:ff:ff:ff

6: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default

link/ipip 0.0.0.0 brd 0.0.0.0

7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default

link/sit 0.0.0.0 brd 0.0.0.0

As I mentioned above, this is an intermittent problem. Here is the situation following the next reboot:

...

Running postinst /etc/rpm-postinsts/104-inetutils-telnetd...

Running postinst /etc/rpm-postinsts/105-inetutils-tftpd...

Running postinst /etc/rpm-postinsts/106-inetutils-ftpd...

update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)

INIT: Entering runlevel: 5

Configuring network interfaces... udhcpc (v1.22.1) started

Sending discover...

Sending select for 192.168.5.109...

Lease of 192.168.5.109 obtained, lease time 86400

/etc/udhcpc.d/50default: Adding DNS 192.168.5.1

done.

Starting system message bus: dbus.

Starting OpenBSD Secure Shell server: sshd

generating ssh RSA key...

generating ssh ECDSA key...

generating ssh DSA key...

generating ssh ED25519 key...

done.

Starting network benchmark server: netserver.

Starting system log daemon...0

Starting kernel log daemon...0

Starting internet superserver: xinetd.

Stopping Bootlog daemon: bootlogd.

fsl-networking (QorIQ SDK) QorIQ-SDK-V1.8 b4860qds /dev/ttyS0

b4860qds login: root

root@b4860qds:~# ifconfig

fm1-mac5 Link encap:Ethernet HWaddr 00:04:9f:00:a9:04

inet addr:192.168.5.109 Bcast:192.168.5.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:126 errors:0 dropped:0 overruns:0 frame:0

TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:11918 (11.6 KiB) TX bytes:996 (996.0 B)

Memory:fe4e8000-fe4e8fff

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Note that a lease was obtained from DHCP on the first discover.

This is an extremely annoying issue. Any help or even clues to help solve the issue would be greatly appreciated.

Thanks.

--ken