AnsweredAssumed Answered

intermittent ethernet connectivity issue on B4860QDS?

Question asked by Ken Lauffenburger on Nov 19, 2015

Hello,

 

We are experiencing intermittent ethernet connectivity problems on multiple B4860QDS boards.

We are using SDK v1.8 out-of-the-box kernel and rootfs (core) images from SDK v1.8.  Here is the RCW:

Reset Configuration Word (RCW):

       00000000: 140e0018 0f001218 00000000 00000000

       00000010: 54980000 9000a000 fc025000 a9000000

       00000020: 01000000 00000000 00000000 0001b1f8

       00000030: 00000000 14000020 00000000 00000011

 

The /etc/network/interfaces file is configured like this:

------------

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface

auto lo

iface lo inet loopback

auto fm1-mac5

iface fm1-mac5 inet dhcp

------------

 

We have at least three boards having this issue.  On two of the boards, the fm1-mac5 port usually connects (initializes and obtains an IP address from the DHCP server) on boot-up more than 50% of the time.  The third board almost never is able to connect.  This is also true if the port is configured for static IP address via the /etc/network/interfaces file.

 

When the port doesn't connect, the TX packets stat for the port increments while the RX packets stat is stuck at zero, making it appear as though either outgoing packets are not making it onto the media or incoming packets are not getting received.

 

When the problem occurs, the port status looks like this right after the system boots:

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:1296 (1.2 KiB)

          Memory:fe4e8000-fe4e8fff

 

Note that the port status shows UP, and there are transmit packets showing, but no receive packets.

 

If I subsequently try to disable the port using the ifconfig command, I get this:

root@b4860qds:~# ifdown -a

ifdown: interface fm1-mac5 not configured

 

But the port status still shows UP:

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:1296 (1.2 KiB)

          Memory:fe4e8000-fe4e8fff

 

If I try to disable the port using the ip command, I get this:

root@b4860qds:~# ip link set dev fm1-mac5 down

cpu6/6: > WARNING (FM-Port) [CPU06, drivers/net/ethernet/freescale/fman/Peripherals/FM/Port/fm_port.c:3548 FM_PORT_Disable]:

cpu6/6: FM-0-port-1g-TX-4: BMI or QMI is Busy. Port forced downcpu6/6:

 

I'm not sure what this means.  However, now the port status does show DOWN:

root@b4860qds:~# ifconfig

root@b4860qds:~# ip link

1: lo: <LOOPBACK> mtu 65536 qdisc noqueue state DOWN mode DEFAULT group default

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: fm1-mac5: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

    link/ether 00:04:9f:00:a9:04 brd ff:ff:ff:ff:ff:ff

3: fm1-mac6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

    link/ether 00:04:9f:00:a9:05 brd ff:ff:ff:ff:ff:ff

4: fm1-mac9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

    link/ether 00:04:9f:00:a9:08 brd ff:ff:ff:ff:ff:ff

5: fm1-mac10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

    link/ether 00:04:9f:00:a9:09 brd ff:ff:ff:ff:ff:ff

6: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default

    link/ipip 0.0.0.0 brd 0.0.0.0

7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default

    link/sit 0.0.0.0 brd 0.0.0.0

 

Here is what happens if I then try to re-enable the port:

root@b4860qds:~# ifup -a

udhcpc (v1.22.1) started

Sending discover...

Sending discover...

Sending discover...

No lease, failing

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:2592 (2.5 KiB)

          Memory:fe4e8000-fe4e8fff

 

If I set the port back to DOWN status using the ip command, and change the port configuration to static, so that the /etc/network/interfaces file looks like this:

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface

auto lo

iface lo inet loopback

auto fm1-mac5

iface fm1-mac5 inet static

    address 192.168.5.200

    netmask 255.255.255.0

    gateway 192.168.5.1

 

...then set the port back up, I get the following:

root@b4860qds:~# ifup -a

ifup: interface lo already configured

IPv6: ADDRCONF(NETDEV_UP): fm1-mac5: link is not ready

root@b4860qds:~# IPv6: ADDRCONF(NETDEV_CHANGE): fm1-mac5: link becomes ready

 

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          inet addr:192.168.5.200  Bcast:192.168.5.255  Mask:255.255.255.0

          inet6 addr: fe80::204:9fff:fe00:a904/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:3240 (3.1 KiB)

          Memory:fe4e8000-fe4e8fff

 

The port appears to be UP, but a ping doesn't work:

root@b4860qds:~# ping 192.168.5.20

PING 192.168.5.20 (192.168.5.20) 56(84) bytes of data.

From 192.168.5.200 icmp_seq=1 Destination Host Unreachable

From 192.168.5.200 icmp_seq=2 Destination Host Unreachable

From 192.168.5.200 icmp_seq=3 Destination Host Unreachable

From 192.168.5.200 icmp_seq=4 Destination Host Unreachable

From 192.168.5.200 icmp_seq=5 Destination Host Unreachable

From 192.168.5.200 icmp_seq=6 Destination Host Unreachable

 

...and the port operational state appears to be the same as when DHCP was used (no packet reception):

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          inet addr:192.168.5.200  Bcast:192.168.5.255  Mask:255.255.255.0

          inet6 addr: fe80::204:9fff:fe00:a904/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:29 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:3618 (3.5 KiB)

          Memory:fe4e8000-fe4e8fff

 

lo        Link encap:Local Loopback 

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:65536  Metric:1

          RX packets:8 errors:0 dropped:0 overruns:0 frame:0

          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:896 (896.0 B)  TX bytes:896 (896.0 B)

 

root@b4860qds:~# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: fm1-mac5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether 00:04:9f:00:a9:04 brd ff:ff:ff:ff:ff:ff

    inet 192.168.5.200/24 brd 192.168.5.255 scope global fm1-mac5

       valid_lft forever preferred_lft forever

    inet6 fe80::204:9fff:fe00:a904/64 scope link

       valid_lft forever preferred_lft forever

3: fm1-mac6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether 00:04:9f:00:a9:05 brd ff:ff:ff:ff:ff:ff

4: fm1-mac9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether 00:04:9f:00:a9:08 brd ff:ff:ff:ff:ff:ff

5: fm1-mac10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether 00:04:9f:00:a9:09 brd ff:ff:ff:ff:ff:ff

6: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default

    link/ipip 0.0.0.0 brd 0.0.0.0

7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default

    link/sit 0.0.0.0 brd 0.0.0.0

 

As I mentioned above, this is an intermittent problem.  Here is the situation following the next reboot:

...

Running postinst /etc/rpm-postinsts/104-inetutils-telnetd...

Running postinst /etc/rpm-postinsts/105-inetutils-tftpd...

Running postinst /etc/rpm-postinsts/106-inetutils-ftpd...

update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)

INIT: Entering runlevel: 5

Configuring network interfaces... udhcpc (v1.22.1) started

Sending discover...

Sending select for 192.168.5.109...

Lease of 192.168.5.109 obtained, lease time 86400

/etc/udhcpc.d/50default: Adding DNS 192.168.5.1

done.

Starting system message bus: dbus.

Starting OpenBSD Secure Shell server: sshd

  generating ssh RSA key...

  generating ssh ECDSA key...

  generating ssh DSA key...

  generating ssh ED25519 key...

done.

Starting network benchmark server: netserver.

Starting system log daemon...0

Starting kernel log daemon...0

Starting internet superserver: xinetd.

Stopping Bootlog daemon: bootlogd.

 

fsl-networking (QorIQ SDK) QorIQ-SDK-V1.8 b4860qds /dev/ttyS0

 

b4860qds login: root

root@b4860qds:~# ifconfig

fm1-mac5  Link encap:Ethernet  HWaddr 00:04:9f:00:a9:04 

          inet addr:192.168.5.109  Bcast:192.168.5.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:126 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:11918 (11.6 KiB)  TX bytes:996 (996.0 B)

          Memory:fe4e8000-fe4e8fff

 

lo        Link encap:Local Loopback 

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:65536  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

Note that a lease was obtained from DHCP on the first discover.

 

This is an extremely annoying issue.  Any help or even clues to help solve the issue would be greatly appreciated.

 

Thanks.

 

--ken

Outcomes