Processor Expert Software中的主题 Re: iMX8MP- bridging eth1 causes unresponsive state

iMX8MP- bridging eth1 causes unresponsive state

magnus3 — Mon, 22 May 2023 10:43:23 GMT

Hello,

We are having an issue with bridging the eth1 interface. The title explains the gist of it, but details below.

Prerequisites:

We have a system image built with yocto for iMX8MP with an applied patch: https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/Workaround-for-issue-Bridge-mode-on-EQoS-module-will-not-work/ta-p/1559302

Further details here: https://variwiki.com/index.php?title=DART-MX8M-PLUS_Release_Notes&release=mx8mp-yocto-kirkstone-5.15-2.0.x-v1.2

All network applications are disabled.

Problem:

Running these two commands causes the system to enter an unresposive state:

ip link add br0 type bridge
ip link set eth1 master br0

However if we do:

ip link set eth1 up
ip link add br0 type bridge
ip link set eth1 master br0

We get

RTNETLINK answers: Device or resource busy

and the following kernel output and no crash:

[   36.072998] imx-dwmac 30bf0000.ethernet eth1: Timeout accessing MAC_VLAN_Tag_Filter
[   36.073086] imx-dwmac 30bf0000.ethernet eth1: Timeout accessing MAC_VLAN_Tag_Filter
[   36.096058] imx-dwmac 30bf0000.ethernet eth1: failed to initialize vlan filtering on this port

Furthermore:

If we connect an RJ45 to the physical port and run the same commands we receive no indication of fault and the link works fine!

eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN group default qlen 1000

Can we get assistance as to what causes these weird interactions and possibly a solution to the first issue i.e. the crash?

Re: iMX8MP- bridging eth1 causes unresponsive state

Chavira — Tue, 23 May 2023 15:36:26 GMT

Hi @magnus3 !

Thank you for contacting NXP Support!

The procedure that you are trying is correct when you have a physical switch layer.

The iMX8MP EVK has two independent Ethernet ports and doesn't have a switch layer between the ports.

I think is possible to do a bridge layer with software, but we don't have the program or examples.

I have tried the commands that are you using and if you are connecting with the board using ssh the communication hangs because the board is trying to connect the ethernet ports, but if you connect the board via USB doesn't hangs for ethernet configurations first use the debug port and then you can use the ssh server to connect the board.

Best Regards!

Chavira

Re: iMX8MP- bridging eth1 causes unresponsive state

magnus3 — Wed, 24 May 2023 07:14:56 GMT

Hello @Chavira,

Are you saying that you managed to get it working while connected on the debug port? Because that is what we have been using across all these tests.
When I looked elsewhere on the forum, I saw patches advertised as a solution for those issues. Could that be a solution for us as well?
E.g. https://community.nxp.com/t5/Processor-Expert-Software/iMX8MP-Bridge-on-eth1-not-supported/m-p/1599252/highlight/true#M5472

Thank you for your assistance!

Best regards,
Magnus

Re: iMX8MP- bridging eth1 causes unresponsive state

Chavira — Wed, 24 May 2023 12:46:10 GMT

Hi @magnus3 !

That patch is only for Android bsp if you need that patch, I can send you that patch via email.

Best Regards.

Chavira

Re: iMX8MP- bridging eth1 causes unresponsive state

magnus3 — Wed, 24 May 2023 13:34:28 GMT

Hi @Chavira,

I would appreciate that, and if you have any other suggestions on how to resolve our issue, I would love to hear them.

Best regards,
Magnus

Re: iMX8MP- bridging eth1 causes unresponsive state

Chavira — Wed, 24 May 2023 15:00:21 GMT

Hi @magnus3 !

If you want to use the evk you have to implement the switch layer with software, if you are developing a custom board, I recommend reviewing the Layerscape Reference Design Boards, you can find a compatible part to iMX8MP and do the switch layer via hardware without complications.

Re: iMX8MP- bridging eth1 causes unresponsive state

btessele — Fri, 27 Oct 2023 07:42:56 GMT

@magnus3point here is absolutely correct. I have done some investigation in the kernel regarding a bridge connection, and got some insights:

Yes, all the bridging / vlan implementation is purely software, not really hardware dependent
However, at some point during the bridging, the MAC driver (hardware dependent) can be called to do a certain vlan configuration. And that's exactly where the error is happening, the EQoS eth1 MAC driver is messed up.

My kernel debugging showed me the following:

When running ip link set eth1 master br0
net/bridge/br_vlan.c calls net/8021q/vlan_core.c:vlan_vid_add which internally calls vlan_add_rx_filter_info

static int vlan_add_rx_filter_info(struct net_device *dev, __be16 proto, u16 vid) { if (!vlan_hw_filter_capable(dev, proto)) return 0; if (netif_device_present(dev)) return dev->netdev_ops->ndo_vlan_rx_add_vid(dev, proto, vid); else return -ENODEV; }

For the case of eth0 the first if matches (eth0 doesn't have vlan hw filter) and so the function returns successfully.
For the case of eth1 QoS, the MAC driver implementation of ndo_vlan_rx_add_vid gets called, which is drivers/net/ethernet/stmicro/stmmac/stmmac_main.c:stmmac_vlan_rx_add_vid()
That's when it gets interesting, within this function another internal call is made until the driver dwmac4 gets called, specifically drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c:dwmac4_update_vlan_hash() - this function does a bunch of writel (I believe that is writing to the Chip's MAC register addresses, but I might be wrong)
Now here, there are 3 possibilities:
- If the interface is down (ip l set eth1 down the kernel hangs on the first writel call, just hangs there forever, the whole system is frozen
- If the interface is up (but no link carrier): the writel operations go through, and the dw4mac eventually writes the hw filtering things - in dwmac4_write_vlan_filter - but it doesn't read back the just written values, and so it prints the line "Timeout accessing MAC_VLAN_Tag_Filter", and returns -EBUSY
- If the interface is up and there's link carrier: everything succeeds!

Summing up, there's something very wrong in these writes operations that's causing this weird freezing. I try to quick look at the writel implementation, but there're a bunch of different definitions in io.h

If this freezing state is not MAC driver related, then I think the driver should at least avoid performing those writes if the link is down, it's a simple check that I added, on the top of the Android patch mentioned.

Now regarding to userspace, systemd-networkd bridges the interfaces when they are down, that's why it freezes in early systemd boot. On the other hand, NetworkManager is a bit more clever, it actually doesn't bridge the interfaces right away, but it waits until the interface gets carrier (connected link) to perform the master operation.