Lx2160 is connected to another switch using dpaa2 switch, mac address table of the switch may shift

[liushuangxin]

hello,nxp:

Currently we are using lsdk20.04, linux5.4.3, using the Ethernet Switch function of dpaa2, dpmac.13 and dpmac.14 as external connection ports, and dpni.3 to connect the kernel protocol stack. To configure this function, use the restool at the application layer according to the LSDK 20.04 document. The configuration process is as follows:

ls-addni -n
ls-addsw -i=3 dpmac.13 dpmac.14 dpni.3
ip link add name br1 type bridge
ip link set br1 up

ip link set eth4 down
ip link set eth4 address xx:xx:xx:xx:xx:xx
ip link set eth4 up
ip link set eth4 master br1

ip link set eth5 down
ip link set eth5 address xx:xx:xx:xx:xx:xx
ip link set eth5 up
ip link set eth5 master br1

ip link set eth6 down
ip link set eth6 address xx:xx:xx:xx:xx:xx
ip link set eth6 up
ip link set eth6 master br1

At present, the traffic forwarding function of the dpaa2 switch has been implemented. However, we enable the dpaa2 switch function on multiple boards and connect only the port of dpmac.13 to the H3C switch. For details about the Network topology, refer to the attached network topological.png. If we hot-swap the optical module connected to dpmac.13, we can find that the mac table is offset from the serial port of H3C switch.The serial port logs are attached as h3c_switch_mac_table.txt and H3C_Switch_mac_mov.txt.

And as you can see in the fdb table of eth4 in lx2160, other mac addresses have been learned.We use wireshark to find that the eth4 port receives traffic and forwards it back from the eth4 port. This problem causes the network of the entire switch to break down.This phenomenon also occurs when we configure nxp lx2160 demo borad to dpaa2 switch mode.

At present, we have not found other ways to avoid this problem, can you provide some good suggestions and ways to avoid this problem? thank you。

[yipingwang]

Could you please share MC log (debug level) and output of below commands.
restool dpsw info dpsw.0
restool dpmac info dpmac.13
restool dpmac info dpmac.14
restool dpni info dpni.0

[liushuangxin]

Hello yiping,
What modules affect dpsw in dpaa2? I have updated mc firmware to 10.38.0 and linux kernel 5.10.74, and the problem still exists. We need to realize this function on our products, can you provide some ideas to solve this problem? Thank you

[yipingwang]

Please refer to the following update from the AE team.

I checked the log, it seems there is no MC error info (will be marked as "E, DPMAC" or "E, DPSW").
Due to all LX2160ARDB boards had been replaced by new-design LX2160ARDB boards, and there are some issues on 10G ports.
So i can't reproduce the issue.
Is there other way to reproduce the issue except plugged/unplugged the cable?

Did customer get some STP info as below output?

root@localhost:~# restool dpsw info dpsw.0
dpsw version: 8.9
dpsw id: 0
plugged state: plugged
endpoints:
interface 0:
connection: dpmac.3
link state: up
Taildrop enabled: false
Taildrop units: BUFFERS
Taildrop threshold: 192
max frame length: 1536
dpsw_cnt_ing_frame: 25
dpsw_cnt_ing_byte: 2436
dpsw_cnt_ing_fltr_frame: 0
dpsw_cnt_ing_frame_discard: 10
dpsw_cnt_ing_mcast_frame: 0
dpsw_cnt_ing_mcast_byte: 0
dpsw_cnt_ing_bcast_frame: 1
dpsw_cnt_ing_bcast_bytes: 64
dpsw_cnt_egr_frame: 25
dpsw_cnt_egr_byte: 2436
dpsw_cnt_egr_frame_discard: 0
dpsw_cnt_egr_stp_frame_discard: 0
dpsw_cnt_ing_no_buffer_discard: 0

[yipingwang]

Now all LX2160ARDB board already replaced by new-design board in LAB.

And the 10G ports didn't work due to some issues.

could you please share the network topology, so i can try to reproduce it with single board.

[liushuangxin]

The condition of reproduction is not only to remove and insert the optical fiber, but also to restart the board.Is your environment the same as what I'm offering Network topology.png is similar？ At least two boards are required. And my restool dpsw info shows the following:

root@root:~# restool dpsw info dpsw.0
state: plugged
endpoints:
interface 0:
connection: dpmac.13
link state: up
interface 1:
connection: dpmac.14
link state: down
interface 2:
connection: dpni.3
link state: up
dpsw_attr.options value is: 0
max VLANs: 16
max FDBs: 1
frame storage memory size: 1152
number of interfaces: 3
current number of VLANs: 1
current number of FDBs: 1

Different from what you provided，Is it because my restool version is too low? Currently restool 1.6。

At the moment, my biggest doubt can be seen from restool dpmac info dpmac.13, in the case of an exception: rx bytes ≈ tx bytes, and we can see from the packet capture of the switch that a flow that does not belong to this port enters through dpmac.13 and forwards it out through dpmac.13。

DPMAC ethernet interface: DPMAC_ETH_IF_XFI
maximum supported rate 10000 Mbps
Counters:
rx all frames: 265578986
rx frames ok: 265578986
rx frame errors: 0
rx frame discards: 0
rx u-cast: 265578513
rx b-cast: 232
rx m-cast: 241
rx 64 bytes: 30997890
rx 65-127 bytes: 233667301
rx 128-255 bytes: 10
rx 256-511 bytes: 16
rx 512-1023 bytes: 2766
rx 1024-1518 bytes: 911004
rx 1519-max bytes: 0
rx frags: 0
rx jabber: 0
rx align errors: 0
rx oversized: 0
rx pause: 0
rx bytes: 27186661770
tx frames ok: 265565295
tx u-cast: 265565289
tx m-cast: 6
tx b-cast: 0
tx frame errors: 0
tx undersized: 0
tx b-pause: 0
tx bytes: 27184470445

[yipingwang]

We are investigating this problem, will provide update later.

[liushuangxin]

Hello yiping,
Thank you for your reply. After the switch is enabled on the board, the network adapter ports are as follows:

root@root:~# ls-listni
dprc.1/dpni.3 (interface: eth3, end point: dpsw.0.2)
dprc.1/dpni.2 (interface: eth0, end point: dpmac.12)
dprc.1/dpni.1 (interface: eth1, end point: dpmac.16)
dprc.1/dpni.0 (interface: eth2, end point: dpmac.17)
root@root:~#
root@root:~#
root@root:~# ifconfig
br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::b46f:17ff:fe01:4062 prefixlen 64 scopeid 0x20<link>
inet6 fe80::45fe:859e:8778:23f7 prefixlen 64 scopeid 0x20<link>
ether 3e:72:68:41:9e:6e txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10 bytes 1040 (1.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.21.100.100 netmask 255.255.255.0 broadcast 192.21.100.255
inet6 fe80::1562:fb44:623b:e842 prefixlen 64 scopeid 0x20<link>
ether 12:2b:c3:55:78:35 txqueuelen 1000 (Ethernet)
RX packets 171 bytes 23182 (23.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 146 bytes 17554 (17.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.21.60.100 netmask 255.255.255.0 broadcast 192.21.60.255
ether 32:4e:e7:fa:8c:d9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.5.215 netmask 255.255.255.0 broadcast 192.168.5.255
ether b2:a5:c1:33:7d:c9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::133a:dd23:19df:4404 prefixlen 64 scopeid 0x20<link>
ether ca:d6:a1:f3:23:af txqueuelen 1000 (Ethernet)
RX packets 293 bytes 20119 (20.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10 bytes 1128 (1.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3d84:93ba:28d:1a83 prefixlen 64 scopeid 0x20<link>
ether 66:1a:ce:38:27:d2 txqueuelen 1000 (Ethernet)
RX packets 13481 bytes 1244770 (1.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12831 bytes 1177146 (1.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth5: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet6 fe80::a325:d7b1:39d4:ec06 prefixlen 64 scopeid 0x20<link>
ether 5a:b8:ae:cf:49:43 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::4c80:c8cd:6a99:8604 prefixlen 64 scopeid 0x20<link>
ether 3e:72:68:41:9e:6e txqueuelen 1000 (Ethernet)
RX packets 6 bytes 540 (540.0 B)
RX errors 0 dropped 2 overruns 0 frame 0
TX packets 654 bytes 67977 (67.9 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1364 bytes 96740 (96.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1364 bytes 96740 (96.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@root:~#

Eth3 is connected to dpni.3, and eth4, eth5, eth6 are virtual nics for switch. Eth4 connects to dpmac.13, eth5 connects to dpmac.14, and eth6 connects to dpni.3. Dpmac.13 and dpmac.14 Connect to external optical modules.
In the current environment, only dpmac13 connects to the H3C switch, and No traffic is sent to eth3 of the board.
When the optical port of dpmac.13 was remove and insert, the entire network crashed. As you can see from switch_exception_log.txt, the eth4 interface sends and receives the same data, about 27GB, but no traffic is actually sent to the board.
Some detailed logs about restool dpni.3 restool dpmac.13 restool dpmac.14 restool dpsw.0 are in the following two txt files, please check.