DPMAC Issue on LX2160A with 16 SFP Ethernet Ports

[o_m_v]

Dear NXP Team,

We are experiencing an issue when trying to use all 16 DPMACs (DPMAC3-18) simultaneously in Linux on our custom board:

Details:

• CPU: LX2160A
• Board: Custom board with 16 1000BaseX SFP Ethernet ports (DPMAC3-18)
• SerDes1 Protocol: 4 (8x 1G Ethernet)
• SerDes2 Protocol: 9 (8x 1G Ethernet)
• SerDes3 Protocol: 3 (2x PCIe)
• Software: Bootloaders and Linux firmware are based on "LSDK-21.08" branches
• MC Firmware: v10.39.0

One of the ports (DPMAC18) is not working correctly: ingress and egress data are not flowing, and both RX/TX "frame discards" MIB counters are increasing.

Here are the relevant NIC statistics from ethtool:

# ethtool -S eth1 | grep -v ": 0"
NIC statistics:
     [hw] tx discarded frames: 34
     [hw] tx confirmed frames: 34
     [hw] tx dequeued bytes: 1804
     [hw] tx dequeued frames: 34
     [drv] tx conf frames: 34
     [drv] tx conf bytes: 1804
     [drv] tx sg frames: 34
     [drv] tx sg bytes: 1804
     [drv] tx converted sg frames: 34
     [drv] tx converted sg bytes: 1804
     [drv] cdan: 34
     [qbman] buffer count: 10248
     [mac] rx 64 bytes: 33
     [mac] rx frame discards: 2
     [mac] rx bytes: 1984
     [mac] rx b-cast: 31
     [mac] rx all frames: 33
     [mac] rx frames ok: 31

There are no errors in dmesg or the MC log (/dev/dpaa2_mc_console), but we do observe the following warnings in the MC log:

[W, PLATFORM]  UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
[W]  [PFS   ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
[W]  [PFS   ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports

We have followed the "Network Subsystem Troubleshooting on DPAA2" guide from this document, and tried the following steps:

1) We checked all 16 ports in u-boot, and they are working properly:

u-boot> setenv ethact DPMAC3@sgmii ; ping 10.0.0.2             
Using DPMAC3@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC4@sgmii ; ping 10.0.0.2             
Using DPMAC4@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC5@sgmii ; ping 10.0.0.2             
Using DPMAC5@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC6@sgmii ; ping 10.0.0.2
Using DPMAC6@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC7@sgmii ; ping 10.0.0.2
Using DPMAC7@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC8@sgmii ; ping 10.0.0.2
Using DPMAC8@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC9@sgmii ; ping 10.0.0.2
Using DPMAC9@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC10@sgmii ; ping 10.0.0.2
Using DPMAC10@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC11@sgmii ; ping 10.0.0.2
Using DPMAC11@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC12@sgmii ; ping 10.0.0.2
Using DPMAC12@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC13@sgmii ; ping 10.0.0.2
Using DPMAC13@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC14@sgmii ; ping 10.0.0.2
Using DPMAC14@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC15@sgmii ; ping 10.0.0.2
Using DPMAC15@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC16@sgmii ; ping 10.0.0.2
Using DPMAC16@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC17@sgmii ; ping 10.0.0.2
Using DPMAC17@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC18@sgmii ; ping 10.0.0.2
Using DPMAC18@sgmii device
host 10.0.0.2 is alive

2) We tried to start Linux firmware with minimal DPL configuration (that contains only 40 DPMCPs) and configure DPL in run-time with "restool" and "ls-addni" utilities, the result is still the same, one port is not working properly, though we found out that the problem is not related to one specific port: port with DPMAC that was initialized last with restool dpmac create --mac-id=X is the one that is not working. For example: if we initialize DPMAC3 last, a port with this DPMAC will not work, and a port with DPMAC18 will work without any problems. We also found out that two warnings in the MC log appear only after 16-th DPMAC is initialized:

• MC log after booting with minimal DPL configuration:
  

[W, PLATFORM]  UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...

• MC log after 15 ports (DPMAC3-17) were configured with commands
  ex restool dpmac create --mac-id=X,
  ex restool dprc assign dprc.1 --object=dpmac.X --plugged=1,
  ex ls-addni -nq=8 -t=1 dpmac.X:

[W, PLATFORM]  UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...​

• MC log after 16-th port is configured with same commands:

root@TinyLinux:~# cat /dev/dpaa2_mc_console 
[W, PLATFORM]  UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
root@TinyLinux:~# restool dpmac create --mac-id=18
root@TinyLinux:~# restool dprc assign dprc.1 --object=dpmac.18 --plugged=1
root@TinyLinux:~# root@TinyLinux:~# root@TinyLinux:~# ls-addni -nq=8 -t=1 dpmac.18
root@TinyLinux:~# cat /dev/dpaa2_mc_console 
[W, PLATFORM]  UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
[W]  [PFS   ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
[W]  [PFS   ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports​

3) We checked out the logs from ifconfig, restool dpni, restool dpmac commands
about non-working port as guide suggested:

root@TinyLinux:~# sudo ping 10.0.0.2 -I eth1 -c 5
PING 10.0.0.2 (10.0.0.2): 56 data bytes

--- 10.0.0.2 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@TinyLinux:~# ifconfig eth1
eth1:   flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether ae:27:42:ac:48:16  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 908 (908.0 B)
        TX errors 14  dropped 0 overruns 0  carrier 0  collisions 0

root@TinyLinux:~# restool dpmac info dpmac.18
dpmac version: 4.10
dpmac object id/portal id: 18
plugged state: plugged
endpoint state: 1
endpoint: dpni.0, link is up
DPMAC link type: DPMAC_LINK_TYPE_BACKPLANE
DPMAC ethernet interface: DPMAC_ETH_IF_SGMII
MAC address: ae:27:42:ac:48:16
maximum supported rate 1000 Mbps
Counters:
rx all frames: 2
rx frames ok: 2
rx frame errors: 0
rx frame discards: 0
rx u-cast: 0
rx b-cast: 2
rx m-cast: 0
rx 64 bytes: 0
rx 65-127 bytes: 0
rx 128-255 bytes: 1
rx 256-511 bytes: 1
rx 512-1023 bytes: 0
rx 1024-1518 bytes: 0
rx 1519-max bytes: 0
rx frags: 0
rx jabber: 0
rx align errors: 0
rx oversized: 0
rx pause: 0
rx bytes: 531
tx frames ok: 0
tx u-cast: 0
tx m-cast: 0
tx b-cast: 0
tx frame errors: 0
tx undersized: 0
tx b-pause: 0
tx bytes: 0

root@TinyLinux:~# restool dpni info dpni.0
dpni version: 8.5
dpni id: 0
plugged state: plugged
endpoint state: 1
endpoint: dpmac.18, link is up
link status: 1 - up
mac address: ae:27:42:ac:48:16
max frame length: 10240
dpni_attr.options value is: 0
num_queues: 8
num_cgs: 1
num_rx_tcs: 1
num_tx_tcs: 1
mac_entries: 16
vlan_entries: 0
qos_entries: 0
fs_entries: 64
qos_key_size: 0
fs_key_size: 56
ingress_all_frames: 0
ingress_all_bytes: 0
ingress_multicast_frames: 0
ingress_multicast_bytes: 0
ingress_broadcast_frames: 0
ingress_broadcast_bytes: 0
egress_all_frames: 0
egress_all_bytes: 0
egress_multicast_frames: 0
egress_multicast_bytes: 0
egress_broadcast_frames: 0
egress_broadcast_bytes: 0
ingress_filtered_frames: 0
ingress_discarded_frames: 0
ingress_nobuffer_discards: 0
egress_discarded_frames: 14
egress_confirmed_frames: 14
ceetm_dequeue_bytes: 908
ceetm_dequeue_frames: 14
ceetm_reject_bytes: 0
ceetm_reject_frames: 0
cgr_reject_frames: 0
cgr_reject_bytes: 0
policer_cnt_red: 0
policer_cnt_yellow: 0
policer_cnt_green: 0
policer_cnt_re_red: 0
policer_cnt_re_yellow: 0
tx_pending_frames_cnt: 0

We have attached a detailed MC log (captured using ls-debug --log=on --console=on --level=2) for your review.

Could you please assist us in identifying and resolving the issue?

[LFGP]

dear @o_m_v , please use the next application note to resolve the issue 

https://www.nxp.com/docs/en/application-note/AN13684.pdf 

[o_m_v]

Thank you for sharing the application note to help resolve the issue.

In the document, it states: "The total sum of ports' FIFOs must not exceed the total FIFO size available. The FIFO size is device specific. Check Device specific features."

Could you please clarify where we can find the Total FIFO size information specifically for our LX2160A processor?

Thank you!

[LFGP]

you can find it at the LX2160ADPAA2 reference manual,
use the below link and go ahead to the "Documentation" section.
https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/layerscape-processors/la...

[o_m_v]

We have reviewed the LX2160ADPAA2RM.pdf document and identified 264 KB (0x42000 bytes) available for ingress FIFO, as well as 264 KB available for egress FIFO. Additionally, we noted the existence of two internal "recycle" ports (PPID 19-20) and an additional internal 1G "management command" port (PPID 0).

Our egress calculations are as follows:

Starting with 0x42000 bytes for egress FIFO:

• PPID 0 ("management command" port) will require: 0x100 * (0x2F + 0x1) = 0x3000 bytes.

• PPID 19-20 ("recycle" ports) will require:

  • 1G: 2 * 0x100 * (0x2F + 0x1) = 0x6000 bytes;
  • 10G: 2 * 0x100 * (0x3F + 0x1) = 0x8000 bytes;
  • 50G: 2 * 0x100 * (0x7F + 0x1) = 0x10000 bytes.

Thus, the remaining egress FIFO available for normal Ethernet ports (PPID 1-18) would be:

• 1G recycle: 0x42000 - 0x3000 - 0x6000 = 0x39000 bytes;
• 10G recycle: 0x42000 - 0x3000 - 0x8000 = 0x37000 bytes;
• 50G recycle: 0x42000 - 0x3000 - 0x10000 = 0x2F000 bytes.

Could you please confirm that our FIFO calculations are correct?

Thank you for your assistance.

[LFGP]

dear o_m_v,
the FIFO size you related is for the CTLU ""Classifier and Table Lookup Unit"".
please review, in the LX2160ADPAA2RM, the "7.6 WRIOP Ports" subject and the related to frame descriptors.

[o_m_v]

Thank you for your previous response regarding our FIFO calculations. We have reviewed the section "7.6 WRIOP Ports" in the LX2160ADPAA2RM document, as advised. However, we are still unclear on a few points.

Could you please elaborate on what specific aspects we should consider from this section regarding our egress FIFO calculations? Specifically, we would like to understand how the WRIOP port configuration and frame descriptors affect the available FIFO memory for Ethernet ports (PPID 1-18).

We would appreciate any additional details or examples to clarify how FIFO allocation differs between CTLU and WRIOP, or any adjustments we should make based on WRIOP's requirements.

Thank you for your assistance.

[LFGP]

dear o_m_v
in order to have a better grasp about the FIFO size and frames please review the Chapter 3, “Frame Descriptor and Frame Description.” also please review the Table 7-5. Frame Descriptor Fields.
you can review the DPAA driver to understood the FIFO allocation
https://github.com/nxp-imx/linux-imx/tree/29549c7073bf72cfb2c4614d37de45ec36b60475/drivers/net/ether...

[LFGP]

I need more info from your side in order to investigate the issue, please review my message on your email