Dear NXP Team,
We are experiencing an issue when trying to use all 16 DPMACs (DPMAC3-18) simultaneously in Linux on our custom board:
Details:
One of the ports (DPMAC18) is not working correctly: ingress and egress data are not flowing, and both RX/TX "frame discards" MIB counters are increasing.
Here are the relevant NIC statistics from ethtool:
# ethtool -S eth1 | grep -v ": 0"
NIC statistics:
[hw] tx discarded frames: 34
[hw] tx confirmed frames: 34
[hw] tx dequeued bytes: 1804
[hw] tx dequeued frames: 34
[drv] tx conf frames: 34
[drv] tx conf bytes: 1804
[drv] tx sg frames: 34
[drv] tx sg bytes: 1804
[drv] tx converted sg frames: 34
[drv] tx converted sg bytes: 1804
[drv] cdan: 34
[qbman] buffer count: 10248
[mac] rx 64 bytes: 33
[mac] rx frame discards: 2
[mac] rx bytes: 1984
[mac] rx b-cast: 31
[mac] rx all frames: 33
[mac] rx frames ok: 31
There are no errors in dmesg or the MC log (/dev/dpaa2_mc_console), but we do observe the following warnings in the MC log:
[W, PLATFORM] UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
[W] [PFS ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
[W] [PFS ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
We have followed the "Network Subsystem Troubleshooting on DPAA2" guide from this document, and tried the following steps:
1) We checked all 16 ports in u-boot, and they are working properly:
u-boot> setenv ethact DPMAC3@sgmii ; ping 10.0.0.2
Using DPMAC3@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC4@sgmii ; ping 10.0.0.2
Using DPMAC4@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC5@sgmii ; ping 10.0.0.2
Using DPMAC5@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC6@sgmii ; ping 10.0.0.2
Using DPMAC6@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC7@sgmii ; ping 10.0.0.2
Using DPMAC7@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC8@sgmii ; ping 10.0.0.2
Using DPMAC8@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC9@sgmii ; ping 10.0.0.2
Using DPMAC9@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC10@sgmii ; ping 10.0.0.2
Using DPMAC10@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC11@sgmii ; ping 10.0.0.2
Using DPMAC11@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC12@sgmii ; ping 10.0.0.2
Using DPMAC12@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC13@sgmii ; ping 10.0.0.2
Using DPMAC13@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC14@sgmii ; ping 10.0.0.2
Using DPMAC14@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC15@sgmii ; ping 10.0.0.2
Using DPMAC15@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC16@sgmii ; ping 10.0.0.2
Using DPMAC16@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC17@sgmii ; ping 10.0.0.2
Using DPMAC17@sgmii device
host 10.0.0.2 is alive
u-boot> setenv ethact DPMAC18@sgmii ; ping 10.0.0.2
Using DPMAC18@sgmii device
host 10.0.0.2 is alive
2) We tried to start Linux firmware with minimal DPL configuration (that contains only 40 DPMCPs) and configure DPL in run-time with "restool" and "ls-addni" utilities, the result is still the same, one port is not working properly, though we found out that the problem is not related to one specific port: port with DPMAC that was initialized last with restool dpmac create --mac-id=X is the one that is not working. For example: if we initialize DPMAC3 last, a port with this DPMAC will not work, and a port with DPMAC18 will work without any problems. We also found out that two warnings in the MC log appear only after 16-th DPMAC is initialized:
[W, PLATFORM] UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
[W, PLATFORM] UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
root@TinyLinux:~# cat /dev/dpaa2_mc_console
[W, PLATFORM] UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
root@TinyLinux:~# restool dpmac create --mac-id=18
root@TinyLinux:~# restool dprc assign dprc.1 --object=dpmac.18 --plugged=1
root@TinyLinux:~# root@TinyLinux:~# root@TinyLinux:~# ls-addni -nq=8 -t=1 dpmac.18
root@TinyLinux:~# cat /dev/dpaa2_mc_console
[W, PLATFORM] UART print failed, all debug data will be printed to buffer.
Running MC app, waiting for events ...
[W] [PFS ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
[W] [PFS ] val=0x0000002f got=0x00000001 Port FIFO size not configured, please reduce FIFO allocated on recycle ports
3) We checked out the logs from ifconfig, restool dpni, restool dpmac commands
about non-working port as guide suggested:
root@TinyLinux:~# sudo ping 10.0.0.2 -I eth1 -c 5
PING 10.0.0.2 (10.0.0.2): 56 data bytes
--- 10.0.0.2 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@TinyLinux:~# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ae:27:42:ac:48:16 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 14 bytes 908 (908.0 B)
TX errors 14 dropped 0 overruns 0 carrier 0 collisions 0
root@TinyLinux:~# restool dpmac info dpmac.18
dpmac version: 4.10
dpmac object id/portal id: 18
plugged state: plugged
endpoint state: 1
endpoint: dpni.0, link is up
DPMAC link type: DPMAC_LINK_TYPE_BACKPLANE
DPMAC ethernet interface: DPMAC_ETH_IF_SGMII
MAC address: ae:27:42:ac:48:16
maximum supported rate 1000 Mbps
Counters:
rx all frames: 2
rx frames ok: 2
rx frame errors: 0
rx frame discards: 0
rx u-cast: 0
rx b-cast: 2
rx m-cast: 0
rx 64 bytes: 0
rx 65-127 bytes: 0
rx 128-255 bytes: 1
rx 256-511 bytes: 1
rx 512-1023 bytes: 0
rx 1024-1518 bytes: 0
rx 1519-max bytes: 0
rx frags: 0
rx jabber: 0
rx align errors: 0
rx oversized: 0
rx pause: 0
rx bytes: 531
tx frames ok: 0
tx u-cast: 0
tx m-cast: 0
tx b-cast: 0
tx frame errors: 0
tx undersized: 0
tx b-pause: 0
tx bytes: 0
root@TinyLinux:~# restool dpni info dpni.0
dpni version: 8.5
dpni id: 0
plugged state: plugged
endpoint state: 1
endpoint: dpmac.18, link is up
link status: 1 - up
mac address: ae:27:42:ac:48:16
max frame length: 10240
dpni_attr.options value is: 0
num_queues: 8
num_cgs: 1
num_rx_tcs: 1
num_tx_tcs: 1
mac_entries: 16
vlan_entries: 0
qos_entries: 0
fs_entries: 64
qos_key_size: 0
fs_key_size: 56
ingress_all_frames: 0
ingress_all_bytes: 0
ingress_multicast_frames: 0
ingress_multicast_bytes: 0
ingress_broadcast_frames: 0
ingress_broadcast_bytes: 0
egress_all_frames: 0
egress_all_bytes: 0
egress_multicast_frames: 0
egress_multicast_bytes: 0
egress_broadcast_frames: 0
egress_broadcast_bytes: 0
ingress_filtered_frames: 0
ingress_discarded_frames: 0
ingress_nobuffer_discards: 0
egress_discarded_frames: 14
egress_confirmed_frames: 14
ceetm_dequeue_bytes: 908
ceetm_dequeue_frames: 14
ceetm_reject_bytes: 0
ceetm_reject_frames: 0
cgr_reject_frames: 0
cgr_reject_bytes: 0
policer_cnt_red: 0
policer_cnt_yellow: 0
policer_cnt_green: 0
policer_cnt_re_red: 0
policer_cnt_re_yellow: 0
tx_pending_frames_cnt: 0
We have attached a detailed MC log (captured using ls-debug --log=on --console=on --level=2) for your review.
Could you please assist us in identifying and resolving the issue?
已解决! 转到解答。
Thank you for sharing the application note to help resolve the issue.
In the document, it states: "The total sum of ports' FIFOs must not exceed the total FIFO size available. The FIFO size is device specific. Check Device specific features."
Could you please clarify where we can find the Total FIFO size information specifically for our LX2160A processor?
Thank you!
We have reviewed the LX2160ADPAA2RM.pdf document and identified 264 KB (0x42000 bytes) available for ingress FIFO, as well as 264 KB available for egress FIFO. Additionally, we noted the existence of two internal "recycle" ports (PPID 19-20) and an additional internal 1G "management command" port (PPID 0).
Our egress calculations are as follows:
Starting with 0x42000 bytes for egress FIFO:
PPID 0 ("management command" port) will require: 0x100 * (0x2F + 0x1) = 0x3000 bytes.
PPID 19-20 ("recycle" ports) will require:
Thus, the remaining egress FIFO available for normal Ethernet ports (PPID 1-18) would be:
Could you please confirm that our FIFO calculations are correct?
Thank you for your assistance.
Thank you for your previous response regarding our FIFO calculations. We have reviewed the section "7.6 WRIOP Ports" in the LX2160ADPAA2RM document, as advised. However, we are still unclear on a few points.
Could you please elaborate on what specific aspects we should consider from this section regarding our egress FIFO calculations? Specifically, we would like to understand how the WRIOP port configuration and frame descriptors affect the available FIFO memory for Ethernet ports (PPID 1-18).
We would appreciate any additional details or examples to clarify how FIFO allocation differs between CTLU and WRIOP, or any adjustments we should make based on WRIOP's requirements.
Thank you for your assistance.