LS 1046 : DPAA ( how to Debug Throughput throttle issue )

Chidananda22 — Fri, 03 Nov 2023 19:07:27 GMT

Hi ,

I am using LS1046 - DPAA .

Running my application and received data over DPAA.

Here I am getting data dip after every time interval.

Ex : data is ruunning fine for 2 mins . then data dip for 2 - 5 sec . Suddenly again data received in burst what are missing in 2-5 sec . Check Sender application and My receiver application , no clue found.

As per LSDK guide tried below still no clue why data dip after certain fixed interval

1. remove the fsl_fm_max_frm=9600 in boot args

2.ethtool -S <intf-name> for Statistics
3.perf output on core 0,1,2,3

can you please suggest how to debug further and any DPAA specific debugging command.

Thank You.

Re: LS 1046 : DPAA ( how to Debug Throughput throttle issue )

yipingwang — Wed, 15 Nov 2023 03:11:32 GMT

Please check FMAN RX port counters and the RX MAC counters during the error occurs.

"For debugging I will put here the MAC base addresses.

The mappings within FMD (FMAN kernel driver) can be found here: https://community.nxp.com/t5/Layerscape-Knowledge-Base/LS1043ARDB-Ethernet-and-FMC-port-mapping/ta-p/1115640

– FM1 mEMAC1: 1AE_0000h

– FM1 mEMAC2: 1AE_2000h

– FM1 mEMAC3: 1AE_4000h

– FM1 mEMAC4: 1AE_6000h

– FM1 mEMAC5: 1AE_8000h

– FM1 mEMAC6: 1AE_A000h

– FM1 mEMAC9: 1AF_0000h

– FM1 mEMAC10: 1AF_2000h

For example: MAC5 is mapped with fm0-port-rx4. This device string is used for port configuration when fmc tool is executed. (fmc is a tool that permits to apply advanced offload functionallity on FMAN ports. The functionallity is encoded into a model called PCD - parse classify distribute - embedded in an xml) In your debugging you will not used this -fmc I mean. I wanted to explain what fm0-port-rx means in the explanations that will follow.

(Hint: MAC - port mapping can be found in the decompiled device tree)

The life of a packet for a LS1046 FMAN port is:

RX(at each level there are counters):

Rx MAC -> Port BMI -> FPM -> BMI prepare to enqueue->QMI -> BMI release internal resources -> CPU

TX(at each level there are counters):

BMI - > QMI (dequeue request from FMAN) - > BMI (DMA transaction from DDR to Internal memory) -> TX MAC -> QMI -> BMI (release resources = EBD)

An example of quick dumping of FMAN counters from sysfs(these represent FMBM_RFRC):

find /sys/devices/ -name 'port_frame' -exec cat {} \;

fm0-port-rx5 counter: 0

fm0-port-tx4 counter: 0

FM Port not configured...

fm0-port-tx0 counter: 1365

FM Port not configured...

fm0-port-tx7 counter: 0

fm0-port-rx4 counter: 47279872

FM Port not configured...

fm0-port-rx0 counter: 6800

fm0-port-tx5 counter: 43045909

fm0-port-oh1 counter: 0

FM Port not configured...

fm0-port-rx7 counter: 0

FM Port not configured...

Counters (detailed):

On Rx

0x204 FMBM_RFRC—Rx Frame Counter R/W All zeros 5.5.4.5.26/5-133

0x208 FMBM_RBFC—Rx Bad Frames Counter R/W All zeros 5.5.4.5.27/5-133 0x20C FMBM_RLFC—Rx Large Frames Counter R/W All zeros 5.5.4.5.28/5-134

0x210 FMBM_RFFC—Rx Filter Frames Counter R/W All zeros 5.5.4.5.29/5-135

0x214 FMBM_RFDC—Rx Frame Discard Counter R/W All zeros 5.5.4.5.30/5-135

0x218 FMBM_RFLDEC—Rx Frames List DMA Error Counter R/W All zeros 5.5.4.5.31/5-136 0x21C FMBM_RODC—Rx Out of Buffers Discard Counter R/W All zeros 5.5.4.5.32/5-137

0x220 FMBM_RBDC—Rx Buffers Deallocate Counter R/W All zeros 5.5.4.5.33/5-138

0x224 FMBM_RPEC—RX Prepare to enqueue Counter. R/W All zeros 5.5.4.5.34/5-138

cat /sys/bus/platform/devices/soc/1a00000.fman/1a8a000.port/fm_port_bmi_regs | grep -i rfr #FMBM_RBFC is not exported. We use a tool such as iomem or devmem ./iomem r32be:2 0x1a8a208 cat /sys/bus/platform/devices/soc/1a00000.fman/1a8a000.port/fm_port_bmi_regs | grep -i rlf cat /sys/bus/platform/devices/soc/1a00000.fman/1a8a000.port/fm_port_bmi_regs | grep -i rff ./iomem r32be:2 0x1a8a214 cat /sys/bus/platform/devices/soc/1a00000.fman/1a8a000.port/fm_port_bmi_regs | grep -i dec cat /sys/bus/platform/devices/soc/1a00000.fman/1a8a000.port/fm_port_bmi_regs | grep -i rodc ./iomem r32be:2 0x1a8a220 ./iomem r32be:2 0x1a8a224

MAC counters:

They can be found here:

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@2/net/eth2/mac_tx_stats

cat /sys/devices/platform/soc/soc:fsl,dpaa/soc:fsl,dpaa:ethernet@2/net/eth2/mac_rx_stats

Note in case the sysfs entries differ from above, identify the counters by:

find / -name "mac_rx_stats"

You can then consult the manual for offsets that you are interested in.

If the frames have reached on RX and you see counters incremented on BMI / QMI that means that everything has passed succesfully through MAC RX.

On the TX, if there are no error counters on MAC incremented that means that everything has passed succesfully through QMI /BMI"

LayerscapeのトピックRe: LS 1046 : DPAA ( how to Debug Throughput throttle issue )

LS 1046 : DPAA ( how to Debug Throughput throttle issue )

Re: LS 1046 : DPAA ( how to Debug Throughput throttle issue )