LayerscapeのトピックRe: LS1046A QDMA Operation Issue

LS1046A QDMA Operation Issue

JSKIM — Mon, 23 Nov 2020 13:38:22 GMT

I use lsdk2004, and PCIe device which physical address is 0x4040000000 and size is 0x800000.

when i test QDMA Transfer fromt host mem to host mem used 32bit address, there is no problem,

but in case of host to pcie device, qdma/aer reports error message which is "aer_uncorrectable_error_string --> CmpltTO".

May i have the reason?

Re: LS1046A QDMA Operation Issue

yipingwang — Tue, 24 Nov 2020 10:08:46 GMT

Probably there is problem with the addresses, please print out source and destination addresses to check.

Please refer to drivers/misc/pci_endpoint_test.c in Linux Kernel.

Re: LS1046A QDMA Operation Issue

JSKIM — Wed, 25 Nov 2020 05:48:24 GMT

I attach lspci result.

root@TinyLinux:~# lspci -v
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c1 (rev 10) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 69
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: 40000000-40bfffff [size=12M]
Prefetchable memory behind bridge: None
Expansion ROM at 4040c00000 [disabled] [size=2K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express <?>
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Memory controller: Xilinx Corporation Device 8024
Subsystem: Xilinx Corporation Device 0007
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at 4040800000 (32-bit, non-prefetchable) [size=2K]
Memory at 4040000000 (32-bit, non-prefetchable) [size=8M]
Memory at 4040800800 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pci-endpoint-test

root@TinyLinux:~#

root@TinyLinux:~# lspci -x
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c1 (rev 10)
00: 57 19 c1 81 07 01 10 00 10 00 04 06 08 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 ff 00 f1 01 00 00
20: 00 40 b0 40 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 40 45 01 02 00

01:00.0 Memory controller: Xilinx Corporation Device 8024
00: ee 10 24 80 06 04 18 00 00 00 80 05 00 00 00 00
10: 00 00 80 40 00 00 00 40 00 08 80 40 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 80 00 00 00 00 00 00 00 45 01 00 00

root@TinyLinux:~#

when i test, src is 0x4040000000 and dst is 0x00DA100000 in dma kernel memory.

dma test program use fsl_qdma_comp_fill_memcpy(struct fsl_qdma_comp *fsl_comp, dma_addr_t dst, dma_addr_t src, u32 len)

( ex: device_prep_dma_memcpy(dma_chan, 0x00DA100000, 0x4040000000, 0x800000, flags); )

Re: LS1046A QDMA Operation Issue

yipingwang — Wed, 25 Nov 2020 08:06:19 GMT

Please refer to PCIe DMA performance test sample code provided in https://community.nxp.com/t5/Layerscape/LS1046-data-transfer-via-PCIe/td-p/1185922.

Re: LS1046A QDMA Operation Issue

JSKIM — Wed, 25 Nov 2020 11:05:41 GMT

thanks for your response.

i had tested with PCIe DMA performance test sample code that you provided.

The test of memory to memory has no problem.

but the test of memory to pcie has problem.

i report the test step and error message of test.

01:00.0 Memory controller: Xilinx Corporation Device 8024
Subsystem: Xilinx Corporation Device 0007
Flags: bus master, fast devsel, latency 0, IRQ 69
Memory at 4040800000 (32-bit, non-prefetchable) [size=2K]
Memory at 4040000000 (32-bit, non-prefetchable) [size=8M]
Memory at 4040800800 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: PCIE_DRV

root@TinyLinux:~#

root@TinyLinux:~# echo 2 > /sys/module/dmatest/parameters/dmatest_option

root@TinyLinux:~# echo 0x4040000000 > /sys/module/dmatest/parameters/pci_phys_addr
root@TinyLinux:~# echo 1 > /sys/module/dmatest/parameters/produ_task_number
root@TinyLinux:~# echo 5000 > /sys/module/dmatest/parameters/iterations
root@TinyLinux:~# echo 64 > /sys/module/dmatest/parameters/test_buf_size
root@TinyLinux:~# echo 1 > /sys/module/dmatest/parameters/run
root@TinyLinux:~[ 65.331445] dmatest: current_mask 1
# [ 65.345799] dmatest: len 5001 save_mask f tmp_mask 2
[ 65.351021] dmatest: pcie_phy_base : 4040000000 pcie_virt_base : 00000000b8287473
[ 66.410460] fsl-qdma 8380000.dma-controller: DMA transaction error! intr 20000000
[ 66.417944] (NULL device *): reg FSL_QDMA_DECFDW0R 7000000
[ 66.423426] (NULL device *): reg FSL_QDMA_DECFDW1R d430a010
[ 66.428994] (NULL device *): reg FSL_QDMA_DECFDW2R 20000000
[ 66.434562] (NULL device *): reg FSL_QDMA_DECFDW3R 40000000
[ 66.440130] (NULL device *): reg FSL_QDMA_DECFQIDR 0
[ 66.445089] (NULL device *): reg FSL_QDMA_DECBR 0
[ 66.449804] pcieport 0000:00:00.0: AER: Uncorrected (Non-Fatal) error received: 0000:00:00.0
[ 66.458252] fsl-qdma 8380000.dma-controller: DMA transaction error! intr 20000000
[ 66.458253] (NULL device *): reg FSL_QDMA_DECFDW0R 7000000
[ 66.458255] (NULL device *): reg FSL_QDMA_DECFDW1R d430a050
[ 66.458256] (NULL device *): reg FSL_QDMA_DECFDW2R 20000000
[ 66.458258] (NULL device *): reg FSL_QDMA_DECFDW3R 40000000
[ 66.458259] (NULL device *): reg FSL_QDMA_DECFQIDR 0
[ 66.458261] (NULL device *): reg FSL_QDMA_DECBR 0
[ 66.497599] fsl-qdma 8380000.dma-controller: DMA transaction error! intr 20000000
[ 66.497600] (NULL device *): reg FSL_QDMA_DECFDW0R 7000000
[ 66.497602] (NULL device *): reg FSL_QDMA_DECFDW1R d430a090
[ 66.497603] (NULL device *): reg FSL_QDMA_DECFDW2R 20000000
[ 66.497604] (NULL device *): reg FSL_QDMA_DECFDW3R 40000000
[ 66.497606] (NULL device *): reg FSL_QDMA_DECFQIDR 0
[ 66.497607] (NULL device *): reg FSL_QDMA_DECBR 0

Re: LS1046A QDMA Operation Issue

yipingwang — Mon, 07 Dec 2020 05:39:31 GMT

Apologies for the delay.

I have looked into the log. I can see the outbound window#1 (0x4040000000) in lspci output has been created for Xilinx . However, it seems like PCIe window has not been translated properly.
1) Recommend to try with "devmem" commands in order to verify window mapping.
2) Please provide the PCIe window information. I want to see the translation address set for outbound window.
3) Also provide the PCIe register dump in order to see other possibilities of errors if any.

Re: LS1046A QDMA Operation Issue

JSKIM — Mon, 07 Dec 2020 13:24:07 GMT

thank for your response.

when i check phy address area with devmem, i realized that writing operation is no problem. but in case of reading the area. there happen kernel panic.

so dose the problem cause the aer message?

Re: LS1046A QDMA Operation Issue

JSKIM — Tue, 08 Dec 2020 02:48:05 GMT

I found document file about pcie.

I think PCIe Extend Config field is enabled, so Root Complex process TLP Timeout Message.

I trid this filed changed with setpci. but after dma process, this field value is reset.

How do i change this filed?

* Filename WBNR_FTF12_NET_F0116.pdf

Completion Timeout Error is logged in the following registers at
Requester side:
− In Freescale proprietary memory-mapped register area:
 Freescale PCIe controller MM offset 0xE00, PEX Error Detect Register [PCT]
− In the optional Advanced Error Reporting (PCIe AER) area:
 Freescale PCIe Config. Offset 0x104, Uncorrectable Error Status Register [CTO]

root@TinyLinux:~# lspci -s 00:00.0 -xxxx
00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c1 (rev 10)
00: 57 19 c1 81 07 01 10 00 10 00 04 06 08 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 ff 00 f1 01 00 00
20: 00 40 b0 40 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 40 46 01 02 00
40: 01 50 23 7e 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 88 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 42 00 01 80 00 00 3f 28 02 00 43 f4 73 00
80: 08 00 42 f0 00 00 00 00 c0 03 40 00 08 00 00 00
90: 00 00 00 00 3f 04 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 01 00 82 14 00 40 00 00 00 00 40 00 30 20 46 00
110: 00 00 00 00 00 60 00 00 ae 00 00 00 00 00 00 00

Re: LS1046A QDMA Operation Issue

JSKIM — Fri, 11 Dec 2020 02:05:03 GMT

I think this issue is about pcie request, non-postred request.

I found out pcie issue related about [2/2] pci/layerscape: change the default error response behavior.

In our case, we just use posted request.

when used posted request, may i have what the value of PCIE_ABSERR_SETTING ?

/* Kernel Code */

#define PCIE_ABSERR 0x8d0 /* Bridge Slave Error Response Register */
#define PCIE_ABSERR_SETTING 0x9401 /* Forward error of non-posted request */

Re: LS1046A QDMA Operation Issue

yipingwang — Fri, 11 Dec 2020 08:05:52 GMT

1. How do you confirm that your writing operation is working fine?
2. Before doing the above experiment, it is recommended to take the trace from PCIe Analyzer in order to see the status of outgoing and incoming packets.
3. Please share the full register dump of PCIe.

Re: LS1046A QDMA Operation Issue

JSKIM — Fri, 11 Dec 2020 09:12:27 GMT

1. How do you confirm that your writing operation is working fine?

-> I use xillinx debug tool and check the data from rc to ep.

2. Before doing the above experiment, it is recommended to take the trace from PCIe Analyzer in order to see the status of outgoing and incoming packets.

-> I just use pcie posted transactions.

Requests that are considered posted transactions are:

Memory Writes

3. Please share the full register dump of PCIe.

-> i attached dump files.

Re: LS1046A QDMA Operation Issue

yipingwang — Tue, 15 Dec 2020 06:07:48 GMT

Thanks for your answers.

I have looked at the PCIe register dump. It looks good.

"Before doing the above experiment, it is recommended to take the trace from PCIe Analyzer in order to see the status of outgoing and incoming packets."

[NXP]: I mean was to take the PCIe Analyzer dump when doing the read transactions from RC. It will allow seeing the outgoing packets (reading request) from RC to EP and incoming packets (completion data) from EP to RC. It will be helpful the customer can share the PCIe Analyzer dump during reading transactions from RC.

Re: LS1046A QDMA Operation Issue

JSKIM — Wed, 16 Dec 2020 00:43:16 GMT

Thanks for your answers.

I will check your advice.

Re: LS1046A QDMA Operation Issue

TrinathK — Sat, 09 Dec 2023 09:29:08 GMT

Hi,

I am also facing the same issue. Non-posted request (memory read request) is going at the end of each memory write transaction. Let me know how to disable sending memory read request from kernel.

Thanks in advance.