As part of the patches attached with this blog, we will relay the pcie write transaction from Endpoint-A to Endpoint-B connected to iMX95FRDM PRO.
Linux-imx used - lf-6.18.2-1.0.0
Attached are the following files:-
imx95-19x19-frdm-pro-pcie0-ep-dtbs - EP A shall use the dtb built with this dtbs
imx95-19x19-frdm-pro-pcie1-ep-dtbs - EP B shall use the dtb built with this dtbs
rc_pcie_dma_relay.c - driver used on RC to relay pcie write from EP-A to EP-B
conf_pcie0.sh - script to be executed on Endpoints A and B to configure the EPF driver
//To build the dtb and relay kernel driver
1. git clone git clone https://github.com/nxp-imx/linux-imx.git
git checkout origin/lf-6.18.y
2. Copy the dtbs to arch/arm64/boot/dts/freescale/
Copy rc_pcie_dma_relay.c to drivers/pci/
3. Make the following changes as per this diff
diff --git a/arch/arm64/boot/dts/freescale/Makefile b/arch/arm64/boot/dts/freescale/Makefile
index aa3cfdf1aafc..56e3db653208 100644
--- a/arch/arm64/boot/dts/freescale/Makefile
+++ b/arch/arm64/boot/dts/freescale/Makefile
@@@ -1205,6 +1205,16 @@ dtb-$(CONFIG_ARCH_MXC) += imx95-15x15-frdm-8mic-reve.dt
dtb-$(CONFIG_ARCH_MXC) += imx95-19x19-frdm-pro.dtb imx95-19x19-frdm-pro-aud-hat.dtb
+
+dtb-$(CONFIG_ARCH_MXC) += imx95-19x19-frdm-pro-pcie0-ep.dtb
+dtb-$(CONFIG_ARCH_MXC) += imx95-19x19-frdm-pro-pcie1-ep.dtb
+
+imx95-19x19-frdm-pro-pcie0-ep-dtbs := imx95-19x19-frdm-pro.dtb \
+ imx95-19x19-frdm-pro-pcie0-ep.dtbo
+
+imx95-19x19-frdm-pro-pcie1-ep-dtbs := imx95-19x19-frdm-pro.dtb \
+ imx95-19x19-frdm-pro-pcie1-ep.dtbo
+
imx95-19x19-frdm-pro-os08a20-isp-dtbs := imx95-19x19-frdm-pro.dtb \
imx95-19x19-frdm-pro-os08a20.dtbo
dtb-$(CONFIG_ARCH_MXC) += imx95-19x19-frdm-pro-os08a20-isp.dtb
4. Add the following to drivers/pci/Makefile
+obj-m += rc_pcie_dma_relay.o
5. Trigger the kernel build. You will obtain rc_pcie_dma_relay.ko, imx95-19x19-frdm-pro-pcie0-ep.dtb and imx95-19x19-frdm-pro-pcie1-ep.dtb.
6. We are only using pcie0 M.2 Key M slots of Endpoint A and Endpoint B so you only need to upload this dtb to both the endpoint boards - imx95-19x19-frdm-pro-pcie0-ep.dtb and boot linux with it after passing 'iommu.passthrough=1' at uboot mmcargs. This is to disable smmu for our tests.
RC will boot with the default dtb - imx95-19x19-frdm-pro.dtb
7. Connect the Endpoint-A to RC's K1 via M.2 Key M to Key M cable. Similarly connect the other Endpoint-B to other RC's K2 M.2 slot via Key M to Key M cable.
8. Execute this script on both the endpoints - ./conf_pcie0.sh
9. Then reboot the RC iMX95 FRDM Pro and ensure that you see both the endpoints:-
0000:01:00.0 and 0001:01:00.0 are the enumerated endpoints.
10. Upload rc_pcie_dma_relay.ko to the RC board and insert it like this:-
insmod rc_pcie_dma_relay.ko src_phys=0x910100000 dst_phys=0xa10100000 relay_len=0x100000 chunk_len=0x10000
you will observe similar logs on dmesg:-
[ 4949.082087] rc_pcie_dma_relay: init src=0x910100000 dst=0xa10100000 len=1048576 chunk=65536
[ 4949.082150] rc_pcie_dma_relay src_before: [0]=0xdeadbeef [1]=0xdeadbeef [2]=0xdeadbeef [3]=0xdeadbeef
[ 4949.082171] rc_pcie_dma_relay dst_before: [0]=0x00000000 [1]=0x00000000 [2]=0x00000000 [3]=0x00000000
[ 4949.125779] rc_pcie_dma_relay dst_zeroed: [0]=0x00000000 [1]=0x00000000 [2]=0x00000000 [3]=0x00000000
[ 4949.141380] rc_pcie_dma_relay src_after: [0]=0xdeadbeef [1]=0xdeadbeef [2]=0xdeadbeef [3]=0xdeadbeef
[ 4949.141427] rc_pcie_dma_relay dst_after: [0]=0xdeadbeef [1]=0xdeadbeef [2]=0xdeadbeef [3]=0xdeadbeef
[ 4954.272981] rc_pcie_dma_relay: verify OK for 1048576 bytes
[ 4954.273000] rc_pcie_dma_relay: DMA relay verify PASSED
11. Finally, via devmem5 on RC, you can verify the data of EP-A transferred to EP-B
./devmem5 r 0xa10100000 w
