[LS1028A]: SoC to SoC communication between two boards via PCIe

[Amit_Keshri]

Dear NXP support Team,

We have two LS1028A processor based custom board,one board we are configuring as root complex(RC) (PEX1 =0 ,PEX2=0) and other board as End point EP(PEX1=1,PEX2=0) through RCW file.Both card are connected over backplane and both card ported with Flexbuild v2012 U-Boot and Linux.

How to prove communication between two processor over PCIe interface?

 I am following Below procedure according to BSP document (LSDKUG_Rev21.08.pdf) section 7.2.8.4,but when i tried to create symbolic link using below command in EP board ,it is showing below error:

root@TinyLinux:~#
root@TinyLinux:~# cd /sys/kernel/config/pci_ep/
root@TinyLinux:/sys/kernel/config/pci_ep# mkdir functions/pci_epf_test/func1
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 0x1957 > functions/pci_epf_test/func1/vendorid
sudo echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 0x80c0 > functions/pci_epf_test/func1/deviceid
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 8 > functions/pci_epf_test/func1/msix_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo ln -s functions/pci_epf_test/func1 controllers/3400000.pcie_ep
ln: failed to create symbolic link 'controllers/3400000.pcie_ep': Operation not permitted

When i tried to create symbolic link in other directory ,it's got created as shown below:

root@TinyLinux:~# ln -s /sys/kernel/config/pci_ep /home/sym_link
root@TinyLinux:~# ls /home/syym__link
controllers functions
root@TinyLinux:~# cd /home/syym__link
root@TinyLinux:/home/syym__link# ls
controllers functions

Why i could not able to create any symbolic link in /sys/kernel/config/pci_ep directory ?.

Please help to solve this issue .

Regards,

Amit Keshri

[yipingwang]

In RCW, please configure the following.

HOST_AGT_PEX1=1

Please configure the following Linux Kernel configurations.

CONFIG_PCI=y

CONFIG_PCI_ENDPOINT=y

CONFIG_PCI_ENDPOINT_CONFIGFS=y

CONFIG_PCI_EPF_TEST=y

CONFIG_PCI_LAYERSCAPE_EP=y

CONFIG_PCI_ENDPOINT_TEST=y

[Amit_Keshri]

@yipingwang ,

Thank you for your reply.

We are doing all above mention things in RCW and Linux Kernel configurations for End point board but still Communication is not happening.

Regards,

Amit Keshri

[Amit_Keshri]

@yipingwang 

Adding some point regarding above information.

We are able to see enumeration of EP board from RC board using lspci command as shown below:

root@TinyLinux:~# lspci
0000:00:00.0 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.1 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.2 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.3 Class 0880: Device 1957:ee01 (rev 01)
0000:00:00.4 Class 0880: Device 1957:ee02 (rev 01)
0000:00:00.5 Class 0208: Device 1957:eef0 (rev 01)
0000:00:00.6 Class 0200: Device 1957:e100 (rev 01)
0000:00:1f.0 Class 0807: Device 1957:e001 (rev 01)
0001:00:00.0 Class 0604: Device 1957:82c0 (rev 10)
0001:01:00.0 Class 0604: Device 10b5:8604 (rev ba)
0001:02:01.0 Class 0604: Device 10b5:8604 (rev ba)
0001:02:04.0 Class 0604: Device 10b5:8604 (rev ba)
0001:02:05.0 Class 0604: Device 10b5:8604 (rev ba)
0001:03:00.0 Class 0604: Device 10b5:8604 (rev ba)
0001:04:00.0 Class 0604: Device 10b5:8604 (rev ba)
0001:04:01.0 Class 0604: Device 10b5:8604 (rev ba)
0001:04:04.0 Class 0604: Device 10b5:8604 (rev ba)
0001:05:00.0 Class 0b20: Device 1957:82c0 (rev 10)
0002:00:00.0 Class 0604: Device 1957:82c0 (rev 10)
0002:01:00.0 Class 0000: Device 11aa:1556

Bleow is the enumeration log from EP board:

root@TinyLinux:~# lspci
0000:00:00.0 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.1 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.2 Class 0200: Device 1957:e100 (rev 01)
0000:00:00.3 Class 0880: Device 1957:ee01 (rev 01)
0000:00:00.4 Class 0880: Device 1957:ee02 (rev 01)
0000:00:00.5 Class 0208: Device 1957:eef0 (rev 01)
0000:00:00.6 Class 0200: Device 1957:e100 (rev 01)
0000:00:1f.0 Class 0807: Device 1957:e001 (rev 01)
0001:00:00.0 Class 0604: Device 1957:82c0 (rev 10)
0001:01:00.0 Class 0000: Device 11aa:1556

Note:

Where, 1957:82c0 is vendor id & device id of LS1028A process of EP board

10b5:8604 is vendor id & device id of PCIe Switch of EP board

11aa:1556 is vendor id & device id of FPGA of EP board

Please find attached diagram for processor.

Regards,

Amit Keshri

[yipingwang]

I checked LSDK 2012 Linux Kernel source code, ls1028a is not in the PCIe EP supported device list.

Please refer to these two patches http://patchwork.ozlabs.org/project/linux-pci/list/?series=267849

to add ls1028a PCIe endpoint support.

[Amit_Keshri]

@yipingwang ,

Thank you for quick reply.

I will update above 2 patches and update you the results

Regards,

Amit Keshri

[Amit_Keshri]

@yipingwang ,

I applied the above patches in my flexbuild_lsdk2012 SDK as shown below and created new itb(kernel image) and loaded in EP board:

In packages/linux/linux/arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi +898 file added below things:

pcie_ep1: pcie-ep@3400000 {
compatible = "fsl,ls1028a-pcie-ep","fsl,ls-pcie-ep";
reg = <0x00 0x03400000 0x0 0x00100000>,
<0x40 0x00000000 0x8 0x00000000>;
reg-names = "regs", "addr_space";
num-ib-windows = <2>;
num-ob-windows = <2>;
status = "disabled";
};

In packages/linux/linux/arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts +650 file added below things:

&pcie_ep1 {
status = "okay";
};

In packages/linux/linux/drivers/misc/pci_endpoint_test.c file added as shown below:
line number :69

#define PCI_DEVICE_ID_LS1028A 0x82c0

static const struct pci_endpoint_test_data default_data = {
.test_reg_bar = BAR_2,
.alignment = SZ_64K,
.irq_type = IRQ_TYPE_MSI,
};

line number +809
{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, PCI_DEVICE_ID_LS1028A),
.driver_data = (kernel_ulong_t)&default_data,},

When i run below command in RC board i got operation not permitted error while creating soft link as shown below in log :

root@TinyLinux:~# cd /sys/kernel/config/pci_ep/
root@TinyLinux:/sys/kernel/config/pci_ep# mkdir functions/pci_epf_test/func1
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 0x1957 > functions/pci_epf_test/func1/vendorid
sudo echo 2 > functions/pci_epf_test/func1/msi_interrupts
sudo echo 8 > functions/pci_epf_test/func1/msix_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 0x82c0 > functions/pci_epf_test/func1/deviceid
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo echo 8 > functions/pci_epf_test/func1/msix_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# sudo ln -s functions/pci_epf_test/func1 controllers/3400000.pcie_ep
ln: failed to create symbolic link 'controllers/3400000.pcie_ep': Operation not permitted

Regards,

Amit Keshri

[yipingwang]

Please refer to the following dts definition.

pcie_ep1: pcie_ep@3400000 {
compatible = "fsl,ls1028a-pcie-ep","fsl,ls-pcie-ep";
reg = <0x00 0x03400000 0x0 0x00100000
0x80 0x00000000 0x8 0x00000000>;
reg-names = "regs", "addr_space";
interrupts = <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>; /* PME interrupt */
interrupt-names = "pme";
num-ib-windows = <6>;
num-ob-windows = <8>;
status = "disabled";
};

[yipingwang]

We had managed to fix the link issue, please modify uboot and rcw code.
nxa13285@lsv07330:/opt/samba/nxa13285/flexbuild/flexbuild/components/firmware/rcw$ git diff
diff --git a/ls1028ardb/R_SQPP_0x85bb/rcw_1500_gpu600.rcw b/ls1028ardb/R_SQPP_0x85bb/rcw_1500_gpu600.rcw
index 057b0f1..2e4fa7d 100644
--- a/ls1028ardb/R_SQPP_0x85bb/rcw_1500_gpu600.rcw
+++ b/ls1028ardb/R_SQPP_0x85bb/rcw_1500_gpu600.rcw
@@ -38,6 +38,7 @@ SRDS_PRTCL_S1_L0=8
SRDS_PRTCL_S1_L1=5
SRDS_PRTCL_S1_L2=11
SRDS_PRTCL_S1_L3=11
+HOST_AGT_PEX2=1

/* Errata for PCIe controller */
#include <../ls1028asi/a008851.rcw>
nxa13285@lsv07330:/opt/samba/nxa13285/flexbuild/flexbuild/components/firmware/rcw$

nxa13285@lsv07330:/opt/samba/nxa13285/flexbuild/flexbuild/components/firmware/uboot$ git diff
diff --git a/arch/arm/dts/fsl-ls1028a.dtsi b/arch/arm/dts/fsl-ls1028a.dtsi
index 7234faea0e..bda9035ea0 100644
--- a/arch/arm/dts/fsl-ls1028a.dtsi
+++ b/arch/arm/dts/fsl-ls1028a.dtsi
@@ -123,6 +123,17 @@
0x82000000 0x0 0x40000000 0x88 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
};

+
+ pcie_ep2: pcie_ep@3500000 {
+ compatible = "fsl,ls-pcie-ep";
+ reg = <0x00 0x03500000 0x0 0x80000
+ 0x00 0x035c0000 0x0 0x40000
+ 0x88 0x00000000 0x8 0x00000000>;
+ reg-names = "regs", "ctrl", "addr_space";
+ num-ib-windows = <6>;
+ num-ob-windows = <8>;
+ };
+
crypto: crypto@8000000 {
compatible = "fsl,sec-v5.0", "fsl,sec-v4.0";
fsl,sec-era = <10>;
diff --git a/configs/ls1028ardb_tfa_defconfig b/configs/ls1028ardb_tfa_defconfig
index 87357d30c0..0afbc815c1 100644
--- a/configs/ls1028ardb_tfa_defconfig
+++ b/configs/ls1028ardb_tfa_defconfig
@@ -94,3 +94,4 @@ CONFIG_OF_LIBFDT_OVERLAY=y
CONFIG_EFI_LOADER_BOUNCE_BUFFER=y
CONFIG_VIDEO=y
CONFIG_VIDEO_LS_HDP_LOAD=y
+CONFIG_PCIE_LAYERSCAPE_EP=y

please check detailed log.

[yipingwang]

I am investigating this issue, will provide more update later.

[Amit_Keshri]

@yipingwang ,

Thank you for your reply ,i wiil add above dts changes and update you the result.

Regards,

Amit Keshri

[Amit_Keshri]

@yipingwang ,

As per suggestion i have added above EP DTS      in flexbuild_lsdk2012/packages/linux/linux/arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi file and created new itb file and loaded in EP board.

After booting linux i entered below command but got same Operation not permitted error after entering below command in RC board.

root@TinyLinux:~# cd /sys/kernel/config/pci_ep/
root@TinyLinux:/sys/kernel/config/pci_ep# mkdir functions/pci_epf_test/func1
root@TinyLinux:/sys/kernel/config/pci_ep# echo 0x1957 > functions/pci_epf_test/func1/vendorid
root@TinyLinux:/sys/kernel/config/pci_ep# echo 0x82c0 > functions/pci_epf_test/func1/deviceid
root@TinyLinux:/sys/kernel/config/pci_ep# echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# echo 8 > functions/pci_epf_test/func1/msix_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# ln -s functions/pci_epf_test/func1 controllers/3400000.pcie_ep
ln: failed to create symbolic link 'controllers/3400000.pcie_ep': Operation not permitted

Regards,

Amit Keshri

[yipingwang]

You should run these commands on the Endpoint board, not on RC board.

[Amit_Keshri]

@yipingwang ,

Yes i should run above command on EP board and later i tried the same and got the same result as shown in RC board.

regards,

Amit Keshri

[yipingwang]

I am working with the AE and development team for this issue.

Would you please capture the whole console log of your target board to me to do more investigation?

[Amit_Keshri]

@yipingwang ,

Please find the attached complete boot log for RC and EP board for your reference.

Regards,

Amit Keshri

[yipingwang]

In LSDK 21.08, LS1028A PCIe EP support is added by default.

I built itb image for LS1028A PCIe EP with LSDK 21.08, and uploaded lsdk2108_yocto_tiny_LS_arm64.itb here https://support.nxp.com/s/case/5002p00002s7J7xAAE/community-ls1028a-soc-to-soc-communication-between...

Please download it and run it on your EP device to try whether it works.

Please boot up EP device first, then boot RC device.

[Amit_Keshri]

@yipingwang ,

Thank you for your reply.

I will check and update you with SDK 21.08.

Regards,

Amit Keshri

[Amit_Keshri]

@yipingwang ,

As per your suggestion,I compiled SDK21.08 and loaded .itb file in EP and RC board and i ensured that EP board is booting first.

After boot linux ,i run below command in EP board but got same error:

root@TinyLinux:~# cd /sys/kernel/config/pci_ep/
root@TinyLinux:/sys/kernel/config/pci_ep# mkdir functions/pci_epf_test/func1
root@TinyLinux:/sys/kernel/config/pci_ep# echo 0x1957 > functions/pci_epf_test/func1/vendorid
root@TinyLinux:/sys/kernel/config/pci_ep# echo 0x82c0 > functions/pci_epf_test/func1/deviceid
root@TinyLinux:/sys/kernel/config/pci_ep# echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# echo 8 > functions/pci_epf_test/func1/msix_interrupts
root@TinyLinux:/sys/kernel/config/pci_ep# ln -s functions/pci_epf_test/func1 controllers/3400000.pcie_ep
ln: failed to create symbolic link 'controllers/3400000.pcie_ep': Operation not permitted

Please find the attached log for EP board.

Regards,

Amit Keshri