T1042D4RDB - End Point Configuration

[mustafapolat_se]

How can someone configure the related board as end point device so that one can access board's RAM over PCIe? I have checked the following docs; however, I could not find a solution. Can someone provide an example? or suggest any other doc? Thanks in advance

https://www.nxp.com/docs/en/supporting-information/QORIQ-SDK-2.0-IC-REV0.pdf  

https://www.nxp.com/docs/en/application-note/AN12246.pdf 

[yipingwang]

You could refer to section "7.11.6 PCIE EP" in QORIQ-SDK-2.0-IC-REV0.pdf.

Please refer to LS1046A RDB in PCIe-Endpoint Mode .

1. Modify the rcw for EP board:

set :

HOST_AGT_PEX=1

SRDS_DIV_PEX_S3=1

SRDS_DIV_PEX_S4=1

modify rcw for RC board as follow to set the linkup speed to 5.0G

SRDS_DIV_PEX_S3=1

SRDS_DIV_PEX_S4=1

Then rebuild the two rcw,

Program the rcw into EP and RC board

2. 

Build the image for EP board

$ bitbake -c menuconfig virtual/kernel

-> Device Drivers

    -><*>VFIO Non-Privileged userspace driver framework

        -><*> VFIO support for Freescale PCI Endpoint devices

    -><*> Userspace I/O drivers

        <*> Freescale DMA support

3.  Boot up the EP board with kerenl rootfs

    Deploy the EP board

    Run PCI DMA EP application to initialize EP device.

    root@p5020ds:~# ./pciep_dma 0 

    pcidma> add 0 0

4. Build the PCIe RC driver

-> Device Drivers

     -> DMA Engine support

         [*] Freescale Elo and Elo Plus DMA support (enable the option)

->Bus options 

        --> [*] PCI IOV support

5.

 On the RC board:

Test RC to EP DMA performance
root@t4240qds:/sys/class# cd pcidma/pcidma0/ root@t4240qds:/sys/class/pcidma/pcidma0# ls bars_info  config_infodevicesubsystem  test_dma_enable  test_ep2rctest_info  test_lens  test_loop  test_rc2ep  test_start  uevent
root@t4240qds:/sys/class/pcidma/pcidma0# echo 1 > test_start test starting root@t4240qds:/sys/class/pcidma/pcidma0#  test info: test0 packet length:64B loop:500times RC->EP throughput:49Mbps test1 packet length:256B loop:500times RC->EP throughput:164Mbps test2 packet length:1024B loop:500times RC->EP throughput:658Mbps test3 packet length:4096B loop:500times RC->EP throughput:1594Mbps test4 packet length:1048576B loop:500times RC->EP throughput:11216Mbps test5 packet length:2097152B loop:500times RC->EP throughput:11337Mbps

6.
Test EP to RC DMA performance
root@t4240qds:/sys/class/pcidma/pcidma0# echo 1 > test_ep2rc root@t4240qds:/sys/class/pcidma/pcidma0# echo 0 > test_rc2ep root@t4240qds:/sys/class/pcidma/pcidma0# echo 1 > test_start test starting root@t4240qds:/sys/class/pcidma/pcidma0#  test info: test0 packet length:64B loop:500times EP->RC throughput:353Mbps test1 packet length:256B loop:500times EP->RC throughput:1270Mbps test2 packet length:1024B loop:500times EP->RC throughput:3893Mbps test3 packet length:4096B loop:500times EP->RC throughput:7710Mbps test4 packet length:1048576B loop:500times EP->RC throughput:12148Mbps test5 packet length:2097152B loop:500times EP->RC throughput:12162Mbps

[mustafapolat_se]

Sorry for the late reply, and thank you for the quick one. We tried your suggestion, but after setting RCWs for EP and RC boards, we experienced some power issues. Can you help us about the following issues? Or can you guide us?

• Can we connect two t1042RDBs via PCIe with male-to-male cable directly? Or do we have to use PCIe switch between them?
• It is hard to find male-to-male PCIe cable. We tried to implement it by merging two male-to-female PCIe cables by crossing TXp to RXp and TXn to PXn and vice versa. We connecting other pins directly. If you ask, I can provide further information. 

[yipingwang]

I confirmed with LSDK development team, please refer to the following.

• Can we connect two t1042RDBs via PCIe with male-to-male cable directly? Or do we have to use PCIe switch between them?

[Yiping] Yes, we use male-to-male PCIe cable connect RC and EP boards, PCIe switch is not necessary. 

• It is hard to find male-to-male PCIe cable. We tried to implement it by merging two male-to-female PCIe cables by crossing TXp to RXp and TXn to PXn and vice versa. We connecting other pins directly.

[Yiping] We’d never try this way.

[mustafapolat_se]

Can your refer an male-to-male PCIe cable? I mean URL or specific name for the product? Thank you for your help

[yipingwang]

Please refer to OSS-KIT-EXP-3500 for PCIe Gen 2 x4 https://www.onestopsystems.com/product/pcie-x4-expansion-kit-3500

OSS-KIT-EXP-8000 for PCIe Gen 2 x8 https://www.onestopsystems.com/product/pcie-x8-gen-2-host-target-kit-8000