i.MX6Q PCIe EP/RC Validation and Throughput
Hardware setup
* Two i.MX6Q SD boards, one is used as PCIe RC; the other one is used as PCIe EP. Connected by 2*mini_PCIe to standard_PCIe adaptors, 2*PEX cable adaptors, and one PCIe cable.
Software configurations
* When building RC image, make sure that
CONFIG_IMX_PCIE=y
# CONFIG_IMX_PCIE_EP_MODE_IN_EP_RC_SYS is not set
CONFIG_IMX_PCIE_RC_MODE_IN_EP_RC_SYS=y
* When build EP image, make sure that
CONFIG_IMX_PCIE=y
CONFIG_IMX_PCIE_EP_MODE_IN_EP_RC_SYS=y
# CONFIG_IMX_PCIE_RC_MODE_IN_EP_RC_SYS is not set
Features
* Set-up link between RC and EP by their stand-alone 125MHz running internally.
* In EP's system, EP can access the reserved ddr memory (default address:0x40000000) of PCIe RC's system, by the interconnection between PCIe EP and PCIe RC.
NOTE: The layout of the 1G DDR memory on SD boards is 0x1000_0000 ~ 0x4FFF_FFFF)
Use mem=768M in the kernel command line to reserve the 0x4000_0000 ~ 0x4FFF_FFFF DDR memory space used to do the EP access tests.
(The example of the RC’s cmd-line: Kernel command line: noinitrd console=ttymxc0,115200 mem=768M root=/dev/nfs nfsroot=10.192.225.216:/home/r65037/nfs/rootfs_mx5x_10.11,v3,tcp ip=dhcp rw)
Throughput results
| ARM core used as the bus master, and cache is disabled | ARM core used as the bus master, and cache is enabled | IPU used as the bus master(DMA) | |
| Data size in one write tlp | 8 bytes | 32 bytes | 64 bytes |
| Write speed | ~109MB/s | ~298MB/s | ~344MB/s |
| Data size in one read tlp | 32 bytes | 64 bytes | 64 bytes |
| Read speed | ~29MB/s | ~100MB/s | ~211MB/s |
IPU used as the bus master(DMA)
Here is the summary of the PCIe throughput results tested by IPU.
Write speed is about 344 MB/s.
Read speed is about 211MB/s
ARM core used as the bus master (define EP_SELF_IO_TEST in pcie.c driver)
write speed ~300MB/s.
read speed ~100MB/s.
Cache is enabled.
PCIe EP: Starting data transfer...
PCIe EP: Data transfer is successful, tv_count1 54840us, tv_count2 162814us.
PCIe EP: Data write speed is 298MB/s.
PCIe EP: Data read speed is 100MB/s.
Regarding to the log, the data size of each TLP when cache is enabled, is about 4 times of the data size in write, and 2 times of the data size in read, when the cache is not enabled.
| Cache is disabled | Cache is enabled | |
| Data size in one write tlp | 8 bytes | 32 bytes |
| Write speed | ~109MB/s | ~298MB/s |
| Data size in one read tlp | 32 bytes | 64 bytes |
| Read speed | ~29MB/s | ~100MB/s |
Cache is not enabled
PCIe EP: Starting data transfer...
PCIe EP: Data transfer is successful, tv_count1 149616us, tv_count2 552099us.
PCIe EP: Data write speed is 109MB/s.
PCIe EP: Data read speed is 29MB/s.
One simple method used to connect the imx6 pcie ep and rc
View of the whole solution:
HW materials:
2* iMX6Q SD boards, 2* Mini PCIe to STD PCIe adaptors, one SATA2 data cable.
the mini-pcie to standard pcie exchange adaptor.
Here is the URL:
http://www.bplus.com.tw/Adapter/PM2C.html
How to make it.
signals connections
Two adaptors, one is named as A, the other one is named as B.
A B
TXM <----> RXM
TXN <----> RXN
RXM <----> TXM
RXN <----> TXN
A1 connected to B3
A2 connected to B4
A3 connected to B1
A4 connected to B2
Connect the cable to the adaptor.
Connect the SATA2 data cable to Mini PCIe to STD PCIe adaptor (A)
Connect the SATA2 data cable to Mini PCIe to STD PCIe adaptor (B)
NOTE:
* Please keep length of Cable as short as possible. Our cable is about 12cm.
* Please connect shield wire in SATA2 Cable to GND at both board.
* Please boot up PCIe EP system before booting PCIe RC system.
Base one imx_3.0.35 mainline, the patch, and the IPU test tools had been attached.
NOTE:
* IPU tests usage howto.
Unzip the xxx.zip, and run xxx_r.sh to do read tests, run xxx_w.sh to do the write tests.
Tests log:
EP:
root@freescale ~/pcie_ep_io_test$ ./pcie-r.sh
pass cmdline 14, ./pcie_ipudev_test.out
new option : c
frame count set 1
new option : l
loop count set 1
new option : i
input w=1024,h=1024,fucc=RGB4,cpx=0,cpy=0,cpw=0,cph=0,de=0,dm=0
new option : O
640,480,RGB4,0,0,0,0,0
new option : s
show to fb 0
new option : f
output file name ipu1-1st-ovfb
new option : ÿ
show_to_buf:0, input_paddr:0x1000000, output.paddr0x18800000
====== ipu task ======
input:
foramt: 0x34424752
width: 1024
height: 1024
crop.w = 1024
crop.h = 1024
crop.pos.x = 0
crop.pos.y = 0
output:
foramt: 0x34424752
width: 640
height: 480
roate: 0
crop.w = 640
crop.h = 480
crop.pos.x = 0
crop.pos.y = 0
total frame count 1 avg frame time 19019 us, fps 52.579000
root@freescale ~/pcie_ep_io_test$ ./pcie-w.sh
pass cmdline 14, ./pcie_ipudev_test.out
new option : c
frame count set 1
new option : l
loop count set 1
new option : i
input w=640,h=480,fucc=RGB4,cpx=0,cpy=0,cpw=0,cph=0,de=0,dm=0
new option : O
1024,1024,RGB4,0,0,0,0,0
new option : s
show to fb 1
new option : f
output file name ipu1-1st-ovfb
new option : ÿ
show_to_buf:1, input_paddr:0x18a00000, output.paddr0x1000000
====== ipu task ======
input:
foramt: 0x34424752
width: 640
height: 480
crop.w = 640
crop.h = 480
crop.pos.x = 0
crop.pos.y = 0
output:
foramt: 0x34424752
width: 1024
height: 1024
roate: 0
crop.w = 1024
crop.h = 1024
crop.pos.x = 0
crop.pos.y = 0
total frame count 1 avg frame time 11751 us, fps 85.099140
root@freescale ~$ ./memtool -32 01000000=deadbeaf
Writing 32-bit value 0xDEADBEAF to address 0x01000000
RC:
Before run "./memtool -32 01000000=deadbeaf" at EP.
root@freescale ~$ ./memtool -32 40000000 10
Reading 0x10 count starting at address 0x40000000
0x40000000: 00000000 00000000 00000000 00000000
0x40000010: 00000000 00000000 00000000 00000000
0x40000020: 00000000 00000000 00000000 00000000
0x40000030: 00000000 00000000 00000000 00000000
After run "./memtool -32 01000000=deadbeaf" at EP.
root@freescale ~$ ./memtool -32 40000000 10
Reading 0x10 count starting at address 0x40000000
0x40000000: DEADBEAF 00000000 00000000 00000000
0x40000010: 00000000 00000000 00000000 00000000
0x40000020: 00000000 00000000 00000000 00000000
0x40000030: 00000000 00000000 00000000 00000000
Dear Richard,
I'm currently trying to transmit data from a iMX6 board to a Xilinx FPGA board through a pcie link x1, using the IPU DMA to have reasonable performance, given the PCIe RC in iMX6 doesn't have DMA.
To ahieve this I adapted your demo to my specific scenario. I didn't have any major problems in the adaptation process. The writing operations from the iMX6 to the FPGA seem to work fine but, in the same way as for Yann (although our scenarios are different), instead of reaching the 344 MB/s stated in the link of the demo, with TLP data size 64 bytes, I only get about 54MB/s due to the fact that the data size of the TLPs I receive in the FPGA side is only 16 bytes. The connector I'm using between the two boards is a mini pcie to pcie
Which configuration parameters should i touch in the iMX6 side in order to increase the size of the TLPs transmitted?
Thanks very much in advance for your answer.
Best Regards,
Eduardo.
Hi,
I'm having trouble getting the PCIE up on the i.MAX6 Quad Sabre Lite Development board (https://boundarydevices.com/product/sabre-lite-imx6-sbc/ ) and interfacing with TW6865 Intersil video chip and I couldn't find the sufficient details on how to connect REFCLK- and REFCLK+ of TW6865 with PCIE port of the i.MAX6Q Sabre Lite as it just have following pin connections.
Can you guide me to interface TW6865 with PCIE of Sabre Lite ? (Presently I'm working 3.14.52 kernel based system)
Thank You,
Peter.
i.max#i.mx6q pcie
Hi,
I am not able to access the mentioned files.
Can you please provide access for the same?
