i.MX6 PCIE to XCHI USB3.0 Performance Issues

[ottawa-imx6]

Hello everyone,
I have an issue that I am trying to resolve which involves PCIE, XHCI, and USB 3.0 on the i.MX6Q systems.
The issue I am having is that my data transfer rates are too slow and as such this leads me to believe that PCIE2.0 and USB3.0 XHCI might not be configured optimally using the freescale kernel.

Attempted hardware:
SABRELITE
SABRESDB
Renesas USB3.0 Controller upd720201
TI TUSB7320

Kernels
up to and including 3.0.35-4.0.0 with PCIE, XHCI_HCD, and MSI enabled.

Examples

/dev/sda = external sata drive connected through USB3.0 and the PCIE controller.
/dev/sdb = sata drive connected directly to the i.MX6
/dev/sdc = second external sata drive

Through seperate tests on intel machines, I have verified that:
All drives are capable of (and proven on the same renesas and ti controllers) of well over 150MB/s write speed.
Both the TI and Rensesas controllers both are also capable of the bandwidth required.

My results on the i.MX6

CASE 1 : dd if=/dev/zero of=/dev/sda bs=1M write-speed ~60-70MB/s

CASE 2 : dd if=dev/sda of=/dev/sdb bs=1M write-speed ~35-40MB/s

CASE 3 : dd if=/dev/sda of=/dev/sdc bs=1M write-speed ~30-35MB/s

As I can tell there is a significant drop in the already limited maximum bandwidth (70MB/s while writing zeros from ram) down to 35MB/s when I try to transfer from one disk to another through the PCIE and usb3.0 interface.
Granted that I am using the interface for two drives simultaneously, it makes sense that it would be slower. I would imagine that if CASE 1 was performing at a higher rate then CASE 3 would be less relatively so.

The main issue seems to be with the baseline slow speed in CASE 1. 70MB/s is nowhere near the functional rates for PCIE2.0 + USB3.0 (I understand that there is overhead, but 70MB/s is still multiple times slower than it should be).

My questions to anyone who has experience with this is:
1. Has anyone seen any better performance than this?
2. Can anyone at freescale verify that this is the maximum performance possible through these interfaces? Are there any plans to update the kernel to improve this?

Thanks!
Brent

[ottawa-imx6]

Wow... Thanks for the detailed response Matt. I am going to read that a few times over, draw some diagrams and hopefully sort out some actionable tests based on that. Much appreciated. Stay tuned!

Brent

[ottawa-imx6]

The 150MB/s was fast enough at the time, and I didn't bother adjusting any block size or write parameters since the test was simply to verify that the Renesas USB3.0 controller was capable of greater than how it was performing  on the mx6 (80MB/s).

This test was simply to rule out the possibility that the controller was at fault.

Either the mx6 hardware does not perform to spec, or the Linux drivers are due for some improvement. (o:

I suspect that as more users look to rely on pcie, xhci and usb3.0 in the future, the drivers will improve.

[TomE]

Are the CPUs maxed out? Does it make any difference how many cores you have running?

I've found the easiest way to find out how busy the CPU really is is to find out how much low-priority spare time it has.

By first doing "nice -15 bash" to get a low-execution priority shell.

Then when the system is idle, type 'time for (( i = 0 ; i < 100000 ; i++ )) do true; done;" and record the time.

Then when it is busy, do it again. If it takes twice as long the CPU is 50% busy. If it takes 10 times as long, 90% busy and so on. You may need to run one of these for every core you have if the main task that is causing the problem can't be split between cores.

Tom