LPC55S69 USB device receives corrupt data

[mditto1]

Hello, we need some help solving a USB data corruption problem on the
LPC55S69 (or perhaps the problem is on the i.MX8M Quad).

We are building a system that includes an i.MX8M acting as a USB host
connecting to an LPC55S69 acting as a high speed USB device. We see USB
messages sent from the host and received by the LPC55S69 that are
corrupted by the time they are stored in RAM. I have used a Beagle USB
480 analyzer to capture the bus traffic and it sees intact data,
suggesting that the corruption is happening in the LPC55S69. However,
this corruption never happens when an x86 laptop is used as the host,
and happens very frequently when the i.MX8M is the host. Using the x86
PC as a host I can send (and receive the replies to) hundreds of
thousands of consecutive messages without error, but using the i.MX8M as
a host I get at most one or two correct transfers before a failure. I
have not been able to identify a specific difference between the
messages sent by the PC and the ones sent by the i.MX8M.

We originally encountered the problem with data packets of the DFU
protocol, but have reduced our code in the MCU to a simple
vendor-defined "echo" protocol to allow the problem to be easily
reproduced. It simply receives a message into a buffer and then allows
the host to request the same buffer to be sent back. The CPU does not
read or write the buffer. The host side software for this test is a
small Python 3 program.

The particular form of data corruption is usually limited to the first
16 bytes of the received message, and usually affects entire 32-bit
words, but not necessarily all of the 32-bit words in the first 16
bytes. For example, we might see that only the second and fourth word
of the buffer are corrupt. Each word that is corrupt contains a copy of
some other 32-bit word later in the message.

Messages of 64 bytes or shorter do not experience this corruption. It
is easy to reproduce with 80-byte transfers and happens at least up to
512 bytes as well. The problem does not happen when the LPC55S69's
full speed interface is used - only on the high speed interface.

The problem can be reproduced using the MCIMX8M-EVKB as the host,
running the NXP reference Linux (4.9.88-imx) image with the addition of
the Python 3 "usb" module, and the LPCXpresso55S69 development board as
the device.

A typical run of the test looks like this:

root@imx8mqevk:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 1fc9:0094 NXP Semiconductors
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@imx8mqevk:~# date | ./usbecho
Found 001:002 1FC9:0094 MCU VIRTUAL COM DEMO None
Data payload is 29 bytes.
Good match.
Success: received data matched sent data.
root@imx8mqevk:~# dd if=/dev/urandom bs=80 count=1 of=/tmp/shortfile
1+0 records in
1+0 records out
80 bytes copied, 0.000155521 s, 514 kB/s
root@imx8mqevk:~# ./usbecho < /tmp/shortfile
Found 001:002 1FC9:0094 MCU VIRTUAL COM DEMO None
Data payload is 80 bytes.
Receiving...
Failure: received data does not match sent data.
< d9eeb869024c4f7fac0b9b79331f7d36e7a04b7c159513d0d2277541c523a006698188b05c3c8db46bb6ace9e5c49341c4b603177eff7dee3bac0bca1b690940ed4f333ddc64ccb8b404a109126ca82a
> ed4f333ddc64ccb8b404a109126ca82ae7a04b7c159513d0d2277541c523a006698188b05c3c8db46bb6ace9e5c49341c4b603177eff7dee3bac0bca1b690940ed4f333ddc64ccb8b404a109126ca82a

In this case you can see that the first 16 bytes of the data have not been transferred correctly - specifically, they have been replaced with a copy of the last 16 bytes of the message.

Please find attached:

• a patch to the NXP example program lpcxpresso55s69_dev_cdc_vcom_freertos that adds the "echo" functionality
  The patched program, built with MCUXpresso, runs on the LPCXpresso55S69 board.
• a Python 3 program to send a packet to the device, then read it back and check it for correctness
  This program runs on x86 desktop Linux or on i.MX8M Linux.
• a Beagle USB data capture file showing a packet on its way to the device and a packet reading back a corrupted copy of the data.
• a screenshot showing the same
• two Beagle USB data capture files showing "good" transfers (no corruption), one from the same i.MX8M board and one from the x86 laptop.

Thanks,

Mike

[rijn]

Hi Jennie , i am getting the same problem with LPC54S018 with the latest SDK

was this patched ?

[mditto1]

With some simple changes to the LPC firmware (which really shouldn't change the behavior) the problem is now reproducible using a Linux x86 PC as the host.  So this eliminates the i.MX8 host as the cause and someone at NXP has reproduced the problem internally and is driving an internal investigation into the LPC55S69 behavior.

[ZhangJennie]

Thanks for your update. Good to know.

As you said this issue was reproduced by nxp internally, I suggest go on tracking it. If any help from community support, please contact us again. On my side, once I get feedback for LPC USB HS known issue. I will let you know as well.

[ZhangJennie]

HI

I got internal feedback, we were able to reproduce the issue and  need to check further why the buffer is being corrupted when the buffer is aligned and placed in USB_RAM section. 

"

Meanwhile, I could find 2 workarounds:

- Option1: Define the echoBuffer array as not aligned in USB_RAM

USB_GLOBAL static uint8_t echoBuffer[256];

- Option 2: Define the echoBuffer array in SRAM (not USB_RAM), aligned to 64 bytes.

USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[256];

This is, instead of the current implementation:

USB_GLOBAL USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[256]; //@TEST

"

See //@TEST in attached examples.

Please test how it works.

Best Regards

Jun Zhang

[mditto1]

Jun, Thanks for the workaround suggestions.  Did you test them using the -l option to usbecho.py?

I changed the buffer to be intentionally unaligned and tried USB SRAM and regular SRAM.  The rate of failures dropped dramatically, but not to zero.  Here is a typical run:

root@imx8mqevk:~# ./usbecho.py -l </tmp/shortfile
Found 001:065 1FC9:0094 MCU VIRTUAL COM DEMO None
Data payload is 80 bytes.
35 successful. Receiving...
Failure: received data does not match sent data.
< c9e461f46959e82c37b3ffebb7655b3151aa06fd487ce46010b7d47df1144205dfc5f121034a0a67c1bcdaeb30104697b4bce9c892377a8c6753bc01bea12747962661584c98df8b5c0f5b53a0398b1f
> 962661584c98df8b5c0f5b53a0398b1f51aa06fd487ce46010b7d47df1144205dfc5f121034a0a67c1bcdaeb30104697b4bce9c892377a8c6753bc01bea12747962661584c98df8b5c0f5b53a0398b1f
Failed after 35 successful matches
root@imx8mqevk:~#

Thanks,
Mike

[ZhangJennie]

Mike,

What's the difference of using and not-using the -l option to usbecho.py on your side?

Could you please specify?

Thanks,

Jun Zhang

[mditto1]

-l runs the test repeatedly until it fails and reports how many successes there were before the failure.

[ZhangJennie]

Both option 1 and option2 has the same issue?

[mditto1]

I tested the code you sent, which I believe uses option 2 as sent, and I tested my code with the simple modification to guarantee misalignment:

USB_GLOBAL USB_DMA_NONINIT_DATA_ALIGN(USB_DATA_ALIGN_SIZE)
static uint8_t echoBuffer[257];

...

controlRequest->buffer = echoBuffer + 1;
controlRequest->length = min(controlRequest->length,
                             sizeof echoBuffer - 1);

...

controlRequest->buffer = echoBuffer + 1;

Both options have the same or very similar behavior - about 1 in 10 failure rate.  This is different from the original aligned behavior which fails at about 9 in 10 rate (rough figures).

If you test with the -l option, if the workaround is truly effective it will run endlessly (hundreds of thousands of successful transfers).  But using the workarounds, I haven't seen more than 44 successful transfers before a failure.

[ZhangJennie]

Our SE get these results with both approaches:

 are you using latest SDK 2.9.0?

[mditto1]

Yes, all of my tests with the LPCXpresso55S69 board have been with 2.9.0.  The modified files usb_device_cdc_acm.c and virtual_com.c that you attached to a message above do not exactly match the versions in 2.9.0 so I only copied and pasted the portions related to the ECHO protocol into the versions from the 2.9.0 example project.

With the same LPC board and firmware, I currently get no errors using the x86 laptop as host and about 1 in 10 errors using the i.MX8 host.

It seems that the ultimate test is to run usbecho.py -l from the i.MX8 host.

Mike

[ZhangJennie]

Mike,

As the workaround is to either place the buffer in USB_RAM without align macro or place the buffer in SRAM, aligned.

In reply I don't see neither of those implemented... if wants a buffer of 257 elements (unaligned to 64b), then please store the buffer in SRAM with alignment macro (shown below) and  make sure you are using the files I shared in previous reply:

USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[257];

Jun Zhang

[mditto1]

Jun,

Just to be sure, I created a fresh project from sdk example dev_cdc_vcom_freertos using SDK 2.9.0 and copied your two files exactly.  I also edited usb_device_config.h to use HS instead of FS.  This has the behavior I described (lower, but not zero, failure rate on i.MX8).

You suggested trying:

USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[257];

But this does not change the alignment of echoBuffer.  It is still at address 20000240.

Note that removing the alignment restriction:

static uint8_t echoBuffer[256];

Does not guarantee that echoBuffer will misaligned.  In practice I found that it gets placed at 2000087C (word aligned but not 64-byte aligned).  But the behavior does not change - there is still a low rate of failures.

I have also tested as I mentioned with a forced misaligned buffer (at an odd address, e.g. 0x20000241 and the behavior changed a little in the form of a somewhat higher failure rate).

I have created a new version of the test program that continues looping even after a failure and generates statistics about the failures.  This is useful to see how changes affect the failure rate and to see what kinds of failures are happening.  Instead of printing the actual buffer contents in hex, I record a pattern showing which words were corrupt.  For example, XXXX________________ means that the first four words were corrupt and the rest of the buffer was correct.  X means a word was replaced with a copy of a different word from some other part of the buffer.  ! means a word is corrupt in some other way.  Here is a run using the two source files that you sent:

root@imx8mqevk:~# ./usbechostats.py </tmp/shortfile
Data payload is 80 bytes.
51 791: ^C

XXXX________________ 731
_XXX________________ 55
_X_X________________ 1
___X________________ 4
25925 attempts in 21.2 seconds (1220.5/s), 791 failures (rate: 3.05%)

This shows that there are four different patterns of failure and the overall failure rate is about 3%.

Mike

[ZhangJennie]

Unfortunately, We don't have an i.MX8 to test exact same scenario but here are my results on both scripts with echoBuffer (256 or 257) aligned in RAM):

echoBuffer aligned in RAM, 256 size

echoBuffer aligned in RAM, 257 size

[ZhangJennie · Accepted solution]

Here is the workaround we received , please try it.

"

The previous workaround should fix the issue described: If echoBuffer is stored in SRAM or in USB_RAM (unaligned), the allocation and memcpy will be done for all packets(chunked as the maximum packet size), so the issue should not show up.

Anyway, please find attached the patch with the fix (patch for SDK 2.9.0).
Replace the file under <project folder>\usb\device\source\lpcip3511\usb_device_lpcip3511.c

The problem was that, if the buffer size is not the multiple of the maximum packet size, the lpcip3511 driver will allocate a USB dedicated RAM to receive data from host when the second-to-last packet transfer is done. As the previous code primes next buffer firstly(will allocate a USB dedicated RAM if needed) and then does the memcpy according to the variable epPacketCopyed, the variable epState->epBufferStatusUnion[odd].epBufferStatusField.epPacketCopyed will be set to 1 firstly and then do the memcpy. Since the control endpoint do not have the double buffer, the variable odd is always 0. Hence, the previous code will do the memcpy from a random USB dedicated RAM to echoBuffer, and thus the echoBuffer is corrputed. Now fix this by doing memcpy firstly and then prime the next buffer for the control endpoint.

This will be included in the upcoming SDK release. I confirmed patch is working, please test it as well.

With patch, user can store echoBuffer in USB_RAM (aligned).

"

Hope this helps,

Jun Zhang

[mditto1]

Thanks, Jun,

The change to usb_device_lpcip3511.c does fix the problem completely for us when using the i.MX8 host.

Thanks to you and everyone who helped investigate and fix this problem.

Mike

[mditto1]

By the way, based on a suggestion via our distributor, I noticed that I had not allocated the data buffer in dedicated USB RAM.  I made that change (adding USB_GLOBAL to the variable declaration) and it did not make any difference.  I updated the usbecho.patch attachment above with this change anyway.

[ZhangJennie]

Could you please provide more information about the setup?

- SDK version (have you tried using SDK2.9.0?).

- USB SDK demo on LPC55 and application description (if not using demo from SDK: what classes are used, what's the application flow?)

- Clk frequencies

- Does it also happen with FS USB?

- Have you tried other PCs?

Thanks.

Jun Zhang

[mditto1]

The problem was first seen using L5.4.3_1.0.0.  I reproduced it using the SD card image in the Linux download page.  I've now downloaded L5.4.70_2.3.0_MX8MND3L and used the SD card image there.  The problem is the same in all cases.

The application I used to repro on the LPC55 is the lpcxpresso55s69_dev_cdc_vcom_freertos demo with the patch I attached to the initial message in this thread.  All it does is receive a control transfer into a buffer or generate a control transfer back to the host from that buffer.

- Clk frequencies

Whatever the USB demo uses, I didn't change them.

- Does it also happen with FS USB?

No, only with HS.

- Have you tried other PCs?

No.

[ZhangJennie]

Hi,

We need to check it internally. Will keep you informed.

Thanks,

Jun Zhang