topic Re: LPC55S69 USB device receives corrupt data in LPC Microcontrollers

LPC55S69 USB device receives corrupt data

mditto1 — Fri, 05 Feb 2021 18:41:11 GMT

Hello, we need some help solving a USB data corruption problem on the
LPC55S69 (or perhaps the problem is on the i.MX8M Quad).

We are building a system that includes an i.MX8M acting as a USB host
connecting to an LPC55S69 acting as a high speed USB device. We see USB
messages sent from the host and received by the LPC55S69 that are
corrupted by the time they are stored in RAM. I have used a Beagle USB
480 analyzer to capture the bus traffic and it sees intact data,
suggesting that the corruption is happening in the LPC55S69. However,
this corruption never happens when an x86 laptop is used as the host,
and happens very frequently when the i.MX8M is the host. Using the x86
PC as a host I can send (and receive the replies to) hundreds of
thousands of consecutive messages without error, but using the i.MX8M as
a host I get at most one or two correct transfers before a failure. I
have not been able to identify a specific difference between the
messages sent by the PC and the ones sent by the i.MX8M.

We originally encountered the problem with data packets of the DFU
protocol, but have reduced our code in the MCU to a simple
vendor-defined "echo" protocol to allow the problem to be easily
reproduced. It simply receives a message into a buffer and then allows
the host to request the same buffer to be sent back. The CPU does not
read or write the buffer. The host side software for this test is a
small Python 3 program.

The particular form of data corruption is usually limited to the first
16 bytes of the received message, and usually affects entire 32-bit
words, but not necessarily all of the 32-bit words in the first 16
bytes. For example, we might see that only the second and fourth word
of the buffer are corrupt. Each word that is corrupt contains a copy of
some other 32-bit word later in the message.

Messages of 64 bytes or shorter do not experience this corruption. It
is easy to reproduce with 80-byte transfers and happens at least up to
512 bytes as well. The problem does not happen when the LPC55S69's
full speed interface is used - only on the high speed interface.

The problem can be reproduced using the MCIMX8M-EVKB as the host,
running the NXP reference Linux (4.9.88-imx) image with the addition of
the Python 3 "usb" module, and the LPCXpresso55S69 development board as
the device.

A typical run of the test looks like this:

root@imx8mqevk:~# lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 1fc9:0094 NXP Semiconductors Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub root@imx8mqevk:~# date | ./usbecho Found 001:002 1FC9:0094 MCU VIRTUAL COM DEMO None Data payload is 29 bytes. Good match. Success: received data matched sent data. root@imx8mqevk:~# dd if=/dev/urandom bs=80 count=1 of=/tmp/shortfile 1+0 records in 1+0 records out 80 bytes copied, 0.000155521 s, 514 kB/s root@imx8mqevk:~# ./usbecho < /tmp/shortfile Found 001:002 1FC9:0094 MCU VIRTUAL COM DEMO None Data payload is 80 bytes. Receiving... Failure: received data does not match sent data. < d9eeb869024c4f7fac0b9b79331f7d36e7a04b7c159513d0d2277541c523a006698188b05c3c8db46bb6ace9e5c49341c4b603177eff7dee3bac0bca1b690940ed4f333ddc64ccb8b404a109126ca82a > ed4f333ddc64ccb8b404a109126ca82ae7a04b7c159513d0d2277541c523a006698188b05c3c8db46bb6ace9e5c49341c4b603177eff7dee3bac0bca1b690940ed4f333ddc64ccb8b404a109126ca82a

In this case you can see that the first 16 bytes of the data have not been transferred correctly - specifically, they have been replaced with a copy of the last 16 bytes of the message.

Please find attached:

a patch to the NXP example program lpcxpresso55s69_dev_cdc_vcom_freertos that adds the "echo" functionality
The patched program, built with MCUXpresso, runs on the LPCXpresso55S69 board.
a Python 3 program to send a packet to the device, then read it back and check it for correctness
This program runs on x86 desktop Linux or on i.MX8M Linux.
a Beagle USB data capture file showing a packet on its way to the device and a packet reading back a corrupted copy of the data.
a screenshot showing the same
two Beagle USB data capture files showing "good" transfers (no corruption), one from the same i.MX8M board and one from the x86 laptop.

Thanks,

Mike

Re: LPC55S69 USB device receives corrupt data

mditto1 — Thu, 28 Jan 2021 20:22:39 GMT

I wrote:

The particular form of data corruption is usually limited to the first
16 bytes of the received message,

I should clarify that this is when using an 80-byte message. Presumably this has something to do with the 64-byte packet size on high speed control transfers, resulting in the last 16 bytes being sent in a second data packet.

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Mon, 01 Feb 2021 10:13:36 GMT

From your description. this problem only happens when message is longer than 64bytes, because you said "Messages of 64 bytes or shorter do not experience this corruption."

If you don't use i.MX8M send data, message still can't longer than 64byes? As we need to identify the problem is on LPC or i.MX8M.

Have a nice day,

Jun Zhang

Re: LPC55S69 USB device receives corrupt data

mditto1 — Tue, 02 Feb 2021 02:27:08 GMT

Messages of any size from a PC are received correctly on the LPC.

Thanks,

Mike

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Tue, 02 Feb 2021 07:05:54 GMT

So the problem is only on data from iMX8M. Could you collect the same data from iMX8M and PC, check them with USB analyzer. What's the difference?

Re: LPC55S69 USB device receives corrupt data

mditto1 — Tue, 02 Feb 2021 14:40:29 GMT

I attached Beagle capture files to my initial report above. You can view them with the "data center" software from https://www.totalphase.com/products/data-center/.

I did not see any great difference between the successful and corrupted runs. In both cases, the 80-byte transfer is broken into a 64-byte packet and a 16-byte packet as expected (high speed control transfers have a maximum packet size of 64 bytes). The 64-byte packet is re-transmitted one time, and the 16-byte packet is transmitted only once. The data in the first packet is correct (in the initial transmission and in the re-transmission) when captured by the analyzer, but in the LPC memory it is corrupt only in the i.MX8M case. The only thing I noticed was minor timing differences. I'm attaching screenshots showing these two test runs here.

Mike

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Fri, 05 Feb 2021 11:02:55 GMT

The problem is mostly on iMX8M side. How do you think? Please let me know if you don't think.

Can you check why your iMX can't generate same USB wave as PC?

Re: LPC55S69 USB device receives corrupt data

mditto1 — Fri, 05 Feb 2021 18:47:16 GMT

It's very difficult to say. Clearly the problem only happens when the iMX8M is involved.

But consider this:

Look at evk-evk-bad.png. At index 43, a "good" transfer of 80 bytes goes from host to device. At index 74, the transfer in the opposite direction shows corrupt data. Since the host->device transfer is expanded in this display, we can look at the individual packets. Index 48 shows the first attempt to send bytes 0-63, and this attempt failed (NAK at index 51). Index 52 shows the second attempt to send those same bytes and the OUT packet at index 58 shows the bytes are not yet corrupted and the NYET packet at index 59 shows the LPC received the packet. We assume that those bytes 0-63 were stored correctly in LPC memory at that point.

But the data transfer is not done yet - it's an 80-byte transfer so there are still 16 more bytes to send. At index 60 there is another OUT transaction with those 16 bytes. At index 66 we see the exact values of those 16 bytes, and the acknowledgement at index 67 (and ACK at index 72 for the entire control transfer). We assume that the final 16 bytes were stored correctly in LPC memory at that point.

But how, exactly did those 16 bytes ALSO get stored at the beginning of the RAM buffer? Those bytes appeared on the wire exactly once (index 66). But they appear in RAM in two places (offset 0-15 and offset 64-79). Why did the LPC's USB hardware and DMA engine store those bytes in RAM twice?

This mystery suggests a flaw on the LPC side. But it does not explain why the problem happens only when the iMX8M is the host.

To reiterate, here are the bytes the iMX8M sent on the wire:

D9EEB869 024C4F7F AC0B9B79 331F7D36
E7A04B7C 159513D0 D2277541 C523A006
698188B0 5C3C8DB4 6BB6ACE9 E5C49341
C4B60317 7EFF7DEE 3BAC0BCA 1B690940
ED4F333D DC64CCB8 B404A109 126CA82A

And here are the bytes that appeared in RAM on the LPC:

ED4F333D DC64CCB8 B404A109 126CA82A <-- why duplicated?
E7A04B7C 159513D0 D2277541 C523A006
698188B0 5C3C8DB4 6BB6ACE9 E5C49341
C4B60317 7EFF7DEE 3BAC0BCA 1B690940
ED4F333D DC64CCB8 B404A109 126CA82A <-- why duplicated?

I think this should be investigated on the LPC side by someone who understands the USB1 high speed receive and DMA hardware.

Thanks,
Mike

Re: LPC55S69 USB device receives corrupt data

mditto1 — Fri, 05 Feb 2021 18:45:33 GMT

By the way, based on a suggestion via our distributor, I noticed that I had not allocated the data buffer in dedicated USB RAM. I made that change (adding USB_GLOBAL to the variable declaration) and it did not make any difference. I updated the usbecho.patch attachment above with this change anyway.

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Sun, 07 Feb 2021 08:25:38 GMT

Hi,

We need to check it internally. Will keep you informed.

Thanks,

Jun Zhang

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Wed, 10 Feb 2021 14:53:48 GMT

Could you please provide more information about the setup?

- SDK version (have you tried using SDK2.9.0?).

- USB SDK demo on LPC55 and application description (if not using demo from SDK: what classes are used, what's the application flow?)

- Clk frequencies

- Does it also happen with FS USB?

- Have you tried other PCs?

Thanks.

Jun Zhang

Re: LPC55S69 USB device receives corrupt data

mditto1 — Wed, 10 Feb 2021 16:10:43 GMT

The problem was first seen using L5.4.3_1.0.0. I reproduced it using the SD card image in the Linux download page. I've now downloaded L5.4.70_2.3.0_MX8MND3L and used the SD card image there. The problem is the same in all cases.

The application I used to repro on the LPC55 is the lpcxpresso55s69_dev_cdc_vcom_freertos demo with the patch I attached to the initial message in this thread. All it does is receive a control transfer into a buffer or generate a control transfer back to the host from that buffer.

- Clk frequencies

Whatever the USB demo uses, I didn't change them.

- Does it also happen with FS USB?

No, only with HS.

- Have you tried other PCs?

No.

Re: LPC55S69 USB device receives corrupt data

mditto1 — Fri, 12 Feb 2021 00:14:39 GMT

With some simple changes to the LPC firmware (which really shouldn't change the behavior) the problem is now reproducible using a Linux x86 PC as the host. So this eliminates the i.MX8 host as the cause and someone at NXP has reproduced the problem internally and is driving an internal investigation into the LPC55S69 behavior.

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Fri, 12 Feb 2021 04:35:41 GMT

Thanks for your update. Good to know.

As you said this issue was reproduced by nxp internally, I suggest go on tracking it. If any help from community support, please contact us again. On my side, once I get feedback for LPC USB HS known issue. I will let you know as well.

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Sat, 13 Feb 2021 09:43:33 GMT

I got internal feedback, we were able to reproduce the issue and need to check further why the buffer is being corrupted when the buffer is aligned and placed in USB_RAM section.

Meanwhile, I could find 2 workarounds:

- Option1: Define the echoBuffer array as not aligned in USB_RAM

USB_GLOBAL static uint8_t echoBuffer[256];

- Option 2: Define the echoBuffer array in SRAM (not USB_RAM), aligned to 64 bytes.

USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[256];

This is, instead of the current implementation:

USB_GLOBAL USB_RAM_ADDRESS_ALIGNMENT(USB_DATA_ALIGN_SIZE) static uint8_t echoBuffer[256]; //@TEST

See //@TEST in attached examples.

Please test how it works.

Best Regards

Jun Zhang

Re: LPC55S69 USB device receives corrupt data

mditto1 — Sat, 13 Feb 2021 21:09:50 GMT

Jun, Thanks for the workaround suggestions. Did you test them using the -l option to usbecho.py?

I changed the buffer to be intentionally unaligned and tried USB SRAM and regular SRAM. The rate of failures dropped dramatically, but not to zero. Here is a typical run:

root@imx8mqevk:~# ./usbecho.py -l </tmp/shortfile
Found 001:065 1FC9:0094 MCU VIRTUAL COM DEMO None
Data payload is 80 bytes.
35 successful. Receiving...
Failure: received data does not match sent data.
< c9e461f46959e82c37b3ffebb7655b3151aa06fd487ce46010b7d47df1144205dfc5f121034a0a67c1bcdaeb30104697b4bce9c892377a8c6753bc01bea12747962661584c98df8b5c0f5b53a0398b1f
> 962661584c98df8b5c0f5b53a0398b1f51aa06fd487ce46010b7d47df1144205dfc5f121034a0a67c1bcdaeb30104697b4bce9c892377a8c6753bc01bea12747962661584c98df8b5c0f5b53a0398b1f
Failed after 35 successful matches
root@imx8mqevk:~#

Thanks,
Mike

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Sun, 14 Feb 2021 14:24:54 GMT

Mike,

What's the difference of using and not-using the -l option to usbecho.py on your side?

Could you please specify?

Thanks,

Jun Zhang

Re: LPC55S69 USB device receives corrupt data

mditto1 — Mon, 15 Feb 2021 07:33:58 GMT

-l runs the test repeatedly until it fails and reports how many successes there were before the failure.

Re: LPC55S69 USB device receives corrupt data

ZhangJennie — Mon, 15 Feb 2021 09:47:08 GMT

Both option 1 and option2 has the same issue?

Re: LPC55S69 USB device receives corrupt data

mditto1 — Tue, 16 Feb 2021 10:40:34 GMT

I tested the code you sent, which I believe uses option 2 as sent, and I tested my code with the simple modification to guarantee misalignment:

USB_GLOBAL USB_DMA_NONINIT_DATA_ALIGN(USB_DATA_ALIGN_SIZE)
static uint8_t echoBuffer[257];
...
controlRequest->buffer = echoBuffer + 1;
controlRequest->length = min(controlRequest->length,
sizeof echoBuffer - 1);
...
controlRequest->buffer = echoBuffer + 1;

Both options have the same or very similar behavior - about 1 in 10 failure rate. This is different from the original aligned behavior which fails at about 9 in 10 rate (rough figures).

If you test with the -l option, if the workaround is truly effective it will run endlessly (hundreds of thousands of successful transfers). But using the workarounds, I haven't seen more than 44 successful transfers before a failure.