MIMXRT1170-EVK Gig-Ethernet receiving bad content

[mjbcswitzerland]

Hi All

I am wondering whether someone has experienced the same or can propose an explanation for what is being seen?

I am using the 1 Gig-Ethernet interface on the MIMXRT1170-EVK with a (non-SDK) driver used on its 10/100M interface that has been extended to support both interfaces at the same time.

I can use one or both of the Ethernet interfaces at 100Mb/s with no (known) issues.

The problem starts when using the Gig-Ethernet interface at 1Gb/s (not on a 100Mb/s network, where it is OK). If I run a ping test there are occasional failures due to the ping response returned by the board having a mis-matching ping payload:

For example

Reply from 192.168.1.3: bytes=32 time<1ms TTL=128
Reply from 192.168.1.3: bytes=32 time<1ms TTL=128
Reply from 192.168.1.3: bytes=32 time=1ms TTL=128
Reply from 192.168.1.3: bytes=32 time<1ms TTL=128
Reply from 192.168.1.3: bytes=32 - MISCOMPARE at offset 22 - time<1ms TTL=128

There are no IP errors as the IP and ICMP checksums are OK, meaning that the ping content was correctly echoed back. To verify this I also check the ping content received and finally verify the data that the Ethernet controller wrote to memory (by analysing the buffer descriptor and the memory that it points to).

Since every ping has the same payload this is easy to do and the result shows that whenever it fails the Ethernet controller has saved two bytes different to the content seen on the network with Wireshark.

Here I have two memory dumps from the RX buffer content pointed to by a buffer descriptor.
First when the ping was good and second when there was this ping error:

md 20006780 b 150
Memory Display
0x20006780 00 00 00 00 00 03 00 e0 4c 68 03 d2 08 00 45 00 ........Lh....E.
0x20006790 00 3c 19 df 00 00 80 01 9d 8d c0 a8 01 01 c0 a8 .<..............
0x200067a0 01 03 00 00 00 00 00 01 05 83 61 62 63 64 65 66 ..........abcdef
0x200067b0 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv
0x200067c0 77 61 62 63 64 65 66 67 68 69 62 63 64 65 66 67 wabcdefghibcdefg
0x200067d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x200067e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x200067f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x20006800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x20006810 00 00 00 00 00 00 ......

md 20006780 b 150
Memory Display
0x20006780 00 00 00 00 00 03 00 e0 4c 68 03 d2 08 00 45 00 ........Lh....E.
0x20006790 00 3c 19 df 00 00 80 01 9d 8d c0 a8 01 01 c0 a8 .<..............
0x200067a0 01 03 00 00 00 00 00 01 05 83 61 62 63 64 65 66 ..........abcdef
0x200067b0 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv
0x200067c0 68 69 62 63 64 65 66 67 68 69 62 63 64 65 66 67 hibcdefghibcdefg
0x200067d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x200067e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x200067f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x20006800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x20006810 00 00 00 00 00 00 ......

There are 6 buffer descriptors allocated to the receiver and the one that makes the error is also random (above I have two cases from the same buffer descriptor's rx buffer).

The difference are ALWAYS two bytes - "wa" changes to 'hi' at the address 0x200067c0.
Curious is that there are these two bytes at 0x200067b1 (15 bytes previously).


As noted, this is a random error (1 in 10 or 1 in 20 pings, for example), can take place in any buffer descriptor, always has exactly the same content difference and only occurs when it auto-negotiates to 1Gb/s and never occurs at 100Mb/s.
The MIB counters show no error.
There are no checksum issues on the IP / ICMP reception code as it is relying on the check-sum offloading in the Ethernet module to flag errors, which must have received the correct data to allow it to pass.
The logical conclusion is that the Ethernet controller successfully checks the ping request received but saves the content incorrectly to memory afterwards.

The question remains as to why and how to solve it?

Regards

Mark

[mjbcswitzerland]

Hi MayLiu

If I use the evkmimxrt1170_lwip_ping_bm_cm7 reference I don't have the same behavior - in 1Gb/s mode the ping responses are OK. (In comparison my setup has about 5% ping responses with the two byte error when operating at 1Gb/s but none at 100Mb/s).

However it is not that easy to do an exact comparison.
The example (as it is delivered) uses no advanced modes - it uses the buffer descriptors in basic mode without their advanced feature. It doesn't have IEEE1588 time sampling enabled. It doesn't use IP checksum off-loading, etc.

Therefore I will need to configure the various projects in a number of modes to see whether there is one that triggers the behavior (at 1Gb/s).

In the meantime it would be useful if you could ask the specialists whether they can think of an explanation for the 1G ENET controller to receive packets correctly (passing IP and ICMP checksum verification) but storing them with occasional two byte error identically in either SDRAM, OCR1, OCR2 or ITC memory?

Today I tested my project with the CM7 core running at different frequencies from 100MHz to 1GHz just to be sure that the processor's own operating speed didn't have any effect. The behavior was always identical, which is however essentially as expected based on the fact that the ENET's clocking and the CM7's clocking are independent from another (but I remember some devices where there was a lower limit to the CPU's frequency for correct Ethernet operation, so I wanted to be sure).

Don't forget that when the ENET auto-negotiates to 100Mb/s there are never any such errors, meaning that the basic operation (buffer and memory alignment and such) doesn't seem to be the underlying factor.

Thanks, regards

Mark

[mayliu1]

Hi @mjbcswitzerland ,

Thanks for your updated information.

Based on your feedback information , this issue is very unlikely to be related to the physical link.

1:   Please enforce Non‑Cacheable + Properly Aligned ENET Buffers
To avoid Cortex‑M7 cache coherency issues, place the RX and TX descriptors and data buffers in a Non‑Cacheable memory region, and make sure they are properly aligned.
You may configure a region in OCRAM or SDRAM as Non‑Cacheable by using the MPU.
You can refer to the NXP SDK demo and the application note below:
https://www.nxp.com/docs/en/application-note/AN12042.pdf

2:  Please verify the ETH_PAD_SIZE configuration
Make sure the padding setting matches your driver usage.

Best Regards

MayLiu

[mayliu1]

Hi @mjbcswitzerland ,

Thank you for your interest in NXP Semiconductor products and for the opportunity to serve you.

The address 0x20006780 is inside the CM7 DTCM.
According to Erratum ERR050396, ENET may have data problems when it writes to CM7 TCM.

https://www.nxp.com.cn/docs/en/errata/ES_IMXRT1170BCE.pdf

To avoid this problem,  Please clear the bit IOMUXC_GPR_GPR28[CACHE_ENET] if ENET writes to TCM.

If this bit is set, then ENET buffers must be placed in OCRAM or external RAM, not in DTCM.

Wish it helps you

Best Reagrds

MayLiu

[mjbcswitzerland]

Hi @mayliu1 

Thank you for the tip - I hadn't checked potential errata.

Unfortunately it doesn't seem to be that errata though - 

1. IOMUXC_GPR_GPR28 is 0x00000000 and so no write caching on ENETs

2. I tried locating the receive buffer in OCR1, OCR2 and SDRAM but in each case the behavior is exactly identical - the exact same two bytes sometimes wrong and at the same frequency.

Here are some memory dumps in the three new RAM areas showing the location of the data (again with the differences being seen)


SDRAM (0x80000000)

0x8000003a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi
0x8000062a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x8000063a     71 72 73 74 75 76 68 69 62 63 64 65 66 67 68 69  qrstuvhibcdefghi
0x80000c2a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x80000c3a     71 72 73 74 75 76 68 69 62 63 64 65 66 67 68 69  qrstuvhibcdefghi
0x8000122a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x8000123a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi

Error seen at 0x80000c40/41


OCR2

0x202c122a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x202c123a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi
0x202c182a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x202c183a     71 72 73 74 75 76 68 69 62 63 64 65 66 67 68 69  qrstuvhibcdefghi
0x202c1e2a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x202c1e3a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi

Error seen at 0x202c1840/41


OCR1

0x20241e3a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi
0x2024002a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x2024003a     71 72 73 74 75 76 68 69 62 63 64 65 66 67 68 69  qrstuvhibcdefghi
0x2024062a     61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70  abcdefghijklmnop
0x2024063a     71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69  qrstuvwabcdefghi

Error seen at 0x2040040/41


3. The errata seems to suggest that by having MUXC_GPR_GPR28 set to 0 should be adequate to avoid the errata in TCM and these results look to confirm that it is not based on that since the same behavior is seen in the memory areas that should also not be affected by it.

Are there any further suggestions?

Regards

Mark

[mayliu1]

Hi @mjbcswitzerland ,

Thanks for your updated information.

Since you are not using the SDK, I suggest trying the NXP MCUXpresso SDK Ethernet example.
Please keep the cache handling and alignment settings as shown in the demo.

If the  two‑byte error still happens when using the SDK demo, then we can continue with deeper checks as the next step.

Best Regards

MayLiu