ls1021a eTSEC tx timeout

[renaud]

I have a LS1021A IOT and a prototype board based on this cpu. On both we use our boot loader.

It works perfectly in the IOT board but on my board I get TX timeout on the Ethernet port. I get a few pings going and then tx timeout and more pings.  This makes tftp rarely possible.

There are hardware difference of course. Our MAC is connected to a BCM54616S then to a LAN9514. The autonegotiation is limited to 100FD. TX timeout also happens in loopback mode.

In order to transmit data, I had to set bit SGMII_AN bit in the TBI PHY on my boards while I did not have to on the IOT (IOT can negotiate at 1000FD).

Not sure why the TX is so intermittent. Could it be related to this speed limitation?

[renaud]

Adding messages or 1ms delay in the send functions allows the correct transfer of data without loss.

I added a DMA flush to the TX descriptors in the send function and this helps. Though, this should not be necessary considering the descriptors are allocated in uncached memory. It could be a defect in the MMU library to which I added LPAE support.

Now I remember that a long time ago I removed the dma-coherent property in the enet device node and this was enough for the LS1021A-IOT to work. This is forcing a DMA flush while allocating uncached memory for the descriptors. But there was no cache flush in the send functions which updates the TX descriptors on each packet. Not sure why it does not work on my prototype (difference in frequency, DDR3L vs DDR4,...)

[Bio_TICFSL · Accepted solution]

Hi,

Thank you for the detailed findings — the combination of workarounds you identified (1 ms delay, manual DMA flush, and removing dma-coherent) is a textbook fingerprint of a cache alias problem. Here is the root cause explanation and the recommended fix path.

Root cause: cached virtual alias over the TX descriptor region

The LS1021A ENET DMA is a non-cache-coherent bus master — it reads DDR directly, with no visibility into the CPU cache. The correct operating model for gianfar on LS1021A is the non-coherent software-managed model: descriptors allocated in uncached memory, with explicit dma_sync_* calls whenever the CPU updates a descriptor field.

What your LPAE changes most likely introduced is a page table entry for the descriptor physical address range that carries the wrong cache attribute — Normal Writeback instead of Device or Normal Non-cacheable. This creates two virtual aliases to the same physical memory with different cacheability: the allocation path (via dma_alloc_noncoherent) maps it uncached, but the per-packet TX descriptor update path in the send function accesses it through a cached alias. The CPU writes the updated descriptor fields into the cache line, they never reach DDR, and the ENET DMA reads stale data and underruns.

Why your three workarounds all mask the same defect

• 1 ms delay / message insertion: adds enough latency for the CPU writeback buffer to drain naturally under low load — timing-dependent, will fail under traffic or frequency change.

• Manual dma_flush in the send function: forces a cache clean-to-PoC before the DMA reads the descriptor — correct behaviour, but should not be necessary if the region is genuinely uncached.

• Removing dma-coherent from the enet device node: causes the kernel to call dma_map_single() / dma_sync_single_for_device() on each TX descriptor before submission, which explicitly cleans the cache — also correct, and explains why the LS1021A-IOT worked reliably without a send-path flush.

The absence of the send-path flush is the missing piece. The LS1021A-IOT worked with dma-coherent removed because the kernel's DMA mapping layer inserted the sync automatically; your prototype does not have that path because the LPAE changes altered the descriptor region's cache attributes without re-introducing the sync.

DDR3L vs DDR4 / frequency difference

This is not the root cause. Different DRAM type and frequency change write latency and buffer drain timing, which is why the failure is more visible on your prototype — but the underlying defect is architectural and will be present on both boards under sufficient load.

Recommended fix

The correct and self-consistent fix is to combine both of the following:

1. Keep dma-coherent removed from the enet device node (non-coherent model). This causes the kernel DMA layer to issue dma_sync_single_for_device() automatically before DMA ownership is transferred for descriptors allocated via dma_alloc_noncoherent.

2. Add explicit dma_sync_single_for_device() calls in the TX send path at the point where TX descriptor fields (status, data_length, data_pointer) are written on each packet. This is the flush you added manually — but it should be formally placed as a dma_sync_single_for_device(dev, desc_dma_addr, sizeof(txbd), DMA_TO_DEVICE) before writing TDAR. This makes the model explicit and correct regardless of how the MMU attributes the descriptor region.

3. Separately, audit the LPAE MMU library change for the AttrIndx / TEX+C+B fields assigned to the physical address range used for ENET descriptors. The descriptor pool should be mapped as Device-nGnRnE (strongly ordered) or Normal Non-cacheable — not Normal Writeback. You can verify the actual attributes in use with the kernel's ptdump debug interface by checking the virtual address of the descriptor pool after allocation.

The DMA flush you added is not a workaround — it is the correct mechanism. The real defect is that it was absent from the send path in the first place, and the LPAE changes exposed this because they altered the effective cacheability of the region.

Regards

[Bio_TICFSL]

Hello,

For 100FD SGMII , verify these items in your boot loader on the prototype board:

1. Set LS1021A eTSEC for SGMII 100 Mbps

   • ECNTRL[TBIM] = 1
   • ECNTRL[SGMIIM] = 1
   • ECNTRL[R100M] = 1
   • MACCFG2[I/F Mode] = 01 for 10/100 mode
     The LS1021A RM specifically notes setting R100M = 1 for SGMII 100 Mbps.

2. Reset and program the internal TBI PHY The RM states that SGMII uses the TBI register set and that it is important to reset the TBI for all interface modes, including SGMII.

3. Keep SGMII_AN set The TBI SGMII_AN bit is documented as “must be set to 1.” So the fact that your board only transmits after setting this bit is not surprising; it suggests your boot-loader initialization was incomplete for this PHY/MAC mode.

4. Do not rely on 1G-style SGMII AN behavior at 100 Mbps There are known LS1021A reports where 100 Mbps SGMII operation shows “SGMII link is not ok” or intermittent no-packet behavior after link cycling, while 1G operation is clean. That does not prove your exact TX timeout root cause, but it makes 100FD SGMII configuration a strong suspect.

The loopback result matters. If your “loopback mode” is external PHY loopback or SGMII-side loopback, the 100FD SGMII/TBI setup can still be involved. If it is internal MAC/eTSEC loopback , then the external BCM54616S/LAN9514 path is mostly out of the picture, and I would focus on eTSEC initialization, descriptor-ring handling, cache coherency, and TX halt/error status.

For debugging, check the eTSEC TX halt status when the timeout occurs. The RM says the transmit halt bits are set when eTSEC is no longer processing transmit frames from a TxBD ring; repeatable causes include bus errors, invalid BD/data addresses, uncorrectable BD/data read errors, and TxBD programming errors such as Ready = 1 with length 0 . Also check whether you are seeing IEVENT_BSY ; NXP material describes BSY as dropped RX frames due to lack of buffers/software not servicing the BD ring fast enough, which is a software/BD-ring symptom rather than a pure SGMII electrical symptom.

Recommended isolation sequence:

1. Force external PHY to 100FD, no autoneg toward copper if needed.
2. Force LS1021A MAC/eTSEC to SGMII 100 Mbps:
   TBIM=1, SGMIIM=1, R100M=1, MACCFG2 I/F mode=10/100.
3. Reset/reinitialize TBI after setting SGMII mode.
4. Set TBI SGMII_AN=1.
5. Confirm TBI link/AN status, eTSEC ECNTRL/MACCFG2, and PHY SGMII-side status.
6. On timeout, dump IEVENT, TX halt registers, DMA status, and TXBD ring contents.

regards 

[renaud]

Forgot to say my port is set for SGMII mode.