MCF52259 FEC transmit data problem

[marcomaioli]

Hi,

I'm using our proprietary board (like demo board).

The FEC module always works well but only if the message size up to 800 byte.

When the size is >950 byte the FEC simply doesn't send out the packet.

I can check it via wireshark: when I try to send a packet of this size or larger, the transmission stops.

So TCP stack perform some retry, but nothing go out. No interrupt errors detected.

Internal bus clock: 80MHz

MAX_FL: 1518

MII_MODE: enabled

1664 byte are allocated for each DB buffer, aligned 16.

FEC registers are so set:

MCF_FEC_EIMR       0xfaf80000

MCF_FEC_RDAR      0x1000000

MCF_FEC_TDAR      0x0

MCF_FEC_ECR        0xf0000002

MCF_FEC_MMFR     0x0

MCF_FEC_MSCR     0x22

MCF_FEC_MIBC       0x40000000

MCF_FEC_RCR        0x6800004

MCF_FEC_TCR         0x4

MCF_FEC_OPD        0x16200

MCF_FEC_IAUR       0x0

MCF_FEC_IALR        0x0

MCF_FEC_GAUR      0xffffffff

MCF_FEC_GALR      0xffffffff

MCF_FEC_TFWR      0x0

I do not understand why the transmission stops, any ideas?

Thanks for any suggestions.

[TomE]

> the FEC simply doesn't send out the packet.

I don't believe that. If the FEC wasn't sending then it should be reporting some sort of error code. I'm pretty sure that the FEC sent the packet over the interface to the PHY chip.

If the FEC truly isn't finishing the transmit, the only reason I can think of would be buffer starvation. The memory system couldn't keep up with the data and the FIFOs underran at that point. Since the FIFOs are preloaded before the transmission starts, it is possible that an underrun could happen after 960 bytes. But you should be getting EIR[UN] (Transmit FIFO underrun) if this happens. Check for this.

That chip doesn't support external SDRAM like the ones I'm used to, so I assume you're transmitting from SRAM and not from some memory on the FlexBus.

Check the Priority bits in the RAMBAR to make sure the FEC has priority over the CPU. Make sure you don't have other DMA happening that might be starving the FEC.

> I can check it via wireshark

There are too many things between the FEC and Wireshark, and they have all sorts of different reasons to drop or not forward packets.

The Ethernet controller on the PC you're running Wireshark on won't receive a packet that has an error, so it won't forward it to Wireshark, and so Wireshark won't report anything. It can't "see" broken packets. If you're running Wireshark on a Linux box, then check the Ethernet statistics for received Framing or CRC errors (/sys/class/net/eth0/statistics, or somewhere like that). If you're going through a single switch you might see CRC errors.

The only way you can know what the FEC is doing is to monitor the pins between it and the PHY. I'm guessing it is sending the whole packet.

You've probably got your PC and your device plugged into an Ethernet switch. Switches can drop/abort packets on errors. Ditch it and connect the PC directly to the device (you may need a crossover cable). That might allow you to receive packets. There are reasons why the switch might be refusing to forward your packets.

Here's what I think is going wrong. I've seen this before. I remember a machine that sent packets that could send short packets anywhere, and long packets directly to PCs on the 10BASE-T ethernet, but which wouldn't go through a hub. The problem was that the CRYSTAL on the transmitter was out of spec. I seem to remember the ethernet specifications require that the crystal be within 100ppm - yes, Google confirms my memory:

Crystal oscillator frequencies - Wikipedia, the free encyclopedia

Here's what happens if you get that wrong. If there's no intervening switch/hub, the receiver can handle pretty much any sort of frequency error. If there is an intervening switch/hub then the transmitter has to be accurate (but the receiver doesn't). Assume the transmitter is 1000ppm slow. That means that over a 1000 byte message, the last bit will be 8 bit-times later than the switch expected. It has an internal "elasticity buffer" to handle different incoming and outgoing speeds, but it can only handle so much. Hubs used to only allow 4 or 5 bits. I imagine switches should be able to handle more, but they may adhere to the specs. So short packets get through, but when the packet hits the length that overruns/underruns the switch buffer, the switch drops/corrupts the packet.

So check your crystal, or check the output clock from the CPU with an accurate frequency meter. Check the PLL filtering to make sure it doesn't have too much jitter.

Type "MCF52259 FEC long" into the Search on this page and check the other matching posts. Try again without the CPU qualifier and check the results.

Tom

[marcomaioli]

Hi Tom,

> But you should be getting EIR[UN] (Transmit FIFO underrun) if this happens. Check for this.

I checked it, but I never see an underrun.

> Check the Priority bits in the RAMBAR to make sure the FEC has priority over the CPU. Make sure you don't have other DMA happening that might be starving the FEC.

The priorities were previously controlled and the FEC has the highest priority.

> I remember a machine that sent packets that could send short packets anywhere, and long packets directly to PCs on the 10BASE-T ethernet, but which wouldn't go through a hub.

I tried to connect the device directly to some PC, but the problem is always there.

> So check your crystal, or check the output clock from the CPU with an accurate frequency meter. Check the PLL filtering to make sure it doesn't have too much jitter.

I will check anyway even if I think it is a different problem.

Reading the manual i found this: 21.4.19 Transmit FIFO Watermark Register (TFWR)

"This allows you to minimize transmit latency (TFWR = 00 or 01) or allow for larger bus access

latency (TFWR = 11) due to contention for the system bus. Setting the watermark to a high value

minimizes the risk of transmit FIFO underrun due to contention for the system bus."

So I try to set TFWR from 00 to 11:

MCF_FEC_TFWR = MCF_FEC_TFWR_X_WMRK(MCF_FEC_TFWR_X_WMRK_192);

This seems to solve the problem, although I do not understand why I never saw any Transmit FIFO underrun.

[marcomaioli]

Hi,

I think I was right to say SEEMS resolved.

Using the device I noticed that the problem occurs again, but with packets of length 1300-1400 bytes.

In any case it seems that the change to the TFWR register has improved the situation.

Now I checked the PLL filtering. The jitter is ~58.4 mV (on 3.3 V).

System clock: 80MHz

CLOCKOUT period jitter:

  - Peak-to-peak: ~ 4MHz (8MHz maximum from the datasheet)

  - Long-term (average over 2 ms interval): ~30-50KHz (8KHz maximum from the datasheet)

We have checked the frequency with an oscilloscope.

[TomE]

> I checked it, but I never see an underrun.

From your symptoms (of changing the TX FIFO limit) it sure looks like you're getting underruns. Are you sure you're handling the EIR register properly?

Other broken chips (from other manufacturers) require to do to a read-clear-write to clear interrupt bits. Freescale/Motorola have a long tradition of having "write one to clear" bits, and all the bits in the OVR register are like this. So you should only ever be writing single bits to the EIR to clear known interrupt sources. If you're reading-then-writing or writing all-ones, then you may be accidentally clearing the OF bit in one piece of your code, which is why other pieces of your code might not be seeing it set.

It looks like (but doesn't say) that there's 512 bytes of FIFO RAM. By default this is partitioned 256 bytes for the receiver and 256 for the transmitter. You can move this boundary if you want, but as you can't increase the TFWR beyond 192 this wouldn't help. You might also start overrunning on receive if you reduced the size of that.

Are you sending a single 1500-byte transmit buffer or are you sending linked smaller buffers? If you're using small ones then the overhead of finding the next buffer can cause underruns. The manual doesn't say this for Transmit Buffers, but for Receive it does say:

21.4.24 Receive Buffer Size Register (EMRBR)

To minimize bus utilization (descriptor fetches), it is recommended that EMRBR be greater than or equal

to 256 bytes.

So you should do the same with Transmit Buffers. These should also be on four-byte boundaries, and possibly also modulo four-bytes long too. If you are using small ones, try transmitting from a single large one instead, and see if that fixes the problem.

I'd check the RAMBAR and SCM RAMBAR registers again. Are they programmed to use the same addresses or different ones?

Do you have any DMA running? Throttle that back if you are. Something is keeping the FEC off the bus. If it isn't the DMA then it must be the CPU. What is the CPU doing while waiting for a transmit, is it polling or running other code in a proper OS? If it is polling, then I'd suggest making the polling loop slower, and don't rapidly poll any FEC control registers, as that may be making the FEC busy. You should be waiting for an interrupt. You cold also stick a divide instruction into the polling loop - that will get the CPU off the bus.

Do you have the AFS bit set in FLASHBAR? That might free the bus up. Can you make the CPU wait in a code loop in RAM instead of in FLASH? That might help.

Tom

[marcomaioli]

Hi Tom,

> From your symptoms (of changing the TX FIFO limit) it sure looks like you're getting underruns. Are you sure you're handling the EIR register properly?

You're right. There was a problem in my code that did not show me the underrun.

So now I have confirmation that the problem was the underrun.

> Are you sending a single 1500-byte transmit buffer or are you sending linked smaller buffers?

I use a single 1500-byte transmit buffer.

> I'd check the RAMBAR and SCM RAMBAR registers again. Are they programmed to use the same addresses or different ones?

They have the same addresses: 0x20000000.

> Do you have any DMA running? Throttle that back if you are. Something is keeping the FEC off the bus.

I use all 4 DMA channels with this configuration:

DMA0: UART1 rx

DMA1: UART1 tx

DMA2: UART2 rx

DMA3: UART2 tx

I tried to disable all DMA channels and the problem disappears. However I need to use them.

So checking the manual I saw this:

21.4.21 FIFO Receive Start Register (FRSR)

"Hardware initializes the FRSR register at reset. FRSR only needs to be written to change the default value."

The default value of FRSR[R_FSTART] is 0x40 and the minimum accepted value is 0x48, but I never write this register.

So I added in the init write 0x48 FRSR[R_FSTART].

With this change, I no longer have had problems.

Thanks Tom for your helpful answers.

[TomE]

> I use all 4 DMA channels with this configuration:

Not using the USB controller? That uses DMA too.

What baud rate are you running at? If at the maximum usual baud rate of 115,200 they should only be 4*11500 or 46,000 bytes/second or one byte every 22us. That shouldn't be any impact. You could be running the UARTs at up to 2.5MHz, which would give a maximum 1MB/s DMA loading. Even that shouldn't be an obvious problem.

The FEC only has to read 12MB/s to feed 100MHz Ethernet, or 24MB/s for full duplex. That's only 6 MW/s (32-bit words). The RAM should be capable of 80MW/s, but the DMA controllers probably aren't that efficient. The Manual says it will run full duplex with a 50MHz clock and you're running at 80MHz.

The DMA controllers are highly configurable.

Reading the UART chapter says to run them one byte per DMA transfer in "cycle steal" mode (read one, write one). It might be possible to set the RAM side up to read and write 4 bytes at a time, which might reduce the loading on the FEC.

The DMA Controllers in these chips aren't as efficient as you might think they should be. They're effectively "microcoded" and can take longer to do the calculations (incrementing addresses and decrementing counters) that the CPU would take. That said, they shouldn't hog the bus, but should only hold it while doing a transfer.

Have a look at the MPARK register. I assume you've left it alone with the FEC having the highest priority? The default is round-robin, so I'd suggest you change it to fixed priority to guarantee the FEC always wins. Play with "Park" as well. If you look at the Arbiter diagram you'll see why it is a bad idea for the CPU to be polling any peripheral registers anywhere. These have to go through the arbiter and slow down the FEC accesses. The CPU can get to Flash and SRAM without going through the arbiter.

> The default value of FRSR[R_FSTART] is 0x40 and the minimum accepted value is 0x48,

Which has to be a documentation error. How can a register be initialized with 0x40 and that value be less than the minimum legal value?

I work on two projects using FECs. Checking their code finds we don't touch this register, and our FECs work fine. Just today I've been hammering the FEC in the MCF5235 with 1300-byte flood pings with 120 packets preloaded AND with 256-byte linked receive buffers (in SRAM). It hasn't dropped a single one at that rate.

If this is an error it is a very common error. Searching finds the phrase "ensure that R_FSTART is set to 0x48 or greater" appears in the manuals for at least the 51CN128, 5208, 52259, 5282, 5301x, 5373, 54455, and 5329. Surprisingly the 5235, 5272 and other manuals don't have this phrase in it. The content depends on which ones were cut/pasted together, and I can see two branches in this documentation tree.

I can't see how changing that register should stop it overrunning. Are you sure you didn't do something else in your code at the same time that may have changed the way it runs and stopped the overrun?

Tom

[TomE]

The biggest competitor (for the bus) to the FEC Transmitter is the FEC Receiver. Make sure you're generating wall-to-wall receive traffic (for instance a flood ping) when you're testing for transmit overrun.

In your original post you said:

> When the size is >950 byte the FEC simply doesn't send out the packet.

So this wasn't an intermittent problem, but always happened at a particular size. That's very strange as I'd expect external influences (FEC receive traffic) to have an effect. So it looks like exactly 950 byte-times after your code started a FEC Transmit it did something specific that locked up the bus and stopped the FEC from filling its FIFO in time. Changing the FIFO depth just changed the timing so it happened at 1300-1400 bytes.

You weren't detecting FEC errors properly before. Will you now detect FEC Receive overruns if they happen?

Will your code detect DMA Bus Error terminations? If for some reason you're starting a DMA transfer and get the address wrong, you might be running into the "IPS Bus Timeout Monitor". The DMA would be stalled until the Monitor times out (default 1024 bus cycles) and during that time the FEC may have underrun. A recent example of this sort of problem was in this thread, where it caused huge interrupt latency:

https://community.freescale.com/message/326456#326456

If you don't find exactly what is going wrong it might happen again when code or anything changes. Make sure you have some code in there to retry failed transmissions a few times if they happen again. And log them somewhere so the device tells you when this happens.

For best CPU performance you should try to have its variables and stack in the opposite half of the RAM to where the FEC buffers are. Otherwise when you start a FEC transmit the CPU may be dead-stalled for the time it takes the FEC to read the first 192 bytes.

Tom