Byte swapping in few places while downloading files from MCF5485.

[Vikki_ERL]

Hi,

We are using MCF5485 processor. Experiencing problem while downloading files from Server(Target). We find no problems with upload of files but while downloading there are lot of retransmissions and because of that speed is very low

(10-20Kbps). When we compared file it is found at some places say (2 or 3 places) the bytes are swapped 1st and 4th byte getting swapped..For 1 MB file swapping is there in only 2 to 3 places.. I tried capturing data while it is passing to DMA descriptors. Threre swapping was not happening..Not sure how to proceed from here.

It would be of great help if anyone help me out in resolving this..

[TomE]

Have you read the chip errata? This chip has more than 30 items on the list.

You may be getting SECF175, which can corrupt data freom the FEC, PSC or PCI.

The workaround is "add checksums to your data", and for Ethernet, 'use TCP instead of UDP".

> When we compared file it is found at some places say (2 or 3 places) the bytes are swapped

Do you mean that after getting through the Ethernet and the network stack the copied FILES are still corrupted? Or do you mean you're storing the raw data prior to the DMA controller.

If your files are corrupted, then either the data is getting so corrupted, that every now and then two corruptions in the one packet cancel each other out.

There's a well known problem with TCP checksumming. It only checksums 16-bit WORDS in a packet. This means that if any 16-bit words are exchanged in a packet the checksum gives the same result and the error is undetected. This can happen for any block of data getting swapped. A non-zero word in a block of zeros can move to the wrong word and not be detected.

> while downloading there are lot of retransmissions and because of that speed is very low

Check to see if your network stack has error counters that say if it is dropping packets, and if so, why. Corrupted packets may be getting dropped because the checksums are wrong. A count of how many bad packets per good packets would be useful.

If the network stack doesn't have these, add some print statements to it to print on errors.

Since the "download" is failing, you might have flow-control related problems. If you're using TCP then a well-written stack should handle this OK. What stack are you using, I've come across some severely broken ones that don't work properly when pushed (like when transferring files and hitting the flow-control points).

Tom

[Vikki_ERL]

Thanks for your reply

We are using Exress logics NetX stack. and for data transfer we are using TCP only.

What I meant by byte swapping is, as you said inside a same packet the 1st and 4th byte(character) is getting swapped..For one MB file it is happening in only 3 or 4  places.

Please find the attached snapshot where the left one is original file and right one same transferred back from target.

I couldn't understand what you mean by flow control points..I'm a newbie to this field..Could you please help me out in checking out this..

Sincere Thanks,

Vignesh

[TomE]

> 1st and 4th byte(character) is getting swapped.

You can't count!

The "swapped bytes" in the example you give are from "receive, it has" to "receivt, ie has".

That's the First and FIFTH byte. That makes a big difference.

The checksum used by TCP will give the same result for the original and corrupted strings. It ignores any exchange of bits that are an even number of bytes apart - that give the same sum when summed 16 bits at a time.

You're probably getting a lot more corrupted packets that the checksum is detecting and dropping.

Have you done any of the things I suggested in my email, like looking at the TCP statistics in your stack?

Have you read the Errata item and then looked to see if you can follow any of the Workarounds?

You could also run Wireshark and capture all the Ethernet packets, to make sure they're good or bad on the wire.

> I couldn't understand what you mean by flow control points

TCP performs flow control. This means that if the sender is supplying data faster than the receiver can get rid of it, the buffers in the sender and receiver will fill up, but they won't overflow. The data rate is limited by the protocol - you don't have to do anything else. If you were using a more primitive protocol then you'd have to do the "flow control" yourself.

That said, there are a lot of simple and badly written versions of TCP that don't work properly when their buffers are full as it exposes a whole lot of bugs that only trigger under these circumstances. I'm not saying the problem is with your stack, only that it might be. You've bought a commercial one. That's good. The "small and free" ones are the ones I've had trouble with.

It is also possible that TCP is advertising more buffer space than your Ethernet controller can handle, and your code isn't working properly to unload the Ethernet chip in time. This could explain your "10 kbytes/second" problem if the packets aren't all getting corrupted. Try dropping the "Advertised Window" and see if it gets better (read your TCP stack documentation).

Can you run your software and TCP stack of different hardware - with a different model of CPU?

> Exress logics NetX

"Express" I assume. Ask them if there are known problems on your CPU. What OS are you running on? It may be an interrupt or scheduling hazard that is causing the problem.

> I tried capturing data while it is passing to DMA descriptors.

How did you manage to do that?

Tom

[TomE]

If you were getting corruptions uploading data, then you could ask the PC to show you its IP and TCP statistics, and monitor how many bad packets are beng received.

If you're using a PC, you should try "netstat -s" from a command prompt and check the IP and TCP statistics anyway. You'll probably see a lot of "TCP: Segments Retransmitted". The same works (only better :smileyhappy: from Linux if that's what you're using.

You should be able to get the same statistics from your embedded stack somehow.

Yes - NextX is VERY good. Read the NetX Documents (Page 120), enable the statistics in your build and start using "nx_tcp_info_get and nx_tcp_socket_info_get services" to get and print the socket statistics. There are also TCP debug options including a Debug Log. Use them.

If you can get the SNMP-based statistics, then so much the better - there are usually more details.

Is the problem due to SECF067? The fix for this is "Fixed in datecodes XXX0445 and later". Check your datecode, but that is a date in 2004 - they couldn't be that old?

Is the problem due to SECF175? If your network stack provider can't provide a reliable workaround following the instructions in the Errata to "disable the communications interface before reading the data from the FIFO using either the DMA or core", then the only solution is to redesign your board using a part other than the MCF5475, MCF5485, possibly a part from a supplier other than Freescale completely, as Freescale mark this problem as "Will not be fixed."

Tom

[TomE]

> When we compared file it is found at some places say (2 or 3 places) the bytes are swapped

These are in the files that eventually (after lots of retries) managed to get received.

I suspect that you're getting lots of different sorts of data corruptions. You data transfer rate is so slow that maybe half of the packets are getting corrupted. I'm still waiting for you to check the TCP statistics so as to tell me that.

Most of these corruptions are detected by the TCP Checksum, resulting in most bad packets being dropped.

Only a very small percentage of corrupted packets are managing to get past the checksum. So the checksum is going to be acting like a "packet filter". If it was a perfect filter it would only let good packets through, but the TCP checksum isn't perfect.

Only packets that have corruptions that *LOOK LIKE* byte-swaps or bit-swaps are going to get through.

So you may not be getting "byte swaps". You're getting lots of corruptions, but the filter is only letting you see ones that looks like that.

> from "receive, it has" to "receivt, ie has".

That is from "e" to "t". which is from 0x65 to 0x74, or 01100101 or 01110100. So that could also be described as a BIT-swap between two nibbles within that byte, and only when two bit-swaps that cancel each other happen in the same packet does it get through and you see it. There's usually a nibble-wide bus between the CPU and the Ethernet PHY. The data might be getting corrupted there, or even in a switch somewhere else on your network. Have you changed the Ethernet cables and the switch? Have you tried other hardware?

This could still be a software problem. It could be a problem with cache management. Could you try disabling the data cache in the CPU and see if the problem still happens?

Tom