FTP Server Large File Data Corruption

[DamonGibson]

I'm using CodeWarrior 10.6 with MQX 4.2.

The problem I’m having is this:

1. I have the MQX Micro SD driver, Ethernet and FTP Server running on the K70 tower kit.
2. Using Windows Explorer as an FTP Client, I can successfully FTP files to and from the Micro SD card.
3. When FTPing a large file (162 MB) from the Micro SD card to my PC, the transfer completes and the file size is exactly correct.
4. However, when I compare the FTPed file to the original file, there are from 2 to 4 16-byte blocks that are incorrect.  The number and location of these “bad data blocks” vary from one transfer to the next.  All other bytes are correct.

I've tried other FTP Clients and other Micro SD cards, with the same results.

Has anyone else seen this problem?  Any ideas on how to troubleshoot would be welcome.

[Carlos_Musich]

Hi Damon,

I find this very strange because we have never received reports about FTP Server.

What kind of file is this?

Are you able to open file on Windows once you retrieve from K70 FTP server?

Does this happens always? Or just some times?

Does this happens with only with one file? Or with different files?

What tool are you using to compare the binary files?

We would need to reproduce this behavior in order to track the failure. Could you please share the file that is causing the problem and some screenshots where you see the difference between the files?

Regards,

Carlos

[DamonGibson]

Hello Carlos,

Thanks for responding.

I have tried several different file types.  My 162 MByte file is just a binary log file created by one of our products.  This week I have been testing with the K70 Reference Manual (K70P256M150SF3RM.pdf), which is 17,846,531 bytes long and I'm seeing the same problem.

Yes, I can open the K70P256M150SF3RM.pdf in Windows after retrieving it.

On the large binary file (162 MB), the errors happen every time, but the number of errors and their locations are different each time.  On the smaller .pdf file (17 MB), I successfully retrieved the file once without error, but found at least one bad 16-byte block in all other tries.

I initially just used the Windows Command Prompt fc command to compare.  Then I started using VBinDiff, so I could see what the errors were.

The VBinDiff screenshot below shows the FTP transferred file in the top pane and the original file in the bottom pane.  The bytes that are in error are highlighted in red.  For this FTP, the highlighted 16-byte block were the only errors.  All other bytes matched.

The K70P256M150SF3RM.pdf file is available on the Freescale web site:

     http://cache.freescale.com/files/microcontrollers/doc/ref_manual/K70P256M150SF3RM.pdf?fsrch=1&sr=1&p...

Thanks again,

Damon

[Carlos_Musich]

Hi Damon,

I need to reproduce this behavior. If you can share a project where I can see the problem it may save a lot of time.

Have you tried TFTP? It is important to determine if the problem is in in FTP or in file system.

And also... have you installed MQX4.2.0.1 and MQX4.2.0.2 patches? Please be sure to install these updates as many RTCS bugs have been fixed here. When installing MQX4.2.0.2 patch you may need to replace manually  the original source files contained in MQX installation with the files contained in the patch.

Regards,

Carlos

[DamonGibson]

Hello Carlos,

Thanks for getting back to me.

Here is a WeTransfer link for my ftp_server_error.zip file, which contains all 10 files from my test project's Sources folder.

     http://we.tl/LA4yasG2NF

The project runs on the K70 Tower Kit and also requires unmodified bsp, mfs, psp, rtcs and shell libraries for the K70 Tower Kit.  Please note, that in the media.c file (derived from Freescale_MQX_4_2\mfs\examples\sdcard\demo.c), I have added code which places the MFS file system in Write-Through mode.

I started to try TFTP a few days ago, as it is a simple matter to add the tftpsrv shell command.  But I didn't have a PC-based TFTP client to try it with.  Do you know where I can get one?

I had installed all of the automatically available updates and when I do a Help | Check for Updates, it says "No updates were found.".  I have the MQX 4.2.0.1 patch, but I have not installed it yet.  I can't find the MQX 4.2.0.2 patch.  Are there also CodeWarrior patches I should install?

Damon

[Carlos_Musich]

Hi Damon,

I recommend tftpd32 TFTP server which you can install on Windows.

Regarding to MQX updates, I apologize, I was confused. MQX4.2.0.2 is not available yet.

I will test your application and let you know my findigs.

Regards,

Carlos

[DamonGibson]

Hello Carlos,

Thanks for the tip regarding tftp32.  Using the 17 MB Freescale K70P256M150SF3RM.pdf file, I did 10 FTP and 10 TFTP file transfers from the K70 Tower's Micro SD card to my PC.

For both FTP and TFTP, 3 of the 10 transfers succeeded without error, but 7 of 10 transfers failed.  The TFTP transfer failure mode was identical to the FTP transfers, in that one or more 16-byte blocks in the received file contained incorrect data (as shown in the example VBinDiff screenshot I posted before).

So, the problem does not appear to be in the FTP Server, which leaves RTCS, MFS or the SDCARD driver as the probable culprit.

As you suggested, I will now apply the MQX 4.2.0.1 patch and do the test again.

Question:  Are there any CodeWarrior 10.6 patches I should install, that might have a bearing on this problem?

Damon

[DamonGibson]

Hello Carlos,

I installed the MQX 4.2.0.1 patch and then re-built and re-ran the FTP test.  I'm still getting one or more 16-byte block errors.

Have you been able to build the project and reproduce the problem?

Damon

[Carlos_Musich]

Hi Damon,

most likely this is an issue with MFS. I have not reproduced this behavior yet, I will test it tomorrow. However, after reproducing the issue the next step will be to report this to MQX development team to fix it in MQX4.2.0.2 patch. Unfortunately there is not much we can do in our side. If MQX developers provide a workaround I will let you know.

Carlos

[DamonGibson · Accepted solution]

Hello Carlos,

I believe I have solved the problem!

Here is what I did:

1.  Completely uninstalled MQX and CodeWarrior

2.  Re-installed CodeWarrior and MQX, being sure to install all updates and patches, before attempting to build anything.  (On my PC, the uninstall and re-install took about 5 hours.)

3.  Built and ran the Hello World example application for the K70 Tower Board (which included building the K70 bsp and psp libraries)

4.  Re-created my FTP Server test project, built it and ran it

5.  FTP transferred my 17 MB test file from K70 Tower to my PC 10 times

6.  No errors occurred!  All 10 copies matched the original file.

7.  FTP transferred my 162 MB test file from K70 Tower to my PC 2 times

8.  No errors occurred!  Both copies matched the original file.

Here is a link to the CodeWarrior 10.6 & MQX 4.2 Installation documentI created, while doing the re-install.

     http://we.tl/SDCdnjMdKl

The document contains a few links to some files on our internal server, but these could easily be replaced with links to the Freescale server.

Maybe my installation document will keep someone else from getting into the same trouble I did.

Sorry to run you through the ringer on this one.

Damon

[Carlos_Musich]

Hi Damon,

so the problem was a corrupted installation. Thank you very much for sharing your document. Actually it would be a good for me to reinstall MQX as I have made a lot of changes in libraries for running different tests. I will follow your guide.

Thanks you!

Best regards,
Carlos

[DamonGibson]

Hello Carlos,

Yes, it seems something was corrupted.  It felt to me like the corruption occurred when I tried adding, editing and deleting Processor Expert components in my project.  At one point, I actually started getting thousands of build errors.  This is why I decided to re-install all the tools and why, in my document, I suggested (perhaps a little too strongly) to avoid using Processor Expert in an MQX project.  Since my re-install, I have done additional Processor Expert component manipulation, with no ill effects (so far).  I'm just always very careful to do a Generate Processor Expert Code and a re-build of the BSP after each change (or set of changes).

Damon

[DamonGibson]

Hi Carlos,

Thanks for staying with me on this.

Speculations:

This feels to me like one of those real-time interaction problems between task and/or interrupt contexts.  The fact that it does not always happen and when it does happen it affects different parts of the file, might mean a missing mutex or interrupt disable/enable pair, to protect a critical section of code (or something similar).

Also, I wonder if the fact that it always seems to affect 16 consecutive bytes, which are always on a 4-byte boundary, might suggest something to someone who is a lot more familiar than I am with the inner workings of MFS and other parts of MQX.  Could it even be hardware-related?  Perhaps a DMA glitch?

I am more than willing to help in the troubleshooting of this problem, if you or someone on the MQX team has any suggestions on where to start, or how I might further narrow down the problem.

Damon