MFS slow when writing many files to usb-memory

[carl-magnusmoon]

I am Writing log files to an usb-memory using MFS and the usb-host stack in MQX 4.2. The files are 1.6Mb each. The tests are performed on a TWR-KF120 board using Sandisk Cruzer Blade 8GB usb-memories.

It starts out pretty good with a write speed of about 500kB/s. However, the performance degrades to half after a couple of hundred files and after like 1000 files it is down to 40kB/s. Is this normal? It seems to me that MFS gets very slow when the usb-memory starts to fill up.

[pavel_chromy]

Dear Carl,

could you be more specific about your use case, in particular, how do you perform write to the logs?

The write operation itself shall not be any slower. Time necessary to seeking to the end of file may increase for large files but it does not depend on number of files in the directory. What gets slower with increasing number of files in the directory is the open operation, while creating a new file being significantly slower.

So if you use case is "open a file, seek to the end, write a record, close" and this all repeated all over it may be very slow once the directory contains many files.

Best regards, Pavel

[carl-magnusmoon]

Yes, you are right. I have done some more tests and it is actually the fopen that takes longer to perform. When using an empty usb-memory it take about 50ms but when the memory starts to get full it take about 8 sec. Maybe it is better to store the files into a number of folders to avoid having so many files in the root folder.

Regarding my logs I open a file, write the log data and close the file. I never seek the end of the file. The log data is stored in nand flash to start with and are written to usb-memory when the user wants to export them. I always write a complete log file at once.

[patrickcaujan]

Hello Carl,

I've exactly the same problem with an SDCARD of 8GB.... In fact my application a file of 64kB on the SDCARD, the time to write the file increase according to the number of files. Since the 130th, it takes 10s to write a file....

Have you got a solution ?

Thanks a lot

[soledad]

Hello Carl,

Which MQX version are you using??

Please try to increase the data buffer size and read more at a time, this will help improve performance.

_mem_alloc(COPY_BLOCK_SIZE); // COPY_BLOCK_SIZE is 512

  if (buffer == NULL) {

  printf("Warning, unable to allocate copy buffer, copy will be slower\n" );

Have a great day,
Sol

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------