Howto flush MFS? (Use case: mfs onto ramdisk)

[Fabi]

Hi all,
Assume the following use case:
1. setup MFS on ramdisk
2. open file, stored on MFS, and update the content
3. readout ramdisk raw data
In step 3 the content of ramdisk is unchanged. Why? Is where a double buffer or so for storing the file content?

I need up-to-date raw data to transmit it over USB device periodically.

The stripped (i.e. error handling and remarks are deleted) setup code is as follows:
_io_mem_install ( "ramdisk:", NULL, DISK_SIZE );
ramdisk_handle = fopen ( "ramdisk:", 0 );
_io_mfs_install ( ramdisk_handle, "ram:", 0 );
fat_handle = fopen ( "ram:", NULL );
ioctl ( fat_handle, IO_IOCTL_DEFAULT_FORMAT, NULL );
cache_mode = MFS_WRITE_THROUGH_CACHE;
_io_ioctl ( fat_handle, IO_IOCTL_SET_FAT_CACHE_MODE, &cache_mode );
_io_ioctl ( fat_handle, IO_IOCTL_SET_WRITE_CACHE_MODE, &cache_mode );
_io_ioctl ( fat_handle, IO_IOCTL_FAT_CACHE_OFF, NULL );

2. Update the file content (stripped code):
handle = fopen ( filename, "w+" );
fseek ( handle, 0, IO_SEEK_SET );
write ( handle, txt_buf, strlen(txt_buf) );
fflush ( handle );
fclose ( handle );
_io_ioctl ( fat_handle, IO_IOCTL_FLUSH_FAT, NULL );

3. readout the raw data:
fseek ( ramdisk_handle, lba_data_ptr->offset, IO_SEEK_SET );
read ( ramdisk_handle, lba_data_ptr->buff_ptr, lba_data_ptr->size );

[pavel_chromy]

Hello,

recently there have been numerous requests related to flushing of MFS buffers. There is a fix for this included in the upcoming release (4.0.1), please see the patch below.

However it is really not a very good idea to work with raw image of a filesystem while it is mounted.

If you export a mounted ramdrive over USB as a mass storage device, even if it is read only, data consistency cannot be guaranteed - there may be always buffering on the other side.

This is not an MFS limitation, the root of the problem is in the principle, thus I strongly discourage you doing this.

diff --git a/mfs/source/generic/mfs_init.c b/mfs/source/generic/mfs_init.c
index 7a1d2e7..71866a5 100644
--- a/mfs/source/generic/mfs_init.c
+++ b/mfs/source/generic/mfs_init.c
@@ -783,9 +783,35 @@ int_32 _io_mfs_ioctl
 #endif
             if (result == MQX_OK)
             {
-                MFS_lock(file_ptr, &drive_ptr);
-                MFS_Flush_caches(drive_ptr);
-                MFS_unlock(drive_ptr,FALSE);      
+                MFS_HANDLE_PTR handle;
+
+                /* lock the filesystem and obtain file handle */
+                result = MFS_lock_handle(file_ptr, &drive_ptr, &handle);
+                if (result == MFS_ERROR_INVALID_FILE_HANDLE)
+                {
+                    /* file_ptr is not associated with a particular file, at least try to lock the filesystem */
+                    handle = NULL;
+                    result = MFS_lock(file_ptr, &drive_ptr);
+                }
+
+                if ( result == MFS_NO_ERROR )
+                {
+                    if (handle != NULL && handle->TOUCHED)
+                    {
+                        TIME_STRUCT time;
+                        DATE_STRUCT clk_time;
+
+                        _time_get(&time);
+                        _time_to_date(&time, &clk_time);
+                        NORMALIZE_DATE(&clk_time);
+                        htods(handle->DIR_ENTRY.TIME, PACK_TIME(clk_time));
+                        htods(handle->DIR_ENTRY.DATE, PACK_DATE(clk_time));
+                        result = MFS_Update_entry(drive_ptr, handle);
+                    }
+
+                    MFS_Flush_caches(drive_ptr);
+                    MFS_unlock(drive_ptr,FALSE);
+                }
             }
             break;
 
diff --git a/mfs/source/generic/mfs_write.c b/mfs/source/generic/mfs_write.c
index aa38dd1..d8f0372 100644
--- a/mfs/source/generic/mfs_write.c
+++ b/mfs/source/generic/mfs_write.c
@@ -343,6 +343,10 @@ uint_32  MFS_Write
     {
         fd_ptr->SIZE = location;
         htodl(handle_ptr->DIR_ENTRY.FILE_SIZE, fd_ptr->SIZE);
+        if ( (drive_ptr->WRITE_CACHE_POLICY == MFS_WRITE_THROUGH_CACHE) && (error == MFS_NO_ERROR) )
+        {
+            error = MFS_Update_entry(drive_ptr, handle_ptr);
+        }
     }
 
     /* LOCATION moves forward only by actual data written, the zero fill does not count */

[jonny]

Hi,

we have been having problems with data corruption on sdcards also (other posts on the matter lead me here).

Just to confirm, does the latest release MQX 4.0.1 greatly improve the situation or do you recommend I still need to apply any of these patches to further minimize corruption issues in the event of power loss or reset?

Thanks

Jon

[pavel_chromy]

Dear Jonathan,

the flushing work as expected since 4.0.1, so if you do regular flushing then the risk of fs corruption is minimize.

There is certainly a chance for power loss to occur just during write operation or between write and flush but besides this it may be overall considered safe (if flushing occurs regularly).

Best regards, Pavel

[christianleeb]

Dear Pawel,

I'm new to MFS. I'll be needing MFS to occasionally store log entries to a file.

If I apply the suggested flushing procedure, what's the worst case that can happen if power fails during a write operation: the entire file can get corrupted or only one or two entries?

Regards, Chris

[t_k]

Hi Fabi,

I was wondering if you managed to get this figured out.  I'm attempting much the same thing as you.  When looking at the file from the PC side of things, my file is only being updated every 12 seconds.    Seems like a strange number for updating.

Thanks,

Travis

[aimozg · Accepted solution]

Helli Fabi,

I had similar problem - I was writing files to SD card which could be ejected or power could be turned off at any time, so I needed flushing. The same operations you used didn't work, when I create file and write data it appeared as 0 bytes size, and missed chunks were found with CHKDSK.

I have a workaround by modifying MFS source code.

At the end of mfs_write.c:

if ( fd_ptr->LOCATION > file_size )
{
fd_ptr->SIZE = fd_ptr->LOCATION;
htodl(handle_ptr->DIR_ENTRY.FILE_SIZE,fd_ptr->SIZE);
handle_ptr->TOUCHED = 1;
}
/// add this:
if (handle_ptr->TOUCHED)
{
drive_ptr->DIR_SECTOR_DIRTY = TRUE;
}

/// ^^^ add this

error = MFS_unlock(drive_ptr, FALSE);

If directory sector corresponding to file won't ve marked as dirty, its updated contents won't be flushed (they will be saved only on fclose()). So file will be listed as zero size with no first sectors (its data will be in lost chain of sectors that could be found with CHKDSK).

With this patch the FAT is valid even when I eject SD card in the middle of write; the last data written are saved.

I am using MQX 3.8 with TWR-K60F120M.

[aimozg]

... And then it stopped working (I get 0 size files). Investigating.

[aimozg]

Sorry, I had an error in my application. The solution above works just fine.

[emilien]

Hi,

     I had the same issue with 0Bytes size if I don't do fclose(file_descriptor). I tried your fix but it still gives the same results. I did try with both MQX3.8 and MQX4.0 ...

I think it's not a cache issue because I disabled it with:

param = MFS_WRITE_THROUGH_CACHE;

error_code = ioctl(filesystem_handle, IO_IOCTL_SET_FAT_CACHE_MODE, &param);

param = MFS_WRITE_THROUGH_CACHE;

error_code = ioctl(filesystem_handle, IO_IOCTL_SET_WRITE_CACHE_MODE, &param);

error_code = ioctl(filesystem_handle, IO_IOCTL_FAT_CACHE_OFF, NULL);

Without cache, when I step into fflush(fd_ptr), I run MFS_Flush_directory_sector_buffer and I have drive_ptr->DIR_SECTOR_DIRTY set to FALSE. However if I set caches options to MIXED_MODE (or WRITE_BACK), I have it set to TRUE and I run MFS_Write_device_sector(...). fflush(fd_ptr) returns no error. However if I remove the SD card at this point, I still have a file with 0 bytes...

The only difference I can see with fclose is that it runs MFS_Update_entry before the flush.

the dosfsck (linux utility) shows this:

emilien@Helferi:~$ sudo dosfsck -v /dev/sdd1

dosfsck 3.0.9 (31 Jan 2010)

dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN

Checking we can access the last sector of the filesystem

Boot sector contents:

System ID "MFS30000"

Media byte 0xf8 (hard disk)

512 bytes per logical sector

16384 bytes per cluster

32 reserved sectors

First FAT starts at byte 16384 (sector 32)

2 FATs, 32 bit entries

7814144 bytes per FAT (= 15262 sectors)

Root directory start at cluster 2 (arbitrary size)

Data area starts at byte 15644672 (sector 30556)

1953028 data clusters (31998410752 bytes)

0 sectors/track, 0 heads

0 hidden sectors

62527456 sectors total

Checking for unused clusters.

Reclaimed 2 unused clusters (32768 bytes).

Checking free cluster summary.

Free cluster summary wrong (1953022 vs. really 1953024)