MIMXRT1180-EVK LittleFS sample fails on external flash unless DCache is disabled

[JasonPD]

Hi,

I am trying to run the Zephyr LittleFS sample (https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/subsys/fs/littlefs) on the external flash (W25Q128JWSIQ) using the CM33 core but I am running into issues.

When I try to run the sample in its default configuration, it fails about 90% of the time. When I turn on the option to wipe the flash and rebuild the filesystem, it appears to fail almost every time.

The point at which it fails varies, but the log usually looks something like this:

スポイラ

*** Booting Zephyr OS build v4.3.0-6239-g28cd602a81ea ***
Sample program to r/w files on littlefs
Area 3 at 0xe20000 on w25q128jw@0 for 1966080 bytes
[00:00:00.022,000] <err> main: Erasing flash area ... 0
[00:00:00.028,000] <inf> littlefs: LittleFS version 2.11, disk version 2.1
[00:00:00.037,000] <inf> littlefs: FS at w25q128jw@0:0xe20000 is 480 0x1000-byte blocks with 512 cycle
[00:00:00.047,000] <inf> littlefs: partition sizes: rd 16 ; pr 16 ; ca 64 ; la 32
[00:00:00.055,000] <err> littlefs: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1386: Corrupted dir pair at {0x0, 0x1}
[00:00:00.066,000] <wrn> littlefs: can't mount (LFS -84); formatting
[00:00:00.118,000] <inf> littlefs: /lfs mounted
/lfs mount: 0
/lfs: bsize = 16 ; frsize = 4096 ; blocks = 480 ; bfree = 478

Listing dir /lfs ...

[00:00:00.150,000] <wrn> littlefs: WEST_TOPDIR/modules/fs/littlefs/lfs.c:2109: Superblock 0x0 has become unwritable
[00:00:00.161,000] <err> fs: file open error (-5)
[00:00:00.167,000] <err> main: FAIL: open /lfs/boot_count: -5
[00:00:00.174,000] <inf> littlefs: /lfs unmounted

/lfs unmount: 0

At some point I get the "Superblock 0x0 has become unwritable" error and then the sample fails.

I found that disabling the DCache with the Kconfig option "CONFIG_DCACHE=n" makes the sample work reliably, leading me to believe that this is a caching issue.

スポイラ

*** Booting Zephyr OS build v4.3.0-6239-g28cd602a81ea ***
Sample program to r/w files on littlefs
Area 3 at 0xe20000 on w25q128jw@0 for 1966080 bytes
[00:00:00.060,000] <err> main: Erasing flash area ... 0
[00:00:00.069,000] <inf> littlefs: LittleFS version 2.11, disk version 2.1
[00:00:00.127,000] <inf> littlefs: FS at w25q128jw@0:0xe20000 is 480 0x1000-byte blocks with 512 cycle
[00:00:00.139,000] <inf> littlefs: partition sizes: rd 16 ; pr 16 ; ca 64 ; la 32
[00:00:00.151,000] <err> littlefs: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1386: Corrupted dir pair at {0x0, 0x1}
[00:00:00.164,000] <wrn> littlefs: can't mount (LFS -84); formatting
[00:00:00.235,000] <inf> littlefs: /lfs mounted

/lfs mount: 0
/lfs: bsize = 16 ; frsize = 4096 ; blocks = 480 ; bfree = 478 

Listing dir /lfs ...

/lfs/boot_count read count:0 (bytes: 0)

/lfs/boot_count write new boot count 1: [wr:1]
[00:00:00.297,000] <inf> main: Test file: /lfs/pattern.bin not found, create one!
------ FILE: /lfs/pattern.bin ------
01 55 55 55 55 55 55 55 02 55 55 55 55 55 55 55
03 55 55 55 55 55 55 55 04 55 55 55 55 55 55 55
05 55 55 55 55 55 55 55 06 55 55 55 55 55 55 55
07 55 55 55 55 55 55 55 08 55 55 55 55 55 55 55
09 55 55 55 55 55 55 55 0a 55 55 55 55 55 55 55
0b 55 55 55 55 55 55 55 0c 55 55 55 55 55 55 55
0d 55 55 55 55 55 55 55 0e 55 55 55 55 55 55 55
0f 55 55 55 55 55 55 55 10 55 55 55 55 55 55 55
11 55 55 55 55 55 55 55 12 55 55 55 55 55 55 55
13 55 55 55 55 55 55 55 14 55 55 55 55 55 55 55
15 55 55 55 55 55 55 55 16 55 55 55 55 55 55 55
17 55 55 55 55 55 55 55 18 55 55 55 55 55 55 55
19 55 55 55 55 55 55 55 1a 55 55 55 55 55 55 55
1b 55 55 55 55 55 55 55 1c 55 55 55 55 55 55 55
1d 55 55 55 55 55 55 55 1e 55 55 55 55 55 55 55
1f 55 55 55 55 55 55 55 20 55 55 55 55 55 55 55
21 55 55 55 55 55 55 55 22 55 55 55 55 55 55 55
23 55 55 55 55 55 55 55 24 55 55 55 55 55 55 55
25 55 55 55 55 55 55 55 26 55 55 55 55 55 55 55
27 55 55 55 55 55 55 55 28 55 55 55 55 55 55 55
29 55 55 55 55 55 55 55 2a 55 55 55 55 55 55 55
2b 55 55 55 55 55 55 55 2c 55 55 55 55 55 55 55
2d 55 55 55 55 55 55 55 2e 55 55 55 55 55 55 55
2f 55 55 55 55 55 55 55 30 55 55 55 55 55 55 55
31 55 55 55 55 55 55 55 32 55 55 55 55 55 55 55
33 55 55 55 55 55 55 55 34 55 55 55 55 55 55 55
35 55 55 55 55 55 55 55 36 55 55 55 55 55 55 55
37 55 55 55 55 55 55 55 38 55 55 55 55 55 55 55
39 55 55 55 55 55 55 55 3a 55 55 55 55 55 55 55
3b 55 55 55 55 55 55 55 3c 55 55 55 55 55 55 55
3d 55 55 55 55 55 55 55 3e 55 55 55 55 55 55 55
3f 55 55 55 55 55 55 55 40 55 55 55 55 55 55 55
41 55 55 55 55 55 55 55 42 55 55 55 55 55 55 55
43 55 55 55 55 55 55 55 44 55 55 55 55 55 55 55
45 55 aa
[00:00:00.659,000] <inf> littlefs: /lfs unmounted

/lfs unmount: 0

However, if I leave the DCache enabled and flush and invalidate the DCache with "sys_cache_data_flush_and_invd_all()" after each modification of the flash (write and erase, on sample level, not on driver level) while interrupts are locked with "irq_lock()", it still does not work.

Disabling the DCache works as a temporary workaround but is not ideal for a production system. Is there something I am missing to get it to work?

The samples accessing the flash that are provided with the MCUXpresso SDK work fine, suggesting it is probably an issue with Zephyr or its configuration.

Other Zephyr samples accessing the flash show similar behavior, where only disabling the DCache allows reliable access but I have not done thorough testing on those.

Technical information:
Board: NXP MIMXRT1180-EVK (CM33 Core), connected via Micro USB to an Ubuntu VM via passthrough from a Windows PC
OS: Zephyr v4.3.0-6239-g28cd602a81ea
IDE: MCUXpresso for VS Code 26.3.72 (with Zephyr Developer Software Kit 4.3) on Ubuntu
Project Kconfig:

スポイラ

#
# Copyright (c) 2019 Peter Bigot Consulting, LLC
#
# SPDX-License-Identifier: Apache-2.0
#

# Optionally force the file system to be recreated
CONFIG_APP_WIPE_STORAGE=y

CONFIG_DEBUG=y
CONFIG_NO_OPTIMIZATIONS=y
CONFIG_LOG=y
CONFIG_PRINTK=y
CONFIG_SHELL=y
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_LOG_BACKEND_UART=y
CONFIG_LOG_MODE_IMMEDIATE=y
CONFIG_UART_CONSOLE=y

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_FLASH=y
CONFIG_FLASH_MAP=y

# workaround, required for flash to work reliably
CONFIG_DCACHE=n

Any help would be appreciated.

Best regards,
Jason

[EdwinHz]

Hi @JasonPD,

You issue is most definitely cache related, specifically what is described by the following article: Using NonCached Memory on i.MXRT

As mentioned there, the issue is most easily observed when using middleware like LwIP, and can be solved by placing critical data into non-cacheable memory regions (instead of outright disabling D-cache).

I recommend you check out that post, as well as the referenced Application Note (Using the i.MXRT L1 Cache) to better understand the issue, and follow the recommendations of the article to ensure a proper functionality of the LwIP middleware. 

BR,
Edwin.

[ThomasR2D2]

I did some more digging and looked in the flash driver 'flash_mcux_flexspi_nor.c'. In there I found cash invalidation after write and erase protected by a define guard:
#ifdef CONFIG_HAS_MCUX_CACHE

For the CM33, which we are currently using exclusively, the KConf file says this:
config SOC_MIMXRT1189_CM33
    select HAS_MCUX_XCACHE

For the CM7 it looks like this:
config SOC_MIMXRT1189_CM7
    select HAS_MCUX_CACHE

I adjusted the define guards in the flash driver like so:
#if defined(CONFIG_HAS_MCUX_CACHE) || defined(CONFIG_HAS_MCUX_XCACHE)

With this change, I can now write and erase the flash without errors. I Re-enabled XIP and DCache and removed all IRQ disabling - seems to work for now.

Could you comment if this solution makes sense from NXPs point of view? Do we still need to disable IRQs when accessing flash?

Regards
Thomas

[ThomasR2D2]

Hello Edwin,

thank you for taking the time to help with this issue, but I don't quite understand the solution you suggested. The same example works with FreeRTOS and there is no LWIP Stack or middleware invovled - it is the plain LittleFS flash read/write example that is not working with Zephyr and working with FreeRTOS.
I understand, that critical buffers data should be put into non-cachable memory - but there is no subsystem or software, other than what Zephyr provides. Are you suggesting, that we try to identify the buffers in the Zephyr driver or subsystems that maybe cause the problem and move them to non-cachable memory?
I’d appreciate it if you could clarify and share any experience you have regarding which buffers are the usual suspects.

Regards
Thomas