Throw out eMMC(usdhc) errors after 18+ days runing

[changbaoma]

Hello, NXP Experts

MY custom imx6ull board use NXP offical Linux 5.4.24.

eMMC is connected to 2194000.usdhc(mmc1) , and mmcblk1p2 is my ext4 rootfs partition. 

Below is the log:

...

2022-05-04 06:47:32[1629904.121578] sdhci-esdhc-imx 2194000.usdhc: Card stuck in wrong state! card_busy_detect status: 0xe00
2022-05-04 06:47:32[1629904.131436] mmc1: cache flush error -110
2022-05-04 06:47:34[1629905.437686] mmc1: tried to HW reset card, got error -110
2022-05-04 06:47:34[1629905.443333] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.447536] blk_update_request: I/O error, dev mmcblk1, sector 5443624 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[1629905.458937] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 starting block 680454)
2022-05-04 06:47:34[1629905.471948] Buffer I/O error on device mmcblk1p2, logical block 679173
2022-05-04 06:47:34[1629905.486168] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[29905.493028] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.496989] blk_update_request: I/O error, dev mmcblk1, sector 5444048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[1629905.508236] EXT4-fs warning (device mmcblk1p2): ext確end_鈏o:315: I/O error 10 writing to inode 49254 (offset 0 size 0 starting block 680507)
2022-05-04 06:47:34[1629905.521143] Buffer I/O error on device mmcblk1p2, logical block 679226
2022-05-04 06:47:34[1629905.534621] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.541442] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.545611] blk_update_request: I/O error, dev mmcblk1, sector 5444984 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio燾lass 0
2022-05-04 06:47:34[1629905.556922] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 starting block 680624)
2022-05-04 06:47:34[1629905.569862] Buffer I/O error on device mmcblk1p2, logical block 679343
2022-05-04 06:47:34[1629905.583953] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.590770] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.5904] blk_update_request: I/O error, dev mmcblk1, sector 5445104 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[1629905.606078] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O 錼ror 10 writing to inode 49254 (offset 0 size 0 starting block 680639)
2022-05-04 06:47:34[1629905.619000] Buffer I/O error on device mmcblk1p2, logical block 679358
2022-05-04 06:47:34[1629905.6329] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.639714] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.643757] blk_update_request: I/O error, dev mmcblk1, sector 5445928 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[1629905.655029] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset size 0 st醨ting block 680742)
2022-05-04 06:47:34[1629905.667954] Buffer I/O error on device mmcblk1p2, logical block 679461
2022-05-04 06:47:34[1629905.681730] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.688545] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.692579] blk_update_request: I/O error, dev mmcblk1, sector 5446104 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio c靉ss 0
2022-05-04 06:47:34[1629905.703847] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 start閚g block 6764)
2022-05-04 06:47:34[1629905.716771] Buffer I/O error on device mmcblk1p2, logical block 679483
2022-05-04 06:47:34[1629905.730597] sdhci-esdhc-imx 2194000.usdhc: error -110 騟questing status
2022-05-04 06:47:34[1629905.737476] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.741464] blk_update_request: I/O error, dev mmcblk1, sector 5446160 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
2022-05-04 06:47:34[1629905.752764] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 starting block 680775)
2022-05-04 06:47:34[1629905.765689] Buffer I/O error on device mmcblk1p2, logical block 679490
2022-05-04 06:47:34[1629905.772473] Buffer I/O error on device mmcblk1p2, logical block 679491
2022-05-04 06:47:34[1629905.779195] Buffer I/O error on device mmcblk1p2, logical block 679492
2022-05-04 06:47:34[1629905.785951] Buffer I/O error on device mmcblk1p2, logical block 679493
2022-05-04 06:47:34[1629905.799698] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting s鬭tus
2022-05-04 06:47:34[1629905.806592] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.810622] blk_update_request: I/O error, dev mmcblk1, sector 5446448 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
2022-05-04 06:47:34[1629905.821900] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 sta騮ing block 680808)
2022-05-04 06:47:34[1629905.841601] sdhci-esdhc-imx 2194000.usdhc: erro-110 requesting status
2022-05-04 06:47:34[1629905.848407] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.852589] blk_update_request: O error, dev mmcblk1, sector 5446704 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[29905.863878] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10 writing to inode 49254 (offset 0 size 0 starting block 680839)
2022-05-04 06:47:34[1629905.883715] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.890522] mmcblk1: recovery fa閘ed!
2022-05-04 06:47:34[1629905.894515] blk_update_request: I/O error, dev mmcblk1, sector 5448096 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
2022-05-04 06:47:34[1629905.905774] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:315: I/O error 10爓riting to inode 49254 (offset 0 size 0 starting block 681013)
2022-05-04 06:47:34[162990524930] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.9312] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.942488] sdhci-es鋒c-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.949302] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.960029] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.966908] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.977556] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629905.984428] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629905.995447] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629906.002355] mmcblk1: recovery failed!
2022-05-04 06:47:34[1629906.012955]爏dhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:34[1629906.021595] mmc1: cache flush error -110
2022-05-04 06:47:36[1629907.324685] mmc1: tried to HW reset card, got error -110
2022-05-04 06:47:36[1629907.330237] mmcblk1: recovery failed![1629907.342269] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.349086] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.359805] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.366695] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.377436] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.384339] 韒cblk1: recovery failed!
2022-05-04 06:47:36[1629907.395036] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.401915] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.412597] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.419403] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.430066] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.436947] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.447641] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.454514] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.465164] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.472028] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.482601] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.489414] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.500074] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.506959] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.518079] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.525093] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.535777] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.542711] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.546857] JBD2: Detected IO errors while flushing file data on mmcblk1p2-8
2022-05-04 06:47:36[1629907.560807] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.567669] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.578239] Aborting journal on device mmcblk1p2-8.
2022-05-04 06:47:36[1629907.586149] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.593080] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.605276] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.612175] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.616329] EXT4-fs error (device mmcblk1p2) in ext4_setattr:5726: Readonly filesystem
2022-05-04 06:47:36[1629907.630954] EXT4-fs (mmcblk1p2): ext4_writepages: jbd2_start: 2147483614 pages, ino 49254; err -30
2022-05-04 06:47:36[1629907.631746] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.647943] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.659350] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.666378] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.678179] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.685160] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.696653] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.704040] mmcblk1: recoverfailed!
2022-05-04 06:47:36[1629907.715558] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.722749] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.734212] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.741028] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.752914] sdhci-esdhc-imx 2194000.usdhcerror -110 requesting status
2022-05-04 06:47:36[1629907.759729] mmcblk1: recovery failed!
2022-05-04 06:47:36[1629907.770168] sdhci-esdhc-imx 2194000.usdhc: error -110 requesting status
2022-05-04 06:47:36[1629907.777082] mmcblk1: recovery failed!

...

 My eMMC node of dts is:

&usdhc2 {
pinctrl-names = "default", "state_100mhz", "state_200mhz";
pinctrl-0 = <&pinctrl_usdhc2_8bit>;
pinctrl-1 = <&pinctrl_usdhc2_8bit_100mhz>;
pinctrl-2 = <&pinctrl_usdhc2_8bit_200mhz>;
bus-width = <8>;
keep-power-in-suspend;
wakeup-source;
non-removable;
no-1-8-v;
status = "okay";
}; 

Everything  return back nornal after re-power on My imx6ll board.

Any suggestion?

[changbaoma]

@shajin Sorry for the late reply, I haven't resolved the issue yet. I also found that Linux 5.4.70 of imx_5.4.70_2.3.0 has the same issue. I suspect that the issue is related to EMMC itself, as I am using the SkyHigh S40FC004 model. What model is the EMMC of the board you are having problems with?

[shajin]

@changbaoma How did you solve this issue ?  

Our custom platform is powered with i.MX6 running Yocto-Dunfell with QT(5.15.2) on top of it. The version of Linux kernel used in the platform is 5.4.161. 

The eMMC is connected to the usdhc2(0x2198000). The Linux rootfs and Applications are separated and present in different partition of the same eMMC. 

I'm facing similar issue during reboot test. We have a custom setup which will reboot the machine continuously. I'm seeing this error once in 500 reboots.  

The below messages appear on the serial console which is similar like your case. These messages was observed after the init successfully mounted all the filesystems.. 

sdhci-esdhc-imx 2198000.usdhc: Card stuck in wrong state! card_busy_detect status: 0xe00
mmc2: cache flush error -110
mmc2: tried to HW reset card, got error -110
mmcblk2: recovery failed!

Thanks for your time. 

Regards

Shajin