imx8qxp reboot issue after multiple reboots and MMC0: CQE recovery

[janardhanreddy]

@nxp

We are trying to reboot the platform multiple times after writing the userdata partition(16) as part of our emmc validation, but we are seeing that platform is not booting after some loops.

from the serial log it is observed that "mmcblk0p16" having the write failure and getting the kernel panic.

we have observed that in logs we are getting [ 1432.287802] mmc0: running CQE recovery log continuously and getting device i/o errors as mentioned in below log before getting the kernel panic.

1. we would like to know is this the reboot failure is because of connection issue or some HW issue?

2. How can we resolve CQE recovery  error and is the eMMC command Queue is enabled by SW/ HW in this platform.

3. Is the issue from eMMC device or issue with MMC driver from Linux kernel. 

Could you please share your inputs on this issue.

Thanks & Regards

janadareddy

Logs:

[ 1394.824482] healthd: battery l=85 v= 3 t=35.0 h=2 st=2 c=400 fc=4000000 cc=32 chg=a

[ 1432.287802] mmc0: running CQE recovery

[ 1432.293830] mmc0: running CQE recovery

[ 1432.299904] mmc0: running CQE recovery

[ 1432.587829] blk_update_request: I/O error, dev mmcblk0, sector 23487488 op 0x1:(WRITE) flags 0x4000 phys_seg 12 prio class 0

[ 1432.599153] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180979 (offset 0 size 524288 starting block 2936064)

[ 1432.612604] Buffer I/O error on device mmcblk0p16, logical block 928896

[ 1432.612604] Buffer I/O error on device mmcblk0p16, logical block 928896

[ 1432.619273] Buffer I/O error on device mmcblk0p16, logical block 928897

[ 1432.625929] Buffer I/O error on device mmcblk0p16, logical block 928898

[ 1432.632587] Buffer I/O error on device mmcblk0p16, logical block 928899

[ 1432.639249] Buffer I/O error on device mmcblk0p16, logical block 928900

[ 1432.645892] Buffer I/O error on device mmcblk0p16, logical block 928901

[ 1432.652773] Buffer I/O error on device mmcblk0p16, logical block 928902

[ 1432.659593] Buffer I/O error on device mmcblk0p16, logical block 928903

[ 1432.666410] Buffer I/O error on device mmcblk0p16, logical block 928904

[ 1432.673058] Buffer I/O error on device mmcblk0p16, logical block 928905

[ 1432.742702] mmc0: running CQE recovery

[ 1432.747130] blk_update_request: I/O error, dev mmcblk0, sector 23519232 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0

[ 1432.758688] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 181006 (offset 0 size 524288 starting block 2940032)

[ 1432.773967] mmc0: running CQE recovery

[ 1432.778491] blk_update_request: I/O error, dev mmcblk0, sector 23517184 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1432.790018] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 181004 (offset 0 size 524288 starting block 2939776)

[ 1432.805098] mmc0: running CQE recovery

[ 1432.809720] blk_update_request: I/O error, dev mmcblk0, sector 23499776 op 0x1:(WRITE) flags 0x4000 phys_seg 12 prio class 0

[ 1432.821054] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180987 (offset 0 size 524288 starting block 2937600)

[ 1432.836064] mmc0: running CQE recovery

[ 1432.840541] blk_update_request: I/O error, dev mmcblk0, sector 23500800 op 0x1:(WRITE) flags 0x4000 phys_seg 11 prio class 0

[ 1432.851851] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180988 (offset 0 size 524288 starting block 2937728)

[ 1432.866810] mmc0: running CQE recovery

[ 1432.871262] blk_update_request: I/O error, dev mmcblk0, sector 23501824 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1432.882582] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180989 (offset 0 size 524288 starting block 2937856)

[ 1432.897535] mmc0: running CQE recovery

[ 1432.902002] blk_update_request: I/O error, dev mmcblk0, sector 23502848 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1432.913314] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180990 (offset 0 size 524288 starting block 2937984)

[ 1432.928278] mmc0: running CQE recovery

[ 1432.932660] blk_update_request: I/O error, dev mmcblk0, sector 23503872 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1432.943995] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180991 (offset 0 size 524288 starting block 2938112)

[ 1432.959060] mmc0: running CQE recovery

[ 1432.963540] blk_update_request: I/O error, dev mmcblk0, sector 23504896 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1432.974828] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180992 (offset 0 size 524288 starting block 2938240)

[ 1432.989791] mmc0: running CQE recovery

[ 1432.994640] blk_update_request: I/O error, dev mmcblk0, sector 23505920 op 0x1:(WRITE) flags 0x4000 phys_seg 10 prio class 0

[ 1433.005930] EXT4-fs warning (device mmcblk0p16): ext4_end_bio:315: I/O error 10 writing to inode 180993 (offset 0 size 524288 starting block 2938368)

[ 1433.020853] mmc0: running CQE recovery

[ 1434.066195] mmc0: running CQE recovery

[ 1434.071591] JBD2: Detected IO errors while flushing file data on mmcblk0p16-8

[ 1434.072202] mmc0: running CQE recovery

[ 1434.115191] mmc0: running CQE recovery

[ 1434.115195] Aborting journal on device mmcblk0p16-8.

[ 1434.117027] EXT4-fs error (device mmcblk0p16) in ext4_reserve_inode_write:6001: Journal has aborted

[ 1434.120296] mmc0: running CQE recovery

[ 1434.170286] mmc0: running CQE recovery

[ 1434.174155] Buffer I/O error on dev mmcblk0p16, logical block 623374, lost async page write

[ 1434.183180] mmc0: running CQE recovery

[ 1434.202048] mmc0: running CQE recovery

[ 1434.205993] Buffer I/O error on dev mmcblk0p16, logical block 622593, lost async page write

[ 1434.492604] EXT4-fs (mmcblk0p16): I/O error while writing superblock

[ 1434.492608] EXT4-fs (mmcblk0p16): previous I/O error to superblock detected

[ 1434.493443] mmc0: running CQE recovery

[ 1434.498997] Kernel panic - not syncing: EXT4-fs (device mmcblk0p16): panic forced after error

[ 1434.499007] CPU: 2 PID: 177 Comm: kworker/u8:2 Not tainted 5.4.24 #7

[ 1434.499014] Hardware name: Freescale i.MX8QXP MEK (DT)

[ 1434.507853] mmc0: running CQE recovery

[ 1434.509741] Workqueue: writeback wb_wo

[ 1434.509748] Call trace:

[ 1434.509759]  dump_backtrace+0x0/0x134

[ 1434.509773]  show_stack+0x14/0x1c

[ 1434.520870] mmc0: running CQE recovery

[ 1434.524651]  dump_stack+0xa0/0xd8

[ 1434.524658]  panic+0x148/0x338

[ 1434.524665]  __ext4_error_inode+0x0/0x154

[ 1434.524670]  __ext4_std_error+0x90/0x18c

[ 1434.524677]  ext4_reserve_inode_write+0x98/0xb4

[ 1434.524682]  ext4_mark_inode_dirty+0x40/0x210

[ 1434.524688]  ext4_writepages+0xa68/0xd40

[ 1434.524696]  do_writepages+0x64/0x104

[ 1434.524708]  __writeback_single_inode+0x34/0x178

[ 1434.531729] mmc0: running CQE recovery

[ 1434.533591]  writeback_sb_inodes+0x44c/0x9e0

[ 1434.533602]  __writeback_inodes_wb+0x84/0x194

[ 1434.541092] mmc0: running CQE recovery

[ 1434.541354]  wb_writeback+0x138/0x244

[ 1434.541367]  wb_workfn+0x2ac/0x434

[ 1434.546730] mmc0: running CQE recovery

[ 1434.548349]  process_one_work+0x17c/0x330

[ 1434.553526] mmc0: running CQE recovery

[ 1434.555402]  worker_thread+0x32c/0x4b8

[ 1434.555407]  kthread+0x10c/0x11c

[ 1434.555413]  ret_from_fork+0x10/0x18

[ 1434.555424] SMP: stopping secondary CPUs

[ 1434.555433] Kernel Offset: disabled

[ 1434.555438] CPU features: 0x0002,20002008

[ 1434.555441] Memory Limit: none

[ 1434.555446] trusty-log panic notifier - trusty version Built: 15:16:23 Mar 20 2020

[ 1434.647150] Rebooting in 5 seconds..

[janardhanreddy]

Hi Igor, 

Thanks for your response.

=>>We would like to know is the iMX8qxp eMMC CMDQ  support is by SW implementation or HW also support the CMDQ  mode.

Since from the spec we are seeing that emmc device come with iMX8 is 5.0 but we are using the 5.1 eMMC sample for our testing.

=>>What is the default speed of eMMC supported by Imx8

Could you please help me on the above listed points

Thanks & Regards

Janardhan Reddy

[igorpadykov]

Hi janardhanreddy

for that issue one can try to disable eMMC CMDQ feature as actually it will not have large

impact on the performance. Disable eMMC CMDQ reverting MLK-20957-3
https://source.codeaurora.org/external/imx/linux-imx/commit/?h=imx_4.14.98_2.3.0&id=861d1d962e90f124...

Also for rebooting may be useful to recheck sect.4.2.2 Power-down sequence

i.MX 8QuadXPlus and 8DualXPlus Industrial Applications Processors - Data Sheet

Note
"When switching off supply group 0 (SNVS), VDD_SNVS_4P2 must be
discharged below 2.4 V before starting the next power-up sequence"

Best regards
igor