Kernel crashed by NAND flash problem

[stoneliu]

Hi,

I met a Kernel crash problem, when kernel crash, can see below information.

Please help me to check. Any information you want to get, please let me know.

CPU is i.MX6DL

Thanks,

Stone

gpmi-nand 112000.gpmi-nand: DMA timeout, last DMA :2
gpmi-nand 112000.gpmi-nand: Show GPMI registers :
gpmi-nand 112000.gpmi-nand: offset 0x000 : 0x00830001
gpmi-nand 112000.gpmi-nand: offset 0x010 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x020 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x030 : 0x00000824
gpmi-nand 112000.gpmi-nand: offset 0x040 : 0x3c000000
gpmi-nand 112000.gpmi-nand: offset 0x050 : 0x3c000800
gpmi-nand 112000.gpmi-nand: offset 0x060 : 0x01c6800c
gpmi-nand 112000.gpmi-nand: offset 0x070 : 0x00010101
gpmi-nand 112000.gpmi-nand: offset 0x080 : 0x90000000
gpmi-nand 112000.gpmi-nand: offset 0x090 : 0x03023336
gpmi-nand 112000.gpmi-nand: offset 0x0a0 : 0x00000070
gpmi-nand 112000.gpmi-nand: offset 0x0b0 : 0xff000005
gpmi-nand 112000.gpmi-nand: offset 0x0c0 : 0x00000101
gpmi-nand 112000.gpmi-nand: offset 0x0d0 : 0x05010000
gpmi-nand 112000.gpmi-nand: Show BCH registers :
gpmi-nand 112000.gpmi-nand: offset 0x000 : 0x00000100
gpmi-nand 112000.gpmi-nand: offset 0x010 : 0x00000010
gpmi-nand 112000.gpmi-nand: offset 0x020 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x030 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x040 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x050 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x060 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x070 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x080 : 0x030a1080
gpmi-nand 112000.gpmi-nand: offset 0x090 : 0x08241080
gpmi-nand 112000.gpmi-nand: offset 0x0a0 : 0x070a4080
gpmi-nand 112000.gpmi-nand: offset 0x0b0 : 0x10da4080
gpmi-nand 112000.gpmi-nand: offset 0x0c0 : 0x070a4080
gpmi-nand 112000.gpmi-nand: offset 0x0d0 : 0x10da4080
gpmi-nand 112000.gpmi-nand: offset 0x0e0 : 0x070a4080
gpmi-nand 112000.gpmi-nand: offset 0x0f0 : 0x10da4080
gpmi-nand 112000.gpmi-nand: offset 0x100 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x110 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x120 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x130 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x140 : 0x00000000
gpmi-nand 112000.gpmi-nand: offset 0x150 : 0x20484342
gpmi-nand 112000.gpmi-nand: offset 0x160 : 0x01000000
gpmi-nand 112000.gpmi-nand: offset 0x170 : 0x00000000
gpmi-nand 112000.gpmi-nand: BCH Geometry :
GF length : 13
ECC Strength : 4
Page Size in Bytes : 2084
Metadata Size in Bytes : 10
ECC Chunk0 Size in Bytes: 512
ECC Chunkn Size in Bytes: 512
ECC Chunk Count : 4
Payload Size in Bytes : 2048
Auxiliary Size in Bytes: 16
Auxiliary Status Offset: 12
Block Mark Byte Offset : 2018
Block Mark Bit Offset : 4
gpmi-nand 112000.gpmi-nand: Chip: 0, Error -110
gpmi-nand 112000.gpmi-nand: Chip: 0, Error -22
gpmi-nand 112000.gpmi-nand: Error in ECC-based write: -22
gpmi-nand 112000.gpmi-nand: Chip: 0, Error -22
gpmi-nand 112000.gpmi-nand: Chip: 0, Error -22

[ceggers]

Meanwhile there are 3 patches addressing this issue:

1. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7671edeb193910482a9b0c...
2. https://patchwork.ozlabs.org/project/linux-mtd/patch/20211012090224.4101984-1-s.riedmueller@phytec.d...
3. https://patchwork.ozlabs.org/project/linux-mtd/patch/20211012090224.4101984-2-s.riedmueller@phytec.d...

In my case (i.MX6ULL), patches 2+3 fixed this issue. The original discussion is here: https://patchwork.ozlabs.org/project/linux-mtd/patch/89ae32a0-9b19-4735-90eb-4ffa22aad704@kernel.org...

[Marc_Immel]

I'm also interested in the root cause for the problem. Does anybody have further information meanwhile?

[bastien_cornut]

Hi Stone Liu,

I am encountering the exact same issue you described. Did you manage to fix this issue ?

Regards,

Bastien

[stoneliu]

We found this is a hardware problem, some signals are not stable.

[simontremblay]

Hi Stone Liu,

Which signals were unstable?  Anything detail that could help us?  We have the same issue after a watchdog reset.

Thanks

Simon

[jp1]

Hello, 

I have exactly the same problem (same trace, verbatim). Could you share which signals were not stable?

I'm under a weird situation where the HW watchdog is rigged but the reboot is incomplete and get stuck in that DMA Timeout issue. 

A complete power cycle of the system solve the issue.

Thanks,

[igorpadykov]

Hi Stone

please check:

1. board memory:

i.MX6/7 DDR Stress Test Tool V2.70 

2. gpmi interface with baremetal sdk

Github SDK
https://github.com/backenklee/swp-report/tree/master/iMX6_Platform_SDK

3. program nand using mfg tool on link

https://www.nxp.com/support/developer-resources/run-time-software/i.mx-developer-resources/i.mx-6ser...

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[stoneliu]

Hi Igorpadykov,

I checked these three items, all are OK, but still has this problem.

Do you know any tools for NAND flash testing?

Thanks,

Stone

[igorpadykov]

Hi Stone

there are no special tools for nand, most simple SDK:
https://github.com/backenklee/swp-report/tree/master/iMX6_Platform_SDK

Also one can boot from other media (like sd) and try to read/write nand.

Best regards
igor