i.MX6ULL errors correcting u-boot bitflips

[da1]

Hi,
we have a reliability problem with some boards based on i.MX6ULL rev1.0 14x14 EVK, booting from a Micron MT29F32G08CBADAWP-12IT:D NAND.

The system boots correctly but - apparently - as soon as some bits in the u-boot partition flip, the system is not able to recover it.

We are using (for both the running system and to flash it with mfgtools):

• a kernel 4.1.15 from NXP, compiled with DEBUGFS
• u-boot 2016.03-imx_v2016.03_4.1.15
• the latest version of kobs-ng from https://github.com/NXPmicro/imx-kobs

All of these from the Freescale Yocto layers.

My question: what are the correct parameters for ECC Strength, m_u32EccBlockNEccType and the other parameters relevant to ECC, for a i.MX6ULL CPU?

Any help is welcome; if more information are needed, we'll try to provide them as soon as possible.

Thanks in advance.

The output of kobs-ng is as follow:

# kobs-ng init -x -v --chip_0_device_path=/dev/mtd0 u-boot.imx
MTD CONFIG:
  chip_0_device_path = "/dev/mtd0"
  chip_1_device_path = "(null)"
  search_exponent = 2
  data_setup_time = 80
  data_hold_time = 60
  address_setup_time = 25
  data_sample_time = 6
  row_address_size = 3
  column_address_size = 2
  read_command_code1 = 0
  read_command_code2 = 48
  boot_stream_major_version = 1
  boot_stream_minor_version = 0
  boot_stream_sub_version = 0
  ncb_version = 3
  boot_stream_1_address = 0
  boot_stream_2_address = 0
     -- We add the 1k-padding to the uboot.
.tmp_kobs_ng: verifying using key '00000000000000000000000000000000'
.tmp_kobs_ng: is a valid bootstream for key '00000000000000000000000000000000'
mtd: use new bch layout raw access mode
mtd: opening: "/dev/mtd0"
NFC geometry :
    ECC Strength       : 40
    Page Size in Bytes : 8762
    Metadata size      : 10
    ECC Chunk Size in byte : 1024
    ECC Chunk count        : 8
    Block Mark Byte Offset : 7692
    Block Mark Bit Offset  : 0
====================================================
mtd: opened '/dev/mtd0' - '(null)'
mtd: max_boot_stream_size_in_bytes = 25165824
mtd: boot_stream_size_in_bytes = 475136
mtd: boot_stream_size_in_pages = 58
mtd: #1 0x01000000 - 0x02800000 (0x01074000)
mtd: #2 0x02800000 - 0x04000000 (0x02874000)
FCB
  m_u32Checksum = 0x00000000
  m_u32FingerPrint = 0x20424346
  m_u32Version = 0x01000000
  m_NANDTiming.m_u8DataSetup = 80
  m_NANDTiming.m_u8DataHold = 60
  m_NANDTiming.m_u8AddressSetup = 25
  m_NANDTiming.m_u8DSAMPLE_TIME = 6
  m_u32PageDataSize = 8192
  m_u32TotalPageSize = 8936
  m_u32SectorsPerBlock = 256
  m_u32NumberOfNANDs = 0
  m_u32TotalInternalDie = 0
  m_u32CellType = 0
  m_u32EccBlockNEccType = 20
  m_u32EccBlock0Size = 1024
  m_u32EccBlockNSize = 1024
  m_u32EccBlock0EccType = 20
  m_u32MetadataBytes = 10
  m_u32NumEccBlocksPerPage = 7
  m_u32EccBlockNEccLevelSDK = 0
  m_u32EccBlock0SizeSDK = 0
  m_u32EccBlockNSizeSDK = 0
  m_u32EccBlock0EccLevelSDK = 0
  m_u32NumEccBlocksPerPageSDK = 0
  m_u32MetadataBytesSDK = 0
  m_u32EraseThreshold = 0
  m_u32Firmware1_startingPage = 2048
  m_u32Firmware2_startingPage = 5120
  m_u32PagesInFirmware1 = 58
  m_u32PagesInFirmware2 = 58
  m_u32DBBTSearchAreaStartAddress = 1024
  m_u32BadBlockMarkerByte = 7692
  m_u32BadBlockMarkerStartBit = 0
  m_u32BBMarkerPhysicalOffset = 8192
  m_u32BCHType = 1
  m_NANDTMTiming.m_u32TMTiming2_ReadLatency = 0
  m_NANDTMTiming.m_u32TMTiming2_PreambleDelay = 0
  m_NANDTMTiming.m_u32TMTiming2_CEDelay = 0
  m_NANDTMTiming.m_u32TMTiming2_PostambleDelay = 0
  m_NANDTMTiming.m_u32TMTiming2_CmdAddPause = 0
  m_NANDTMTiming.m_u32TMTiming2_DataPause = 0
  m_NANDTMTiming.m_u32TMSpeed = 0
  m_NANDTMTiming.m_u32TMTiming1_BusyTimeout = 0
  m_u32DISBBM = 0
  m_u32BBMarkerPhysicalOffsetInSpareData = 0
  m_u32OnfiSyncEnable = 0
  m_NANDONFITiming.m_u32ONFISpeed = 0
  m_NANDONFITiming.m_u32ONFITiming_ReadLatency = 0
  m_NANDONFITiming.m_u32ONFITiming_CEDelay = 0
  m_NANDONFITiming.m_u32ONFITiming_PreambleDelay = 0
  m_NANDONFITiming.m_u32ONFITiming_PostambleDelay = 0
  m_NANDONFITiming.m_u32ONFITiming_CmdAddPause = 0
  m_NANDONFITiming.m_u32ONFITiming_DataPause = 0
  m_NANDONFITiming.m_u32ONFITiming_BusyTimeout = 0
  m_u32DISBBSearch = 0
  m_u32RandomizerEnable = 0
  m_u32ReadRetryEnable = 0
  m_u32ReadRetrySeqLength = 0
DBBT
  m_u32Checksum = 0x00000000
  m_u32FingerPrint = 0x54424244
  m_u32Version = 0x01000000
  m_u32DBBTNumOfPages = 0
Firmware: image #0 @ 0x1000000 size 0x74000 - available 0x1800000
Firmware: image #1 @ 0x2800000 size 0x74000 - available 0x1800000
-------------- Start to write the [ FCB ] -----
mtd: erasing @0:0x0-0x200000
mtd: Writing FCB0 [ @0:0x0 ] (22e8) *
mtd: erasing @0:0x200000-0x400000
mtd: Writing FCB1 [ @0:0x200000 ] (22e8) *
mtd: erasing @0:0x400000-0x600000
mtd: Writing FCB2 [ @0:0x400000 ] (22e8) *
mtd: erasing @0:0x600000-0x800000
mtd: Writing FCB3 [ @0:0x600000 ] (22e8) *
mtd_commit_bcb(FCB): status 0

-------------- Start to write the [ DBBT ] -----
mtd: erasing @0:0x800000-0xa00000
mtd: Writing DBBT0 [ @0:0x800000 ] (2000) *
mtd: erasing @0:0xa00000-0xc00000
mtd: Writing DBBT1 [ @0:0xa00000 ] (2000) *
mtd: erasing @0:0xc00000-0xe00000
mtd: Writing DBBT2 [ @0:0xc00000 ] (2000) *
mtd: erasing @0:0xe00000-0x1000000
mtd: Writing DBBT3 [ @0:0xe00000 ] (2000) *
mtd_commit_bcb(DBBT): status 0

---------- Start to write the [ .tmp_kobs_ng ]----
mtd: Writting .tmp_kobs_ng: #0 @0: 0x01000000 - 0x01074000
mtd: erasing @0:0x1000000-0x1200000
mtd: We write one page for save guard. *
mtd: Writting .tmp_kobs_ng: #1 @0: 0x02800000 - 0x02874000
mtd: erasing @0:0x2800000-0x2a00000
mtd: We write one page for save guard. *

imx6ull imx-kobs kobs-ng 

[igorpadykov · Accepted solution]

Hi Davide

>what are the correct parameters for ECC Strength, m_u32EccBlockNEccType and
>the other parameters relevant to ECC, for a i.MX6ULL CPU?

kobs-ng sets these parameters automatically based on specific nand part entry in

linux/drivers/mtd/nand/nand_ids.c

nand_ids.c\nand\mtd\drivers - linux-imx - i.MX Linux kernel 

also one can look on

Handling bit flip in erased page 

[PATCH v7] mtd: gpmi: Deal with bitflips in erased regions 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[da1]

Hi igorpadykov,

thanks for the quick reply and the useful links.

We'll try to dig down the problem. What we find odd is that those parameters seems correct for our NAND and should be compatible with a i.MX 6ULL CPU.

Another thing that puzzles us is this: running nandtest -k /dev/mtd0 (were mtd0 is the partition containing u-boot) will break the system (no output at all at boot). Is it an expected behavior? The system works fine testing other partitions.

We have also tried to force a different ECC Strength (26) in the get_ecc_strength function of drivers/mtd/nand/gpmi-nand/gpmi-nand.c, but kobs-ng still uses 40 (even if the debugfs is unmounted); that was unexpected.

We'll try to apply the patch to fix bitflips in erased regions.

Another questions:

• is there a way to verify that kobs-ng has written the correct ECC codes? What do we have to check, in the output of nanddump --oob ?
• is there a way to emulate a bitflip in the NAND? (I guess we can read a random location with nanddump --oob, change it and write it back with nandwrite --oob, correct? Is it safe to do on the u-boot partition?)
• is it normal that two consecutive runs of nanddump --oob returns different results?

Thank you very much for your support!

[igorpadykov]

Hi Davide

I am afraid such tools or documentation for nand errors analysis are not available.

In general one can try extended support with

NXP Professional Services | NXP 

Best regards
igor

[da1]

Hi Igor,

thanks for the hint, we'll surely evaluate it.

As a last note, for other people that may have a similar problem: we dumped (without the OOB data) the content of the u-boot partition and we can confirm that it's identical to the data we have written (and identical to a working system), despite this, the system doesn't boot. Once the partition is flashed again with kobs-ng, it works again.

Best regards.