topic T1024RDB NAND UIB ECC error in T-Series

T1024RDB NAND UIB ECC error

rdk — Thu, 16 Apr 2020 06:33:13 GMT

Hi,

I'm working on the T1024RDB and trying to have the root filesystem on the NAND flash (MT29F8G08ABBCAH4).
I think I have a correct UBI image with an UBIFS as root filesystem. Only the root filesystem is on the NAND, u-boot, kenel and device tree are stored in the NOR flash.

UBIFS is created with command : mkfs.ubifs -r root-fs/ -m 4096 -e 253952 -c 4227 -o agc-ubifs.img
The root-fs folder contains the extracted rootfs tarball that has been generated by Yocto.

UBI image is created with command : ubinize -v -o agc-ubi.img -m 4096 -p 256KiB ubinize.cfg
The content of the configuration file ubinize.cfg is as follow :

[rootfs_volume]
mode=ubi
image=agc-ubifs.img
vol_id=0
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize

The T1024RDB boots almost correclty (I'm able to log on the system).
During the startup of the board I see ECC error that can also arise when using the system.
The error is :
ubi0 warning: .ubi_io_read: error -74 (ECC error) while reading 77 bytes from PEB 412:135936, read only 77 bytes, retry
ubi0 warning: .ubi_io_read: error -74 (ECC error) while reading 77 bytes from PEB 412:135936, read only 77 bytes, retry
ubi0 warning: .ubi_io_read: error -74 (ECC error) while reading 77 bytes from PEB 412:135936, read only 77 bytes, retry
ubi0 error: .ubi_io_read: error -74 (ECC error) while reading 77 bytes from PEB 412:135936, read 77 bytes
CPU: 0 PID: 2152 Comm: rc Not tainted 4.14.122-rt62+ge3aede3681a6 #1
Call Trace:
[c00000007c3b35a0] [c000000000a5dfcc] .dump_stack+0xac/0xf0 (unreliable)
[c00000007c3b3630] [c00000000069523c] .ubi_io_read+0x208/0x448
[c00000007c3b3710] [c000000000692250] .ubi_eba_read_leb+0xd8/0x5a0
[c00000007c3b37f0] [c000000000690b8c] .ubi_leb_read+0xcc/0x168
[c00000007c3b38a0] [c0000000003ffaf8] .ubifs_leb_read+0x44/0xbc
[c00000007c3b3940] [c000000000401d54] .ubifs_read_node+0xac/0x310
[c00000007c3b39f0] [c000000000426874] .ubifs_tnc_read_node+0x5c/0x138
[c00000007c3b3a80] [c000000000404350] .tnc_read_hashed_node+0xd8/0x2a4
[c00000007c3b3b10] [c0000000004084ac] .ubifs_tnc_next_ent+0x1f0/0x29c
[c00000007c3b3bd0] [c0000000003f7210] .ubifs_readdir+0x498/0x52c
[c00000007c3b3cc0] [c00000000022bce0] .iterate_dir+0x90/0x23c
[c00000007c3b3d70] [c00000000022c94c] .SyS_getdents64+0x8c/0x160
[c00000007c3b3e30] [c0000000000006b0] system_call+0x60/0x6c

When I do a reboot of the system, it will never boot back on the NAND, the kernel fails to mount the ubifs of the flash and thus panic on the mount of VFS.

I have done several flashing of the UBI image and I have observed that the ECC error is often raised on PEB 412.

I don't know where to look to investigate this issue, any hint would be welcome.

Thanks in advance,

P.S. Here some more information about what I have observe and the complete kernel log is attached.

When erasing the flash u-boot reports 4 bad blocks.
Skipping bad block at 0x3ff00000
Skipping bad block at 0x3ff40000
Skipping bad block at 0x3ff80000
Skipping bad block at 0x3ffc0000

Startup logs about attaching and mounting nand flash.
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: volume 0 ("agc-rootfs") re-sized from 2115 to 4012 LEBs
ubi0: attached mtd1 (name "fff800000.flash", size 1024 MiB)
ubi0: PEB size: 262144 bytes (256 KiB), LEB size: 253952 bytes
ubi0: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
ubi0: VID header offset: 4096 (aligned 4096), data offset: 8192
ubi0: good PEBs: 4092, bad PEBs: 4, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 1581963432
ubi0: available PEBs: 0, total reserved PEBs: 4092, PEBs reserved for bad PEB handling: 76
ubi0: background thread "ubi_bgt0d" started, PID 1692
rtc-ds1307 0-0068: setting system clock to 2020-04-16 05:35:18 UTC (1587015318)
ALSA device list:
No soundcards found.
Waiting 5 sec before mounting root device...
UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 1695
UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "agc-rootfs"
UBIFS (ubi0:0): LEB size: 253952 bytes (248 KiB), min./max. I/O unit sizes: 4096 bytes/4096 bytes
UBIFS (ubi0:0): FS size: 1016315904 bytes (969 MiB, 4002 LEBs), journal size 9404416 bytes (8 MiB, 38 LEBs)
UBIFS (ubi0:0): reserved for root: 0 bytes (0 KiB)
UBIFS (ubi0:0): media format: w4/r0 (latest is w5/r0), UUID 6873C4DA-A87B-44A5-80B8-5C95564B437B, small LPT mode

Re: T1024RDB NAND UIB ECC error

yipingwang — Fri, 17 Apr 2020 04:19:26 GMT

Please refer to Linux Kernel configuration.

Device Drivers --->

<*> Memory Technology Device (MTD) support --->

< > Parallel port support --->

<*> Enable UBI - Unsorted block images --->

<*> NAND Device Support --->

File systems --->

[*] Miscellaneous filesystems --->

[*] Network File Systems --->

<*> UBIFS file system support

2. Please erase the MTD device.
# flash_eraseall /dev/mtdNum
# ubiformat /dev/mtdNum -y

3. Capture the parameter of ubifs and then create the rootfs.ubi.

# ubiattach /dev/ubi_ctrl -m 4

UBI: attaching mtd4 to ubi0

UBI: scanning is finished

UBI: attached mtd4 (name "NAND Compressed RFS Image", size 500 MiB) to ubi0

UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes

UBI: min./max. I/O unit sizes: 2048/2048, sub-page size 2048

UBI: VID header offset: 2048 (aligned 2048), data offset: 4096

UBI: good PEBs: 3996, bad PEBs: 4, corrupted PEBs: 0

UBI: user volume: 0, internal volumes: 1, max. volumes count: 128

UBI: max/mean erase counter: 0/0, WL threshold: 4096, image sequence number: 590956223

UBI: available PEBs: 3916, total reserved PEBs: 80, PEBs reserved for bad PEB handling: 76

UBI: background thread "ubi_bgt0d" started, PID 1235

UBI device number 0, total 3996 LEBs (507396096 bytes, 483.9 MiB), available 3916 LEBs (497238016 bytes, 474.2 MiB), LEB size 126976 bytes (124.0 KiB)

>mkfs.ubifs -q -r rootfs/ -m 2048 -e 126976 -c 4000 -o ubifs.img

>ubinize -o ubi.img -m 2048 -p 128KiB -s 2048 ubinize.cfg

Where the ubinize.cfg, here is the e.g:

###################

[ubifs]

mode=ubi

image=ubifs.img

vol_id=0

vol_size=200MiB

vol_type=dynamic

vol_name=rootfs

####################

For the value of the parameters listed above can be got the info:
-m smallest flash I/O unit
-e logical eraseblock size(   May be "logical erase block size + 1k (or 2k) = physical eraseblock size ")
-c available LEBs
-p physical eraseblock size
-s -sub-page-size=<bytes> minimum input/output unit used for UBI headers,
                           e.g. sub-page size in case of NAND flash
                           (equivalent to the minimum input/output
                             unit size by default)

4. Detach the ubi device, then format the rootfs into the MTD block and re-attach the UBI

# ubidetach /dev/ubi_ctrl -m 4

# ubiformat /dev/mtd4 -f ./ubi.img

# ubiattach /dev/ubi_ctrl -m 4

# mount -t ubifs ubi0_0 /mnt

# ls /mnt/

=>setenv bootargs ubi.mtd=<MTD_num> root=ubi0_0 rw rootfstype=ubifs console=ttyS0,115200

Re: T1024RDB NAND UIB ECC error

rdk — Sun, 19 Apr 2020 11:48:03 GMT

Hi Yiping,

Thanks for the reply.

My kernel config is ok regarding you comment and the parameters passed to mkfs.ubifs are also ok.

My configuration for ubinize had no "vol_size" entry, is this entry mandatory ?
Also I had the entry "vol_flags=autoresize", does this entry could lead to some instabilities ?

The difference between your procedure and mine is that you are flashing the image with ubiformat while
I'm doing it with u-boot (nand write command). I don't know if ubiformat is doing more (or less) things then
directly writing the ubi image to the flash.

After running your procedure, the systems seems to work and being stable :-)

Thanks again,

Renaud