After writing ubifs and booting into Linux bad nodes are found

JohnKlug · ‎04-05-2021

I used Yocto to build my image.

I use these for options:

MKUBIFS_ARGS = "-m 4096 -e 253952 -c 1932"
UBINIZE_ARGS = "-p 256KiB -m 4096"

On booting Linux immediately after flashing my NAND, I see:

[  OK  ] Started Flush Journal to Persistent Storage.
[    9.208493] UBIFS error (ubi0:0 pid 149): ubifs_check_node: bad node length 32840
[    9.225422] UBIFS error (ubi0:0 pid 149): ubifs_check_node: bad node at LEB 302:208896
[    9.233369]  magic          0x6101831
[    9.237193]  crc            0x4064dc01
[    9.240959]  node_type      2 (direntry node)
[    9.245391]  group_type     1 (in node group)
[    9.249759]  sqnum          59940
[    9.253080]  len            32840
[    9.256447]  key            (5258, direntry, 0xbd23f8)
[    9.261595]  inum           6490
[    9.264831]  type           0
[    9.267839]  nlen           15
[    9.270900]  name           .#gshadowYhgOMs
[    9.335212] CPU: 0 PID: 149 Comm: systemd-sysuser Not tainted 5.4.24-2.1.0+gbabac008e5cf #1
[    9.343593] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[    9.349827] [<80111854>] (unwind_backtrace) from [<8010c3e8>] (show_stack+0x10/0x14)

If I then boot into U-Boot there are errors doing the mount, but if I re-flash, U-Boot can mount the UBIFS just fine and I can list directories.

It is also possible to boot my device from SD card. Here is what I see when booting from SD card:

root@imx6ull14x14evk:~# mtdinfo -u /dev/mtd0
mtd0
Name:                           gpmi-nand
Type:                           nand
Eraseblock size:                262144 bytes, 256.0 KiB
Amount of eraseblocks:          2048 (536870912 bytes, 512.0 MiB)
Minimum input/output unit size: 4096 bytes
Sub-page size:                  4096 bytes
OOB size:                       256 bytes
Character device major/minor:   90:0
Bad blocks are allowed:         true
Device is writable:             true
Default UBI VID header offset:  4096
Default UBI data offset:        8192
Default UBI LEB size:           253952 bytes, 248.0 KiB
Maximum UBI volumes count:      128

JohnKlug · ‎04-16-2021

This problem was due to incorrect flashing from U-Boot.

The Yocto build creates *.ubi and *.ubifs. The *.ubi file contains the volume information and the file systems. I was using nand write to write this image. U-Boot could mount the file system, and everything seems just fine in U-Boot, but you cannot boot a 2nd time and get the root file system to mount if you use the *.ubi file and write it with nand write.

I then found that if I use an SD card and boot to Linux, you could write the *.ubi file using ubiformat. Then remove the SD card, and boot over and over again using the NAND flash.

There does not seem to be a similar command to ubiformat in U-Boot. So I looked some more, and uboot has a function called ubi write, and it turns out that this command can write a UBI file system. So to write the UBI file system images using U-Boot I used the following commands (nandrootfs is the mtd partition name for my ubi partition):

FB: download -f core-image-minimal-test-imx6ull14x14evk.ubifs
FB[-t 25000]: ucmd ubi part nandrootfs
FB[-t 10000]: ucmd ubi create rootfs
FB[-t 10000]: ucmd ubi info l
FB[-t 10000]: ucmd ubi info
FB[-t 25000]: ucmd ubi write ${fastboot_buffer} rootfs ${fastboot_bytes}
FB[-t 10000]: ucmd ubifsmount ubi0:rootfs
FB: acmd reset
FB: Done

View solution in original post

JohnKlug · ‎04-16-2021

This problem was due to incorrect flashing from U-Boot.

The Yocto build creates *.ubi and *.ubifs. The *.ubi file contains the volume information and the file systems. I was using nand write to write this image. U-Boot could mount the file system, and everything seems just fine in U-Boot, but you cannot boot a 2nd time and get the root file system to mount if you use the *.ubi file and write it with nand write.

I then found that if I use an SD card and boot to Linux, you could write the *.ubi file using ubiformat. Then remove the SD card, and boot over and over again using the NAND flash.

There does not seem to be a similar command to ubiformat in U-Boot. So I looked some more, and uboot has a function called ubi write, and it turns out that this command can write a UBI file system. So to write the UBI file system images using U-Boot I used the following commands (nandrootfs is the mtd partition name for my ubi partition):

FB: download -f core-image-minimal-test-imx6ull14x14evk.ubifs
FB[-t 25000]: ucmd ubi part nandrootfs
FB[-t 10000]: ucmd ubi create rootfs
FB[-t 10000]: ucmd ubi info l
FB[-t 10000]: ucmd ubi info
FB[-t 25000]: ucmd ubi write ${fastboot_buffer} rootfs ${fastboot_bytes}
FB[-t 10000]: ucmd ubifsmount ubi0:rootfs
FB: acmd reset
FB: Done

JohnKlug · ‎04-15-2021

It appears the problem is the ubi image. The following creates a correct image on the device (booted from SD card):

flash_erase /dev/mtd6 0 0
ubiformat  /dev/mtd6
ubiattach /dev/ubi0 -m6
ubimkvol /dev/ubi0 -N root -s 150MiB
mkdir /run/root
mount -t ubifs ubi0:root /run/root/
cd /run/root
tar -xf /tmp/core-image-minimal-test-imx6ull14x14evk-20210415160553.rootfs.tar.gz

But if I use nand_write to write the image to /dev/mtd6, mount the root file system, and then try to use it, lots of ECC errors occur.

JohnKlug · ‎04-15-2021

In mtd-utils-tests there is a program called runubitests.sh. This test passes, but requires deleting the volume created by U-Boot.

# ./runubitests.sh /dev/ubi0
Running mkvol_basic /dev/ubi0
Running mkvol_bad /dev/ubi0
Running mkvol_paral /dev/ubi0
Running rsvol /dev/ubi0
Running io_basic /dev/ubi0
Running io_read /dev/ubi0
Running io_update /dev/ubi0
Running io_paral /dev/ubi0
Running volrefcnt /dev/ubi0
SUCCESS

mkvol_bad test generated many log messages.

[ 1214.188146] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.215176] Volume creation request dump:
[ 1214.219221]  vol_id    -2
[ 1214.221849]  alignment 1
[ 1214.262392]  bytes     489111552
[ 1214.265651]  vol_type  3
[ 1214.268192]  name_len  22
[ 1214.270823]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.305239] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.311536] Volume creation request dump:
[ 1214.344232]  vol_id    128
[ 1214.346969]  alignment 1
[ 1214.349511]  bytes     489111552
[ 1214.376857]  vol_type  3
[ 1214.379422]  name_len  22
[ 1214.382055]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.416760] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.441618] Volume creation request dump:
[ 1214.463337]  vol_id    0
[ 1214.465902]  alignment 0
[ 1214.468446]  bytes     489111552
[ 1214.471681]  vol_type  3
[ 1214.502296]  name_len  22
[ 1214.504949]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.510434] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.551208] Volume creation request dump:
[ 1214.562990]  vol_id    0
[ 1214.565554]  alignment -1
[ 1214.568184]  bytes     489111552
[ 1214.571420]  vol_type  3
[ 1214.594912]  name_len  22
[ 1214.597565]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.617108] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.634719] Volume creation request dump:
[ 1214.638765]  vol_id    0
[ 1214.641307]  alignment 253953
[ 1214.661658]  bytes     489111552
[ 1214.672298]  vol_type  3
[ 1214.674863]  name_len  22
[ 1214.677494]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.701828] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.720791] Volume creation request dump:
[ 1214.731061]  vol_id    0
[ 1214.760617]  alignment 4097
[ 1214.768935]  bytes     489111552
[ 1214.779536]  vol_type  3
[ 1214.787403]  name_len  22
[ 1214.790055]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.811225] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.827163] Volume creation request dump:
[ 1214.831209]  vol_id    0
[ 1214.847970]  alignment 1
[ 1214.850535]  bytes     -1
[ 1214.865295]  vol_type  3
[ 1214.867860]  name_len  22
[ 1214.870491]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.893850] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1214.900146] Volume creation request dump:
[ 1214.922081]  vol_id    0
[ 1214.929835]  alignment 1
[ 1214.938454]  bytes     0
[ 1214.941018]  vol_type  3
[ 1214.954109]  name_len  22
[ 1214.956762]  1st 16 characters of name: mkvol_bad:test_m
[ 1214.976558] ubi0 error: ubi_create_volume: not enough PEBs, only 1926 available
[ 1214.997017] ubi0 error: ubi_create_volume: cannot create volume 0, error -28
[ 1215.015050] ubi0 error: ubi_create_volume: not enough PEBs, only 1926 available
[ 1215.036720] ubi0 error: ubi_create_volume: cannot create volume 0, error -28
[ 1215.055972] ubi0 error: ubi_cdev_ioctl: bad volume creation request
[ 1215.073167] Volume creation request dump:
[ 1215.077213]  vol_id    0
[ 1215.079754]  alignment 1
[ 1215.098829]  bytes     253952
[ 1215.101827]  vol_type  7
[ 1215.115895]  name_len  22
[ 1215.118548]  1st 16 characters of name: mkvol_bad:test_m
[ 1215.169287] ubi0 error: ubi_create_volume: volume 0 already exists
[ 1215.187754] ubi0 error: ubi_create_volume: cannot create volume 0, error -17
[ 1215.205757] ubi0 error: ubi_create_volume: volume "mkvol_bad:test_mkvol()" exists (ID 0)
[ 1215.219761] ubi0 error: ubi_create_volume: cannot create volume 1, error -17
[ 1215.287605] ubi0 error: ubi_create_volume: volume "mkvol_bad:test_mkvol()" exists (ID 0)
[ 1215.301938] ubi0 error: ubi_create_volume: cannot create volume 1, error -17
[ 1220.457401] ubi0 error: ubi_open_volume: cannot open device 0, volume 128, error -22
[ 1220.484091] ubi0 error: ubi_open_volume: cannot open device 0, volume -1, error -22
[ 1220.491845] ubi0 error: ubi_open_volume: cannot open device 0, volume 128, error -22
[ 1220.549379] ubi0 error: ubi_open_volume: cannot open device 0, volume 0, error -19
[ 1220.622112] ubi0 error: ubi_open_volume: cannot open device 0, volume 0, error -19

JohnKlug · ‎04-06-2021

So I got the device to boot from SD card, and uuu created the NAND file system with U-Boot.

As long as I just read from the NAND, I am OK. I can then write, and things are still OK (using the cp command and copying from the SD card to the NAND). But if I try to read back data I have written, I get errors:

[  848.721208] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 82 bytes from PEB 316:220808, read only 82 bytes, retry
[  848.734112] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 82 bytes from PEB 316:220808, read only 82 bytes, retry
[  848.752438] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 82 bytes from PEB 316:220808, read only 82 bytes, retry
[  848.766859] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 82 bytes from PEB 316:220808, read 82 bytes
[  848.779137] CPU: 0 PID: 352 Comm: find Not tainted 5.4.24-2.1.0+gbabac008e5cf #1
[  848.786560] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[  848.792790] [<80111854>] (unwind_backtrace) from [<8010c3e8>] (show_stack+0x10/0x14)
[  848.800563] [<8010c3e8>] (show_stack) from [<80c3418c>] (dump_stack+0x90/0xa4)
[  848.807819] [<80c3418c>] (dump_stack) from [<806fdc2c>] (ubi_io_read+0x164/0x3c4)
[  848.815334] [<806fdc2c>] (ubi_io_read) from [<806fb44c>] (ubi_eba_read_leb+0xd4/0x4fc)
[  848.823280] [<806fb44c>] (ubi_eba_read_leb) from [<806f9ac8>] (ubi_leb_read+0x94/0x10c)
[  848.831315] [<806f9ac8>] (ubi_leb_read) from [<803a444c>] (ubifs_leb_read+0x2c/0x78)
[  848.839089] [<803a444c>] (ubifs_leb_read) from [<803a603c>] (ubifs_read_node+0xac/0x2a8)
[  848.847210] [<803a603c>] (ubifs_read_node) from [<803c5c74>] (ubifs_tnc_read_node+0x5c/0x214)
[  848.855760] [<803c5c74>] (ubifs_tnc_read_node) from [<803a7bc0>] (tnc_read_hashed_node+0x98/0x1b8)
[  848.864743] [<803a7bc0>] (tnc_read_hashed_node) from [<803ab9a8>] (ubifs_tnc_next_ent+0x1b0/0x27c)
[  848.873725] [<803ab9a8>] (ubifs_tnc_next_ent) from [<8039c028>] (ubifs_readdir+0x178/0x518)
[  848.882105] [<8039c028>] (ubifs_readdir) from [<80257d24>] (iterate_dir+0x74/0x15c)
[  848.889791] [<80257d24>] (iterate_dir) from [<80258500>] (ksys_getdents64+0x8c/0x160)
[  848.897646] [<80258500>] (ksys_getdents64) from [<80101000>] (ret_fast_syscall+0x0/0x54)
[  848.905748] Exception stack(0x937a7fa8 to 0x937a7ff0)
[  848.910822] 7fa0:                   00000020 024725e0 00000007 02472600 00008000 00000000
[  848.919020] 7fc0: 00000020 024725e0 024725e4 000000d9 00000000 02472600 00000000 00454601
[  848.927210] 7fe0: 000000d9 7edff74c 76e9d62b 76e451d6
[  848.954326] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:237328, read only 67 bytes, retry
[  848.972781] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:237328, read only 67 bytes, retry
[  848.988355] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:237328, read only 67 bytes, retry
[  849.002934] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:237328, read 67 bytes
[  849.015340] CPU: 0 PID: 352 Comm: find Not tainted 5.4.24-2.1.0+gbabac008e5cf #1
[  849.022761] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[  849.028993] [<80111854>] (unwind_backtrace) from [<8010c3e8>] (show_stack+0x10/0x14)
[  849.036769] [<8010c3e8>] (show_stack) from [<80c3418c>] (dump_stack+0x90/0xa4)
[  849.044024] [<80c3418c>] (dump_stack) from [<806fdc2c>] (ubi_io_read+0x164/0x3c4)
[  849.051537] [<806fdc2c>] (ubi_io_read) from [<806fb44c>] (ubi_eba_read_leb+0xd4/0x4fc)
[  849.059482] [<806fb44c>] (ubi_eba_read_leb) from [<806f9ac8>] (ubi_leb_read+0x94/0x10c)
[  849.067518] [<806f9ac8>] (ubi_leb_read) from [<803a444c>] (ubifs_leb_read+0x2c/0x78)
[  849.075294] [<803a444c>] (ubifs_leb_read) from [<803a603c>] (ubifs_read_node+0xac/0x2a8)
[  849.083413] [<803a603c>] (ubifs_read_node) from [<803c5c74>] (ubifs_tnc_read_node+0x5c/0x214)
[  849.091967] [<803c5c74>] (ubifs_tnc_read_node) from [<803a7bc0>] (tnc_read_hashed_node+0x98/0x1b8)
[  849.100950] [<803a7bc0>] (tnc_read_hashed_node) from [<803ab9a8>] (ubifs_tnc_next_ent+0x1b0/0x27c)
[  849.109933] [<803ab9a8>] (ubifs_tnc_next_ent) from [<8039c028>] (ubifs_readdir+0x178/0x518)
[  849.118311] [<8039c028>] (ubifs_readdir) from [<80257d24>] (iterate_dir+0x74/0x15c)
[  849.125994] [<80257d24>] (iterate_dir) from [<80258500>] (ksys_getdents64+0x8c/0x160)
[  849.133850] [<80258500>] (ksys_getdents64) from [<80101000>] (ret_fast_syscall+0x0/0x54)
[  849.141953] Exception stack(0x937a7fa8 to 0x937a7ff0)
[  849.147027] 7fa0:                   00000020 024725e0 00000007 02472600 00008000 00000000
[  849.155225] 7fc0: 00000020 024725e0 024725e4 000000d9 00000000 02472600 00000000 00454601
[  849.163415] 7fe0: 000000d9 7edff74c 76e9d62b 76e451d6
[  849.177321] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:233144, read only 67 bytes, retry
[  849.203861] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:233144, read only 67 bytes, retry
[  849.218943] ubi0 warning: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:233144, read only 67 bytes, retry
[  849.233214] ubi0 error: ubi_io_read: error -74 (ECC error) while reading 67 bytes from PEB 316:233144, read 67 bytes

 ...