i.MX6: eMMC hangs at mkfs "discard"

[holgerschurig]

Hi,

I'm on Linux 3.16.3 on a i.MX6 and have an eMMC card that worked quite well. I had partitioned it and was able to boot from it.

However, the eMMC itself was unconfigured and thus in MLC mode. I then used the mmc utility from https://git.kernel.org/cgit/linux/kernel/git/cjb/mmc-utils.git/ to enable "enhanced mode" (pseudo-SLC) mode in the user area. So I looked at the output of "mmc extcsd read /dev/mmcblk0" for MAX_ENH_SIZE_MULT and used that value in "mmc enh_area set -y 0 7651323 /dev/mmcblk0".

This seemed to work. After a power cycle did the originally existing partition /dev/mmcblk0p1 vanish, and the new /dev/mmcblk0 is now half in size. So I thought I'd now have reduced wear-layout.The "fdisk" commmand worked also as expected.

But then "mkfs.ext3 -v -j /dev/mmcblk0p1" hangs at "Discarding device blocks: 4096/196608" stage. The funny thing: if I use "-E nodiscard", then partitioniong works like a charm. But isn't the discard feature something that I should use with flash-based memory?

A discussion from Can't access eMMC when using LTIB rootfs with newer (3.11.4) kernel, why? had a similar discussion (about discard handling), but the posted patches are already part in kernel 3.16.3.

[holgerschurig]

BTW; with newer kernel (I think already with 3.16) the problem went away. No need to specify "-o discard" anymore.

So it was definitely some driver problem.

[patricksears_pr]

I experienced the same problem with the mke2fs utility hanging at the point where it makes the ioctl call to perform the BLKDISCARD.  With my situation, the ioctl call never returns.  I verified this by letting it sit for 15 hours (overnight) and it never completed the BLKDISCARD operation.  For comparison, if I fill up the device with all zeros, by cat'ing "/dev/zero" into it, it only takes 24 minutes to complete.

Here are some details about by environment:

     Processor:  i.MX6 Quad Core
           DDR:  2GB
Kernel version:  3.0.35
            OS:  Android JellyBean 4.2.2

Our system is setup to load the bootloaders from NAND and then everything else from an 8GB eMMC chip (attached to SD1).  We also use a removable micro SD card (attached to SD2) as a mass storage device.  The SD card is a Kingston 16GB micro SD card with a rating of 45 MB/sec read and 10-12 MB/sec write.

So far, I haven't had any problem formatting the partitions on the eMMC chip.  And I didn't used to have any problems formatting the partitions on the SD card with an older batch of the Kingston 16GB SD cards.  The problem only started to happen with the latest batch of Kingston 16GB SD cards that we purchased.

NOTE:  The newer Kingston 16GB SD cards are slightly smaller in size (15.7 GB) than the older 16GB cards (which were 16.0 GB).

I've noticed 2 seemingly unusual factors involved in whether I experience the hang in the BLKDISCARD operation:

1)  I have 2 partitions on the device and I only experience the hang when formatting the 2nd partition.

We partition our SD card into 2 partitions.  The first partition is 2GB and the second takes up the remainder of the space (roughly 14GB).  And we format both partitions as ext4 using the mke2fs utility.  I was originally using the default options for mke2fs with the exception of using "-T" to specify an ext4 filesystem and "-L" to specify a volume label.

I find it odd that the 1st partition would not also experience the problem.  It might have made some sense if the 2nd partition was larger than it had been on the older SD cards.  Because, then, it might have been related to some sort of size limit.  But, with the newer SD cards, the 2nd partition is actually smaller.

2)  If I wipe the master boot record (first 512 bytes) on the SD card, prior to partitioning and formatting it, then I don't experience the hang when using mke2fs.

By wiping the first 512 bytes, I wind up losing the original disk configuration and so a default configuration winds up getting assigned.  The following shows the difference between the original disk configuration and the default disk configuration:

  ---------------------------
  Original disk configuration
  ---------------------------
  root@android:/ # fdisk -l /dev/block/mmcblk1

  Disk /dev/block/mmcblk1: 15.7 GB, 15720251392 bytes
  255 heads, 63 sectors/track, 1911 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes

                Device Boot      Start         End      Blocks  Id System
  /dev/block/mmcblk1p1               1         250     2008093+ 83 Linux
  /dev/block/mmcblk1p2             251        1911    13341982+ 83 Linux

  ---------------------------------------
  Disk configuration after wiping the MBR
  ---------------------------------------
  root@android:/ # fdisk -l /dev/block/mmcblk1

  Disk /dev/block/mmcblk1: 15.7 GB, 15720251392 bytes
  4 heads, 16 sectors/track, 479744 cylinders
  Units = cylinders of 64 * 512 = 32768 bytes

                Device Boot      Start         End      Blocks  Id System
  /dev/block/mmcblk1p1               1       62501     2000024  83 Linux
  /dev/block/mmcblk1p2           62502      479744    13351776  83 Linux

You can see that wiping the MBR causes the number of heads, sectors per track, and cylinder size to change.  After these changes, I no longer experience the hang while doing the BLKDISCARD.

CONCLUSION:

The recommended work-around, of passing the "-E nodiscard" option to the mke2fs command, is working for me.

I imagine it is less interesting to everyone else that I am encountering this problem with JellyBean 4.2.2 and the 3.0.35 kernel.  However, we are looking at moving to a newer BSP (newer version of Android and the kernel).  So I am wondering if there is a definitive answer on which kernel version provides the fix for this problem.

Holger mentioned that it is fixed in the 3.16 kernel but it is not clear which SUBLEVEL (minor) version of the kernel has the fix since his original problem statement reports states that he was running 3.16.3.

I imagine it is possible that Holger was actually having the problem on an older kernel version and later realized that 3.16.3 didn't have the problem.  If that is the case, then does anyone know if the 3.10.53 kernel might also have the fix?

[timjaacks]

Oh sorry, I did not realize that it was you who started the thread. So you were on 3.16.3 before and updated the kernel now? To which version?

[timjaacks]

Holger, are you sure about this? The thread starter reported that he already is on kernel 3.16. Maybe you have an eMMC device implementing the eMMC spec v4.41 or lower? The discard command was introduced with spec v4.5. See IMX6: mkfs.ext4 on emmc is delayed for reference.

[timjaacks]

I know that this question is already a few months old, but since I stumbled upon the same problem I thought that people, who find this post, might be interested in a solution.

What happens: When calling mkfs.ext3 without any options specified, the discard option is used by default. This makes the storage device erase all blocks during the formatting process. Since an eMMC uses NAND flash memory internally, physical blocks that have been in use already must be erased before they can be re-written. Erasing blocks takes time. In order to keep write performance at a high level, it is best to assure that all unused blocks are erased. Otherwise, the according blocks have to be erased during a write operation, which decreases overall write performance.

For this purpose, the discard option has been introduced to both mkfs.ext3 and the ext3 filesystem itself (as a mount option). If an ext3 filesystem is mounted with the discard option, blocks are discarded on deletion of a file.

On embedded systems, the decision of whether to use the discard option or not can be based on two general questions:

1. How much data do I write to the eMMC?

If you format your device only once and it is unlikely that you will ever use the complete space, you do not have to care about discarding blocks at all. The eMMC comes fully erased out of the factory, so the initial formatting does not need any erasing (you can use "-E nodiscard" option). As long as you do not write much data to the partition, there will always be enough free blocks available which can be used for write operations.

2. Do I need a specifically high guaranteed write performance?

If you don't care about write performance, you do not have to care about discarding blocks either. Physical blocks will be erased when they are needed. If you need to guarantee a spcific write performance, you have to think about when to erase blocks.

In general there are three options for the time of erasing blocks:

a) Mount your partition normally. Blocks will be erased as soon as a write operation needs free blocks and there are none.

b) Use the "-o discard" option when mounting your ext3 partition. Blocks will be erased on each deletion of a file.

c) Mount your partition normally and erase blocks manually by calling "trimfs". This can be done in a periodic cron job for example.

Take these questions and options into account when creating the filesystem. Depending on your application it might be possible to use the "-E nodiscard" on mkfs.ext3.

[holgerschurig]

Sorry for the confusion, because what I said was wrong.

I had the 3.16.x kernel out of my memory, but that was faulty. Maybe me DDR3 timings are bad :-)

In my git tree the time where I remove the "-E discard" was when I was using Kernel 3.17.7. That doesn't mean that the patch from 3.17.6 to 3.17.7 fixed the issue, it just means when using 3.17.7 i re-visited the problem.

[alejandrolozan1]

Hi,

Have you tried that in a previous kernel version? in case it works I would start checking the differences in the MMC driver.

Best Regards,

Alejandro