i.MX6 - System hang after "Freeing unused kernel memory"

[NathanJozwiak]

Hi all,

I am having issues with a system hang during kernel startup. Right off the bat, I suspect this is a memory configuration issue, but I am not sure where... here are the symptoms.

I have a custom board running an i.MX6 with 1GB of DDR. The design was based on the Boundary Devices Nitrogen6x (2GB) board. I am running Linux 3.10.53 from Yocto 1.8.

The board is booting off of the SD card using u-boot 2014.07 with the IVT header configuring DDR. DDR appears stable as I am able to run memory tests from a JTAG emulator. I also have no issues booting u-boot and reading/writing to memory from there.

The kernel is loaded into memory from the SD card which also contains the root files system. Things appear to boot and run fine until roughly the time init.d is kicked off. From there things either stall before the beginning of init.d or crash during init.d. A normal crash will produce a "Unable to handle kernel NULL pointer dereference at virtual address" message in the console followed by a stack trace. The behavior is inconsistent in terms of if/when things will crash.

I am confident there are no issues with the rootfs or init.d as I have tested the SD card with the same rootfs, init.d, and kernel on the Nitrogen6x board and it boots without issue. The only difference between booting the two boards is obviously the device tree binary used. I have also scrutinized the device tree for the custom board. It's pretty bare-bone, but I have validated there are no conflicts in the hardware pinmux.

There are a couple of interesting things about startup...

First, the virtual kernel memory layout does not match a system with 1GB of memory.

Memory: 1024MB = 1024MB total

Memory: 767624k/767624k available, 280952k reserved, 0K highmem

Virtual kernel memory layout:

    vector  : 0xffff0000 - 0xffff1000   (   4 kB)

    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)

    vmalloc : 0xc0800000 - 0xff000000   (1000 MB)

    lowmem  : 0x80000000 - 0xc0000000   (1024 MB)

    modules : 0x7f000000 - 0x80000000   (  16 MB)

      .text : 0x80008000 - 0x807c5054   (7925 kB)

      .init : 0x807c6000 - 0x808042c0   ( 249 kB)

      .data : 0x80806000 - 0x80874ee0   ( 444 kB)

       .bss : 0x80874ee0 - 0x808d54d4   ( 386 kB)

While it reports a correct amount of memory (1024MB), the virtual table allocates all of that to lowmem, and then 1000MB(?) to vmalloc which is out of range for only having 1GB.

Second, I found a thread with a similar hang issue (System hang after "Freeing init memory: 200K" with YOCTO imge ) but with no resolution. He was able to get to the bash shell by setting the kernel parameter to init=/bin/sh. I did not have success with that. I was (1/10 boot attempts) able to get to the bash prompt, but a simple 'ls' command from the prompt caused a crash.

I ran some additional tests using this init parameter. I built my own test application that entered an infinite loop where it would simply log a message to the console and sleep one second. This proved to produce similar results in terms of crashes and stalls. I think this proves that the issue is not in init.d itself, but am unsure of how to continue troubleshooting.

The kernel that is being ran here runs fine on the Nitrogen6x where there is 2GB of memory. This behavior seems to point to a memory issue, I'm just unsure where.

Any thoughts?

[igorpadykov · Accepted solution]

Hi Nathan

this looks like ddr issues so one can run ddr tests

i.MX6/7 DDR Stress Test Tool V2.40

rebuild uboot with  updated *.cfg file (obtained from ddr test),

memory size should be adjusted in uboot/include/configs/*board*.h file.

For Boundary Devices Nitrogen6x suggest to use its latest uboot

https://boundarydevices.com/switching-u-boot-versions-on-i-mx6/

Best regards

igor

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------