Kernel panic - not syncing: No working init found for RAMDISK

[EmbEng]

I am trying to boot a T1042 based custom board but having Kernel panic while booting the device:

Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.35-rt41 #3
Call Trace:
[e9049ec0] [c070d4e0] dump_stack+0x84/0xb0 (unreliable)
[e9049ed0] [c070b4b8] panic+0xd8/0x234
[e9049f30] [c000280c] kernel_init+0x10c/0x110
[e9049f40] [c000f198] ret_from_kernel_thread+0x5c/0x64
Rebooting in 180 seconds..

An important observation that is needed to be mentioned here is that the issue occurs 2 or 3 out of 10 times I boot the system. Means the board boots successfully most of the time but throws above error on some boot attempts.

I have tired using fsl-image-core and fsl-image-minimal  as RAMDISKs (built using SDK 2.0-1703) but both are exhibiting same behavior. Can someone comment on the issue? Thanks.

[yipingwang]

Hello Embedded Engineer,

Would you please attach your whole console log? Did you always use same command to boot the system?

Do you boot the system form NOR flash or DDR memory?

Please check whether there is problem with DDR memory, have you used mtest provided in u-boot to test DDR memory.

Thanks,

Yiping

[EmbEng]

Thanks yipingwang‌ for reply. Please find below answers to your queries:

Would you please attach your whole console log?

Kernel Logs - Pastebin.com 

Did you always use same command to boot the system?

Yes same boot command is used every time. i.e.

=> printenv bootcmd
bootcmd=setenv bootargs root=/dev/ram rw console=ttyS0,115200 ramdisk_size=1000000;bootm e8020000 e9300000 e8800000

Do you boot the system form NOR flash or DDR memory?

Kernel and RAMDISK images are stored in NOR flash which are loaded to DDR and then execution takes place from DDR.

Please check whether there is problem with DDR memory, have you used mtest provided in u-boot to test DDR memory.

I enabled mtest in u-boot and ran the command for quite a long time but it didn't report any error.

[yipingwang]

The system boot images from NOR flash, would you please try to deploy rootfs filesystem to other place on NOR flash to check whether this problem remains?

Thanks,

Yiping

[EmbEng]

I have tried boot the system numerous times using the alternate roofts location and thought the issue has been resolve until I got the following error today

RAMDISK: gzip image found at block 0
VFS: Mounted root (ext2 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 316K (c090e000 - c095d000)
EXT2-fs (ram0): error: ext2_check_page: bad entry in directory #2275: : rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT2-fs (ram0): error: ext2_check_page: bad entry in directory #2831: : rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT2-fs (ram0): error: ext2_check_page: bad entry in directory #2428: : rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.1.35-rt41 #1
Call Trace:
[ea049ec0] [c0700e30] dump_stack+0x84/0xb0 (unreliable)
[ea049ed0] [c06fee08] panic+0xd8/0x234
[ea049f30] [c000280c] kernel_init+0x10c/0x110
[ea049f40] [c000f198] ret_from_kernel_thread+0x5c/0x64
Rebooting in 180 seconds..

You can see that this error is different from the one I posted earlier.

[yipingwang]

Please format and regenerate the ext2 filesystem.

[EmbEng]

yipingwang‌ I tried the above and tested for quite a long period, and noticed same behavior (unable to mount rootfs). However, I noticed one thing that this behavior occurs only when I add my custom application on startup to root filesystem image. If the custom application is removed from startup, board boots fine all the times. Can you provide your opinion why just adding an application on startup make the whole root filesystem image unstable?