2143954_en-US

[#1]

i.MX8MM Kernel crashes on cpuidle

Hi,

We have a custom i.MX8MM board that crashes after running for some time. This issue occurs with both Linux kernel version 6.1.55 (Yocto 4.2) and Linux kernel version 5.15.71 (Yocto 4.0). The system's default CPU governor is ondemand. We've tested changing the CPU governor to performance, and the system no longer crashes.

Do you have any suggestions on how to determine if this is a hardware issue or a kernel issue? And how to solve this issue?

Here's the relevant log:

[32832.280148] audit: type=1327 audit(1751365801.520:22): proctitle=2F7573722F7362696E2F63726F6E64002D6E
[36432.304306] audit: type=1006 audit(1751369401.584:23): pid=3094 uid=0 subj=kernel old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=15 res=1
[36432.317968] audit: type=1300 audit(1751369401.584:23): arch=c00000b7 syscall=64 success=yes exit=1 a0=3 a1=ffffd19f6b10 a2=1 a3=ffffa11e9020 items=0 ppid=265 pid=3094 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=15 comm="crond" exe="/usr/sbin/crond" subj=kernel key=(null)
[36432.344959] audit: type=1327 audit(1751369401.584:23): proctitle=2F7573722F7362696E2F63726F6E64002D6E
[37877.552323] rcu: INFO: rcu_preempt self-detected stall on CPU
[37877.558090] rcu:     2-...!: (1 ticks this GP) idle=d95/0/0x3 softirq=199976/199976 fqs=0
[37877.566100]  (t=5958 jiffies g=1820793 q=55)
[37877.570369] rcu: rcu_preempt kthread timer wakeup didn't happen for 5957 jiffies! g1820793 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[37877.581933] rcu:     Possible timer handling issue on cpu=0 timer-softirq=157944
[37877.589067] rcu: rcu_preempt kthread starved for 5958 jiffies! g1820793 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[37877.599676] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[37877.608805] rcu: RCU grace-period kthread stack dump:
[37877.613854] task:rcu_preempt     state:I stack:    0 pid:   13 ppid:     2 flags:0x00000008
[37877.622211] Call trace:
[37877.624655]  __switch_to+0x104/0x15c
[37877.628241]  __schedule+0x2b8/0x710
[37877.631733]  schedule+0x88/0x100
[37877.634962]  schedule_timeout+0x80/0xf0
[37877.638801]  rcu_gp_fqs_loop+0x118/0x2e0
[37877.642726]  rcu_gp_kthread+0x104/0x11c
[37877.646564]  kthread+0x150/0x160
[37877.649795]  ret_from_fork+0x10/0x20
[37877.653373] rcu: Stack dump where RCU GP kthread last ran:
[37877.658855] Task dump for CPU 0:
[37877.662081] task:swapper/0       state:R  running task     stack:    0 pid:    0 ppid:     0 flags:0x00000008
[37877.671999] Call trace:
[37877.674443]  __switch_to+0x104/0x15c
[37877.678021]  cpuidle_enter_state+0x25c/0x2f0
[37877.682296]  cpuidle_enter+0x38/0x50
[37877.685873]  do_idle+0x210/0x2a0
[37877.689103]  cpu_startup_entry+0x24/0x80
[37877.693027]  rest_init+0xe4/0xf4
[37877.696256]  arch_call_rest_init+0x10/0x1c
[37877.700356]  start_kernel+0x610/0x650
[37877.704020]  __primary_switched+0xbc/0xc4
[37877.708037] Task dump for CPU 0:
[37877.711264] task:swapper/0       state:R  running task     stack:    0 pid:    0 ppid:     0 flags:0x00000008
[37877.721181] Call trace:
[37877.723625]  __switch_to+0x104/0x15c
[37877.727203]  cpuidle_enter_state+0x25c/0x2f0
[37877.731475]  cpuidle_enter+0x38/0x50
[37877.735052]  do_idle+0x210/0x2a0
[37877.738281]  cpu_startup_entry+0x24/0x80
[37877.742205]  rest_init+0xe4/0xf4
[37877.745433]  arch_call_rest_init+0x10/0x1c
[37877.749531]  start_kernel+0x610/0x650
[37877.753194]  __primary_switched+0xbc/0xc4
[37877.757206] Task dump for CPU 2:
[37877.760431] task:swapper/2       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00000008
[37877.770349] Call trace:
[37877.772792]  dump_backtrace+0x0/0x19c
[37877.776457]  show_stack+0x18/0x70
[37877.779775]  sched_show_task+0x154/0x180
[37877.783701]  dump_cpu_task+0x44/0x58
[37877.787280]  rcu_dump_cpu_stacks+0xe8/0x12c

i.MX 8M | i.MX 8M Mini | i.MX 8M NanoLinuxYocto ProjectRe: i.MX8MM Kernel crashes on cpuidle

Hi joanxie,

After our tests, we found that the PMIC was causing this issue.

Thank you for your support.

Re: i.MX8MM Kernel crashes on cpuidle

if just one board has this issue, as I mentioned before, you can do the AB test to confirm if this issue is related to the chip, since other boards are fine

Re: i.MX8MM Kernel crashes on cpuidle

Hi joanxie,

Thanks for your information.

But we don't plan to change the kernel configuration just for one faulty device. We're going to have the distributor help us analyze the problem.

Re: i.MX8MM Kernel crashes on cpuidle

For optimized performance on the i.MX, you can try to set the stpes as below: CONFIG_ARM_PSCI_CPUIDLE=n in the defconfig files, to avoid cpu enter idle state,  and CONFIG_ARM_PSCI_CPUIDLE is depends on the CONFIG_CPU_FREQ, if couldn't set CONFIG_CPU_FREQ=n, also suggest set CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y to the CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y, I think this should be the same as you use "echo performance" command in the user space

Re: i.MX8MM Kernel crashes on cpuidle

Hi joanxie,

All of your boards get the same result when you use the command "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies"?

=> Yes.

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1200000 1600000

What is the frequency when you use ondemand and performance?

=> The frequency is the same for both ondemand and performance mode.

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
1600000

Re: i.MX8MM Kernel crashes on cpuidle

if all of your boards have the same SW and HW, only one board has issue, you can do the AB test, to check if the chip issue, all of your boards get the same result when you use the command "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies"? and what is the frequency when you use ondemand and performance? I mean when you set ondemand or performance, use the command to check "cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq", are the ondemad and performance share the same cpu frequency? 

Re: i.MX8MM Kernel crashes on cpuidle

Hi joanxie,

Thanks for your information.

Our kernel has only been modified for peripheral hardware; we haven't made any changes to the CPU frequency or other CPU-related settings. Many of our devices can boot and run normally. Currently, only one device will crash after running for a day.

Here is the scaling_available_frequencies result:

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1200000 1600000

Re: i.MX8MM Kernel crashes on cpuidle

what do you change in the kernel? the imx8mm default frequency is 1.2Ghz, so you couldn't boot up the board as default, right? what do you get "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies"? the uboot set the init clock under

/arch/arm/mach-imx/imx8m/clock_imx8mm.c

for the kernel, set the available cpu frequency in the dtsi file and the cpufreq driver is

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/drivers/cpufreq/imx-cpufreq-dt.c

if your bsp is the similar as nxp, it seems the system crash under 1.2G, you can refer to the chapter 2.5.3 CPU Frequency Scaling (CPUFREQ) of enclosed file

Re: i.MX8MM Kernel crashes on cpuidle

Hi joanxie,

System crashed while CPU governor is ondemand mode.

Here is the scaling_available_governors result:

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance schedutil

Re: i.MX8MM Kernel crashes on cpuidle

 The system's default CPU governor is ondemand. We've tested changing the CPU governor to performance, and the system no longer crashes.

> is it your typo？when do you get the system crash? with ondemand or with performance?

do you mind giving me the result with command "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors" ?

Re: i.MX8MM Kernel crashes on cpuidle

Hi @adv-johnchang 

We are working with a custom i.MX8M Plus DDR4 board that includes 32 MB QSPI NOR flash.
Our objective is to boot U-Boot, Linux kernel, device tree, and initramfs (rootfs.img) entirely from QSPI flash, without using SD/eMMC.

We are able to successfully boot up to U-Boot (flash.bin) from QSPI flash.

-----Flash Layout and Image Sizes----------------

QSPI flash size: 32 MB

flash.bin (U-Boot + SPL) ~2.4 MB
Image.gz (Kernel) ~15 MB
imx8mp-evk.dtb ~85 KB
rootfs.img ~12.6 MB

----QSPI Offset Layout----------
flash.bin @ 0x00000000
Image.gz @ 0x0024D000
imx8mp-evk.dtb @ 0x010A6B00
rootfs.img @ 0x010BB800

U-Boot Environment Setup

After booting into U-Boot from QSPI, we configure the following environment variables to load the kernel, device tree, and initramfs from flash:

sf probe 0
setenv loadaddr 0x40480000 // kernel
setenv fdt_addr 0x43000000 //dtb
setenv initrd_addr 0x43800000 //rootfs (minimal initramfs)
setenv kernel_comp_addr_r 0x50000000
setenv kernel_comp_size 0x04000000

Reading Images from QSPI Flash

sf read ${loadaddr} 0x0024D000 0x00E59A6F
sf read ${fdt_addr} 0x010A6B00 0x00014BBA
sf read ${initrd_addr} 0x010BB800 0x00C04DB0

Boot Command

setenv bootargs console=${console},${baudrate} earlycon=${earlycon},${baudrate} rdinit=/dev

booti ${loadaddr} ${initrd_addr} ${fdt_addr}

Issue Observed

• The kernel starts executing but hangs at “Starting kernel …”.

• Please refer to the attached kernel boot log (kernel_bootlog1).

When using the following boot arguments:

setenv bootargs setenv bootargs "console=ttymxc1,115200 earlycon root=/dev/ram0 rw rdinit=/sbin/init loglevel=8"

• The kernel progresses further but hangs midway during boot.

• Please refer to the attached kernel boot log (kernel_bootlog2).

Attached the kernel DTS and kernel defconfig for reference.
Kindly let us know if any modifications are required.

Also getting same error like you Unable to handle kernel paging request at virtual address. can you please give any suggistions to resolve it and successful kernel boot. 

Questions:

1. Are there any recommended changes required in the kernel DTS for booting initramfs entirely from QSPI on a custom i.MX8M Plus DDR4 board?

2. Are there any additional boot arguments recommended by NXP for initramfs-based boot on i.MX8MP from QSPI flash?

3. What could be the possible reason for the kernel not booting correctly?
We followed the same procedure that works successfully on the i.MX8MP LPDDR4  EVK, where the kernel and root filesystem boot properly from QSPI flash.
The only differences in our setup are the flash.bin and device tree, which were modified according to the DDR configuration of our custom board.

thank you.