Error: Unable to handle kernel paging request at virtual address

[Bob_NXP_user]

Hello everybody,

I'm tryng to debug my Linux OS for a custom iMX93 board. I'm able to start the OS but sometimes I get the following error while booting:
  

the weird fact is that this happens randomly! Sometimes I don't have problems at all, other times it hangs with this error and in different "locations" of the log! I understand that this error means that it cannot initialize a driver, comparing the error log with the correct one I'm able to discover which driver is not working. I'm wondering if could be a problem linked to the RAM settings. In particular when the OS starts, I notice that the RAM memory is not correctly recognized by the kernel (2 GB while I have only 256 MB):
 

Can anyone help me to understand what's going on?

Regards, 

Bob

[NXP_USER_05]

Hi @Bob_NXP_user 

We are working with a custom i.MX8M Plus DDR4 board that includes 32 MB QSPI NOR flash.
Our objective is to boot U-Boot, Linux kernel, device tree, and initramfs (rootfs.img) entirely from QSPI flash, without using SD/eMMC.

We are able to successfully boot up to U-Boot (flash.bin) from QSPI flash.

-----Flash Layout and Image Sizes----------------

QSPI flash size: 32 MB

flash.bin (U-Boot + SPL) ~2.4 MB
Image.gz (Kernel) ~15 MB
imx8mp-evk.dtb ~85 KB
rootfs.img ~12.6 MB

----QSPI Offset Layout----------
flash.bin @ 0x00000000
Image.gz @ 0x0024D000
imx8mp-evk.dtb @ 0x010A6B00
rootfs.img @ 0x010BB800

U-Boot Environment Setup

After booting into U-Boot from QSPI, we configure the following environment variables to load the kernel, device tree, and initramfs from flash:

sf probe 0
setenv loadaddr 0x40480000 // kernel
setenv fdt_addr 0x43000000 //dtb
setenv initrd_addr 0x43800000 //rootfs (minimal initramfs)
setenv kernel_comp_addr_r 0x50000000
setenv kernel_comp_size 0x04000000

Reading Images from QSPI Flash

sf read ${loadaddr} 0x0024D000 0x00E59A6F
sf read ${fdt_addr} 0x010A6B00 0x00014BBA
sf read ${initrd_addr} 0x010BB800 0x00C04DB0

Boot Command

setenv bootargs console=${console},${baudrate} earlycon=${earlycon},${baudrate} rdinit=/dev

booti ${loadaddr} ${initrd_addr} ${fdt_addr}

Issue Observed

• The kernel starts executing but hangs at “Starting kernel …”.

• Please refer to the attached kernel boot log (kernel_bootlog1).

When using the following boot arguments:

setenv bootargs setenv bootargs "console=ttymxc1,115200 earlycon root=/dev/ram0 rw rdinit=/sbin/init loglevel=8"

• The kernel progresses further but hangs midway during boot.

• Please refer to the attached kernel boot log (kernel_bootlog2).

Attached the kernel DTS and kernel defconfig for reference.
Kindly let us know if any modifications are required.

Also getting same error like you Unable to handle kernel paging request at virtual address. can you please give any suggistions to resolve it and successful kernel boot. 

Questions:

1. Are there any recommended changes required in the kernel DTS for booting initramfs entirely from QSPI on a custom i.MX8M Plus DDR4 board?

2. Are there any additional boot arguments recommended by NXP for initramfs-based boot on i.MX8MP from QSPI flash?

3. What could be the possible reason for the kernel not booting correctly?
We followed the same procedure that works successfully on the i.MX8MP LPDDR4  EVK, where the kernel and root filesystem boot properly from QSPI flash.
The only differences in our setup are the flash.bin and device tree, which were modified according to the DDR configuration of our custom board.

thank you.

[Bio_TICFSL]

Hello,

The "Unable to handle kernel paging request at virtual address" error, often accompanied by a kernel panic, typically indicates a problem with memory access or mapping during kernel operations. This error can be caused by issues with device drivers, incorrect RAM settings, or even kernel bugs. 

Possible Causes and Solutions:

• Incorrect RAM Settings: The kernel might not be correctly recognizing or mapping the physical RAM.  Device Driver Issues: A faulty or incompatible device driver could be trying to access memory incorrectly, triggering the error. 

• Kernel Bugs: In some cases, the error might be due to a bug in the kernel itself, especially if the error occurs randomly or consistently at different virtual addresses. 
• DMA Problems: DMA (Direct Memory Access) issues can also lead to paging errors, particularly if the DMA engine attempts to access a memory region that's not properly mapped. 
• Memory Corruption: If there's memory corruption within the kernel, it can cause issues with paging. 

Troubleshooting Steps:

1. 1. Identify the Specific Virtual Address:

   The error message will include a virtual address where the problem is occurring. This address can help pinpoint the potentially problematic driver or memory region. 

• 2. Check Driver Compatibility:

  If you recently installed or updated a driver, try rolling back to a previous version to see if that resolves the issue. 
• 3. Verify RAM Settings:

  Ensure the RAM is correctly recognized by the kernel, and try different RAM settings if possible. 
• 4. Enable Kernel Debugging:

  If possible, enable kernel debugging options to gather more information about the error, particularly related to MMU and VMA structures. 
• 5. Check for Kernel Bugs:

  If the error is reproducible, it could be a kernel bug. Consult the kernel's documentation or relevant forums for known issues. 
• 6. Examine DMA Usage:

  If your system uses DMA, investigate how memory is allocated and mapped for DMA operations. Incorrect memory allocation can lead to paging errors. 
• 7. Consider Stack Overflow:

  If the error occurs during a specific kernel function, check for potential stack overflow issues. 
• 8. Monitor Memory Usage:

  If the system runs out of memory, it can also cause paging errors. Ensure you have sufficient memory available. 
  Regards

[Bob_NXP_user]

Hello,

Regarding the troubleshooting steps:

1. the virtual address changes! Everytime I reboot the board, and the problem occurs, the virtual address is never the same (when it doesn't work, because sometimes the kernel works!).

2. I didn't update any driver, I used the kernel version provided by NXP 6.12.y. I'm currently disabling all the drivers that I don't need in the menuconfig.

3. I set a bootarg ( mem=256M) in U-boot environment to let the kernel recognize the RAM size, otherwise it set 2GB by default. I don't know if it is the right way, please tell me.

5. The error changes.

6. Please, can you tell me how it has to be?

7,8 need to be examined.

Thanks in advance.

Regards,

Bob

[Bio_TICFSL]

Hello,

A1. Erase yocto and restart building again.

A2, try to check the CMEM variable in uboot.

Regards

[Bob_NXP_user]

Hello  @Bio_TICFSL ,

Already done various builds, I still get the same random problem. Furthermore I didn't find the cmem variable.

Do you have any other suggestion regarding the possible cause of this weird behavior?

Regards,

Bob