i.MX8MM GPU crashed on yocto 5.0

[adv-johnchang]

Hi,

Recently, we upgraded from Yocto 4.2 to Yocto 5.0 on our custom i.MX8MM board. We are now experiencing random GPU crashes during boot-up, which prevents the Weston desktop system from starting. This issue did not occur with Yocto 4.2.

Do you have any suggestions on what might be causing this problem?

Here's the relevant log:

dmesg_gpu_crash.log

[joanxie]

could you tell me what operation cause this issue? and what detailed yocto bsp version do you use? nxp yocto bsp?

[adv-johnchang]

Hi joanxie,

• could you tell me what operation cause this issue? 

=> It happens right after boot, without any user interaction.The sequence is:

power on => kernel logo display => psplash logo display => screen stays black =>

we found that Weston failed to start => checking dmesg reveals a GPU crash.

• what detailed yocto bsp version do you use? nxp yocto bsp?

=> Our yocto 5.0 BSP base on imx-6.6.52-2.2.1.xml.

By the way, we also noticed that when the GPU crashes, the dmesg log simultaneously shows 'failed to allocate memory for node 'linux,cma',' but we are unsure if these two issues are related.

[joanxie]

did you use your own board with your own customized BSP? I tested on the nxp imx8mm evk board with official bsp without any issue，if you have cma issue, you need check your dts settings for cma is correct or not

[ OK ] Started Configuration for i.MX GPU (Former rc_gpu.S).

[ OK ] Reached target Login Prompts.
Starting Hostname Service...
Starting Weston, a Wayland compositor, as a system service...
Starting WPA supplicant...

[joanxie]

did you use nxp offical default image and bsp or do you use your customized image？I couldn't reproduce this issue on my imx8mm evk board, did you test nxp official demo image with nxp imx8mlm evk board? or your own customized bsp with your own board?

[adv-johnchang]

Hi joanxie,

We are using our customized board and BSP.

Since Yocto 4.2 works and the problem appears only after upgrading to Yocto 5.0. 

So, what could be the possible causes?

[adv-johnchang · Accepted solution]

In the end, we found that the GPU crash was caused by CMA allocation failure. The failure itself was due to the random placement of kernel code/data, since the original CMA alloc-range size was only 1 GB, while the CMA required 640 MB of contiguous memory. Unfortunately, if the kernel code/data happened to fall in the middle of that 1 GB range, CMA could not allocate the required 640 MB block.

Both Yocto 4.2 and Yocto 5.0 have KASLR enabled, but I still cannot figure out why this issue does not occur with Yocto 4.2. For Yocto 4.0, the issue does not occur because KASLR is disabled due to the lack of a seed.

As a workaround, the CMA alloc-range size is increased to 2 GB or by disabling KASLR.