[IMX8MM] Android 14 GPU crashed when first boot

[mayqueen]

Platform: IMX8MM

OS: Android 14

Kernel: 6.1.55

Good news is our IMX8MP product already done using Android 14, but bad news is we face another issue for IMX8MM product as title.

That strange because it only happens on two conditions:

1. first boot after UUU image flashing (it's fixed, 100% happened), it happened after boot complete and initialing the GUI desktop.

2. Playing 1080P video for about 30mins, and press back for the instance time (randomly)

current CMA size is 550MB, I'm tried change to 800MB but the issue still the same, note that Android 13 has no this problem, so I think it's GPU driver bug? The crash log as attached file, have any directions? Because I have no IMX8MM EVK board, could you have the same issue on the EVK?

I checked the dumpsys meminfo, it seems no memory leak.

Another concern is I saw the imx8mm.dtsi, the GPU power domain already change to GPC based, is it the root cause? BTW, my DRAM controller frequency is 1200MHz (2400MTS), I don't know GPU or VPU pll need change or not, BUT Android 13 is normal!! I have no any change on it.

Thanks a lot

[AshutoshNama]

@mayqueen Try below way, I hope this will help you:
Driver Analysis: Examine the GPU driver for incompatibilities between IMX8MM and Android 14, paying particular attention to startup and edge cases for video playback.

CMA Configuration: Verify that the CMA size is enough in high-memory circumstances; 800MB might not resolve because of memory requirements unique to the GPU.

Power Domain: Check for configuration errors or regressions affecting the stability of Android 14 by looking at GPU power domain changes in imx8mm.dtsi.

Clock Frequencies: Verify GPU/VPU PLL settings against Android 14 specifications; discrepancies might cause instability even if Android 13 is functioning well.

EVK Validation: To identify hardware-specific problems and collect repeatable data, test on an EVK board.

Best Regards,
Ashutosh Nama.

[prakashm]

i have two imx8mm mini with 2gb and 4gb ram ..2gb works with cma=800M@0x400M-0xb80M in android.

but in 4gb i see the crashes in first boot & sometimes random boot crashes with same above cma memory. tried to reduce the cma memory but it doesn't solve it.

the below log when i try to change the cma memory to
cma=640M@0x92000000

also i see this dma stuck problem:
AQ_HI_CLOCK_CONTROL = 0x00040100
[ 105.714347][ T8] DMA appears to be stuck at this address:
[ 105.720195][ T8] 0x92D10C50
[ 105.723612][ T8] dmaLow = 0x48000000
[ 105.727810][ T8] dmaHigh = 0x00000701
[ 105.732006][ T8] dmaState = 0x00000915
[ 105.736205][ T8] command state = 21 (PAR_STALL_ST)
[ 105.742142][ T8] command DMA state = 1 (CMD_START_ST)
[ 105.747996][ T8] command fetch state = 2 (FET_VALID_ST)
[ 105.753847][ T8] DMA request state = 0 (REQ_IDLE_ST)
[ 105.759607][ T8] cal state = 0 (CAL_IDLE_ST)
[ 105.765368][ T8] VE request state = 0 (VER_IDLE_ST)
[ 105.771129][ T8] Debug registers:

[prakashm]

also our android 13_2.2.0 which uses Linux kernel 6.1 and its working fine there.. i see the same device tree configuration for both android 13 and android 14 in terms of gpc and gpu

[mayqueen]

I aleady resolved about few months ago, thanks

[robertosartori]

Hi, how did you solve this issue?

[prakashm]

‎arch/arm64/boot/dts/freescale/imx8mm.dtsi

gpu: gpu@38000000 {
compatible ="fsl,imx8mm-gpu", "fsl,imx6q-gpu";
reg = <0x0 0x38000000 0x0 0x8000>, <0x0 0x38008000 0x0 0x8000>,
- <0x0 0x40000000 0x0 0x80000000>, <0x0 0x0 0x0 0x8000000>;
+ <0x0 0x40000000 0x0 0xC0000000>, <0x0 0x0 0x0 0x8000000>;

you can change like this to adapt 4gb. ths works for me.

[robertosartori]

@Zhiming_Liu do you have any information about this problem? I'm observing the same on a custom board (iMX8MM) with 4GB of ram. The same device works if configured to work with 2GB of ram (same hardware).

[prakashm]

@mayqueen have you managed to solve this issue on android14 ..because i'm facing the first time boot problem and my gpu driver crashing. it would be helpful if you could share your experience on this. thanks in advance

[mayqueen]

Yes, I was fixed.

Crashed at the 1nd boot is must be happened, actually it still trigger random crashed since 2nd boot.

In my experience, it's due to memory and CMA allocation does not stable, I guess you adapt a special memory chip, right? Double check the memory calibration or hardware signal has no problem. Secondly, a workaround is trying to reduce the usable memory size in Android (modify the dts and increase the reserve memory), it will works fine.

My solution is change back to micron memory chip which NXP EVK used.

[prakashm]

in my case the android 13 works with cma that i used on adnroid14 ..but wondering how it could be the memory problem..

[mayqueen]

YES, Android 13 works because the GPU driver and power domain driver were still old structure, but it's a big jump so cannot downgrade to the same driver on Android 14, detail can follow Ashutosh Nama comment as above.

I think memory allocation is very sensitive on new drivers, this is why I decide change the memory chip or decrease memory usage to fix this issue, thanks.

[prakashm]

thanks mayqueen for your comment.

[prakashm]

thank you so much for your reply mayqueen

[Zhiming_Liu]

Hi @mayqueen 

Can you share your Android13 version?

The GPU library in  Android13_2.2.0 is same as Android14, but not different in Android13_1.0.0.

I can't reproduce same issue with Android14_1.2.0. Can you also try this verison?

Thanks

Zhiming

[mayqueen]

android-14.0.0_1.2.0 still has this issue, but it can observe the happen probability is lower than 1.0.0.
My MIPI-DSI panel is 1280x800 base on panel-simple.c, is it the root cause?

But anyway, Android 13 is works perfectly, so I'm so confusing...

So I have to give up Android 14 and lose the order?

[Zhiming_Liu]

Hi @mayqueen 

As we can't reproduce same issue when first boot on EVK. Here is GPU SE team suggestion:

Customer can use Android13_1.2.0 GPU library in Android14. If they use Android 14 image, they have to use android 14 environment to compile Android 13 galcore.ko, otherwise it won't start. If you can't compile Android 13 galcore.ko in Android 14, please create a case and SE team can help(Please create case to request this in SFDC).

[mayqueen]

Update status:

1. I adapt NON-GKI defconfig as default setting, if use GKI kernel, the first boot into the desktop is no problem, but when check Android 14 version in Setting -> About, it still stuck and happen GPU crash, using webview browser also trigger this issue.

2. I tried to downgrade GPU driver to andrord-13-1.2.0 version in Android 14, I use non-GKI kernel but the issue still existed, because I face compile failed using GKI version to build ko, so just pending this way, because after observed, GPU version should not the root cause, looks like power domain issue, NON-GKI and GKI have difference crash points, e.g.

3. If I use non-GKI defconfig, it means maybe I can revert your GPCV2, power domain and clock tree modification for GKI? BUT in Android 13, you also have those chagnes but I never revert them, it works perfectly.

[mayqueen]

My Android 13 version: android-13.0.0_1.2.0

Sure, I'll try Android 14 1.2.0, my current Android 14 is 1.0.0, I think this version has unstable on GPU.