GC7000 GPU hang galcore timeout

[Ram2]

Issue Summary

During execution of NavApp, the display freezes completely. Investigation of the kernel logs indicates that this is not an application crash, but a Vivante GC7000 GPU (Galcore) hang.

Observations:

• The kernel reports a GPU timeout after 30 seconds (gpuTimeout = 30000).
• Galcore generates a GPU State Dump and invokes:
• gckKERNEL_Recovery()
• gckHARDWARE_DumpGPUState()
• The driver then reports:
  [galcore]: Stop driver to keep scene.
• The GPU state dump indicates multiple GPU blocks are stuck:
• FE not idle
• SH not idle
• TX not idle
• MC not idle
• DMA appears to be stuck

Driver Configuration

Current Galcore configuration shows:

recovery = 0 gpuTimeout = 30000

Since GPU recovery is disabled (recovery = 0), the driver stops after detecting the hang, leaving the last rendered frame on the display, which explains the observed screen freeze.

GPU Memory Status:

GPU memory statistics do not indicate memory exhaustion.

• Total GPU memory: 256 MB
• Used: ~155 MB
• Free: ~113 MB

Therefore, the issue does not appear to be caused by GPU out-of-memory.

GPU Client:

The GPU database shows that NavApp is the only active GPU client at the time of the hang.

The GPU state dump references multiple command buffers submitted by: NavApp

This indicates the GPU hang occurred while executing rendering commands submitted by NavApp.

Application Timeline:

From the application logs, the following sequence is observed immediately before the GPU hang:

• Continuous map zoom in/out operations
• Offline map/tile access
• Route calculation
• Route information display
• GPU timeout and state dump

The GPU hang occurs immediately after intensive rendering activity.

Conclusion:

Based on the collected evidence:

• There is no kernel panic, OOM, or application crash.
• The failure is a GPU command processing hang detected by the Galcore watchdog.
• NavApp is the rendering client when the hang occurs, but the failure is observed inside the GPU driver.
• GPU memory usage is within limits and does not indicate resource exhaustion.

Request to TVS team please investigate:

1. Vivante GC7000 / Galcore GPU driver hang during rendering.
2. Whether this issue is a known GPU driver or firmware limitation.
3. Whether GPU recovery (recovery=1) is supported and recommended on this platform.
4. Analysis of the GPU state dump to identify which GPU command or hardware block caused the timeout.
5. Whether there are updated BSP, GPU driver, or firmware releases addressing similar GPU timeout issues.

[Zhiming_Liu]

Hi @Ram2 

Please provide SOC part number, BSP verison and reproduce steps.

Best Regards,
Zhiming

[Ram2]

Hi @Zhiming_Liu 

SOC - iMX8qxpComek

BSP version - Scarthgap L6.6.5

スポイラ

スポイラ

[Zhiming_Liu]

Hi @Ram2 

Please provide steps to reproduce the issue on the EVK.

Best Regards,
Zhiming

[Ram2]

Hi @Zhiming_Liu 

Reproduce Steps :

The screen freeze issue does not have a fixed set of reproduction steps, as it occurs randomly and intermittently. However, I have observed that it is more likely to occur when the available system memory becomes very low (typically below 60 MiB).

To increase the likelihood of reproducing the issue, I performed the following activities continuously:

• Created 2-3 long-distance routes.
• Started navigation and exited within about a minute.
• Repeatedly created routes using the Home, Office, and Hotel quick action buttons.
• Explored different map regions by panning and performing frequent zoom in/out operations.

During these activities, the screen freeze was observed when the available memory dropped to a low level. For example:

MiB Mem : 1709.5 total, 55.8 free, 1298.1 used, 567.2 buff/cache

MiB Swap: 0.0 total, 0.0 free, 0.0 used, 411.4 avail Mem

This suggests the issue is more likely to occur under sustained usage when free memory is nearly exhausted.