imx8mp gpu Shader frequency drop

[jerryc]

Hello,

When performing extensive burn-in testing on our IMX8MP-based IP cameras, we noticed that some cameras start showing abnormal video output after running for a period of time. Using gputop, we observed that the 3D: GC8000, Rev: 8002 Shader frequency throttles down to 15 MHz (as shown in the image).
We checked the temperature using the command:cat /sys/class/thermal/thermal*/temp

and the temperature never exceeded 75°C.

Could you please help explain what might cause the 3D: GC8000, Rev:8002 Shader to throttle down?What can we do to prevent this issue?

P.S. The IP camera software performs H.264 encoding and fire/smoke video analytics.

 

 

[Zhiming_Liu]

Hi @jerryc 

Can you share the BSP verison and reproducible materials?

The possible reasons:

1. PMIC powerrail instability (voltage supplying the GPU) or undervoltage causes automatic throttling.

2. When shared memory bandwidth are exhausted (when the ISP/H.264 encoder and NPU simultaneously compete for bandwidth/power), the system reduces the GPU to its minimum frequency to ensure overall stability.

You can try using a simpler H.264 stream to reduce the load and see if the issue persists.

Best Regards,
Zhiming

[jerryc]

Hi Zhiming_Liu:

BSP verison:6.6-nanbield

Sensor:os05a10(2592x1944) or os08a20(3840x2160)

Brief description of the software functionality:
There are three H.264 encoded streams (1920×1080 @ 30 fps, 320×240 @ 30 fps, and 1920×1080 @ 5 fps).
The system uses V4L2 functions to obtain VI YUV data and perform H.264 encoding.
The G2D API is used for resizing and RGB conversion.
For the AI module, image analysis is performed using the original-resolution VI RGB data.

[Zhiming_Liu]

Hi @jerryc 

Will there still be problems after reducing the load?

Best Regards,
Zhiming

[jerryc]

Hi Zhiming_Liu:

As described previously, when performing a burn-in test with the software functionalities, excluding AI image analysis but retaining the camera's original features, we did not observe any GPU downclocking.

However, adding AI image analysis to the camera's original features is the purpose for which we developed this product using the i.MX 8M Plus. Therefore, the architecture of the camera's original features cannot be modified.

We can only attempt to mitigate the abnormal GPU downclocking phenomenon by reducing the load of the AI image analysis or by lowering the GPU frequency. Currently, our burn-in test is being performed with the GPU frequency adjusted from 1000 MHz down to 900 MHz.