topic Re: Watchdog reboot on overloading GPU in SCM IMX6Q in i.MX Processors

Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Mon, 14 Aug 2017 07:05:05 GMT

Hi,

I am using Linux imx-4.1.15-2.0.0_ga release and Yocto krogoth build.

I have stress tested GPU with glmark2 GPU application to check for over heating of IMX6 SCM when running GPU.

IMX6 SCM is very quickly reaching the threshold of 85 degree and reducing the GPU clock to 1/64. I have launched multiple instances of glmark2 application to overheat the module.

On doing this test occasionally i got system reboot, this happens after being in non-responsive state for sometime. Display is going blank when system enters the non-responsive state.

Is there any latest patch which fix this issue?

Is there any known issue particular to SCM module?

There is a separate release for SCM https://community.nxp.com/docs/DOC-333955

Will this make any difference in BSP/Kernel or in fixing this issue?

Looking forward for some response. Meantime i am planning to build above release and test it.

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

art — Fri, 18 Aug 2017 11:50:57 GMT

Q. Is there any latest patch which fix this issue? Is there any known issue particular to SCM module?

A. There seems to be nothing to fix in software since overheating can cause any unpredictable behaviour of the product. The only possible "fix" is to provide better cooling for the SCM.

Q. There is a separate release for SCM Yocto BSP for SCM-i.MX L4.1.15-2.0.0_ga

A. Yes, you have to use this build for SCM operation.

Have a great day,
Artur

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Mon, 21 Aug 2017 11:32:19 GMT

alejandrolozano‌ juangutierrez maybe you can help with the patch "WA GPU 3D OT" to fix the arbitration within the memory controller?

The priority between GPU2D/GPU3D’s QoS transfer and non QoS transfer from the core may be overloading one of the bus fabric (PL301) then to the MMDC. MMDC (memory controller) prioritizes QoS flag and prevents access from non-QoS data traffic (CPU). If GPU process requirements gets heavy, overrun error may occur due to logic race and lack of resources on bus fabric.

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

juangutierrez — Mon, 21 Aug 2017 15:54:27 GMT

Can you try to add this on the board_late_init function of the board file?

--- a/board/freescale/mx6dqscm/mx6dqscm.c
+++ b/board/freescale/mx6dqscm/mx6dqscm.c
@@ -1259,6 +1259,12 @@ int board_late_init(void)
#ifdef CONFIG_ENV_IS_IN_MMC
board_late_mmc_env_init();
#endif
+
+ printf("Limit GPU 3D OT=1 WA\n");
+
+ writel(0x3, 0x00C43108);
+ writel(0x3, 0x00C48108);
+
return 0;
}

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Wed, 23 Aug 2017 15:36:35 GMT

Thanks @Michael Guntli for pointing to the workaround.

Thanks @Juan Antonio Gutierrez Rosas for the patch.

Looks like it is working properly without any watchdog triggered reboot.

I don't find any description about the modified registers(0x00C43108 and 0x00C48108) in TRM, is it not available to all?

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Tue, 29 Aug 2017 07:13:04 GMT

Hi michaelguntli‌ juangutierrez‌

Thanks for the patch, now we are not getting the watchdog reboot during GPU operation but we are getting display freeze while running GPU application. This happens on a long run(for more than a day).

Debug consoles are active and peripherals are active except the GPU.

Do you have any suggestions on tuning QoS values to solve this issue?

Thanks in advance,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Tue, 29 Aug 2017 08:49:43 GMT

You mean your system locks up when you run a ARM CPU intensive application?

Interesting why the watchdog does not reset the system in that case..

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Thu, 31 Aug 2017 07:07:17 GMT

Hi michaelguntli‌

Sorry for replying late.

No, there is no system lock up now. Only the display freeze when i run GPU application for a long time and i am able to access the debug console(so the processor is active).

Looks like GPU rendering is not happening(locked), only after manual reset i am getting the proper display.

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Thu, 31 Aug 2017 18:16:20 GMT

Which SCM-i.MX6Q memory configuration are you using? 512MB / 1GB / 2GB?
If you could post a picture of the chip that would help to identify the exact chip revision.

Example: SCM-i.MX6D with 1GB LPDDR2

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Fri, 01 Sep 2017 11:05:33 GMT

Hi Michael,

We are using 1GB LPDDR2 from Micron(MT42L128M64D2).

Display freeze happens even with 640 x 480 resolution in HDMI.

So, without the above patch watchdog reset is happening due to CPU freeze and with the above patch display freeze is happening.

Looking forward for some help in solving this issue.

Thanks,
Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Fri, 06 Oct 2017 14:37:12 GMT

Hi Partha. Any luck so far?
We are currently experimenting a little bit, since we also observe sporadically a similar behavior.

You can try the following:

Change min GPU clock to 3/64 instead of 1/64
..to prevent GPU hang in case the thermal protect mechanism is activated.
https://community.nxp.com/thread/319210#comment-378939
Increase VDD_ARM_IN and VDD_SOC_IN voltage to 1.275V
..according to the values of the datasheet in kernel-imx/arch/arm/boot/dts/imx6q.dtsi for LDO enabled mode.

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Thu, 12 Oct 2017 07:57:02 GMT

Hi michaelguntli‌

Thanks for the patch.

With this update we are not seeing reset or hang, we are continuing the test further to conclude on this.

Do you have any idea on the impact of this voltage change in SoC power consumption?

Will keep update once the testing is completed.

Thanks again for your continued support.

Regards,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Thu, 12 Oct 2017 11:21:27 GMT

Good to hear!

We are exactly working right now on the same problem, that's why I was able to respond so quickly.

So I assume you are using the internal LDO of the i.MX6, and not use LDO-bypass for voltage generation?

Explanation of ldo-enabled vs. ldo-bypass: ventana/power – Gateworks

The setpoints above are only applied to the internal LDO. Besides power consumption, a major problem for us with the internal LDO is the additional heat it produces.

Good for you that we are already one step further: we are testing stability with LDO-bypass configuration. With LDO bypass we were able to reduce the temperature by around 15%.

LDO-Bypass:

SCM-i.MX6D has the PF0100 PMIC built in, so if you decide to use LDO bypass:

Enable fsl,ldo-bypass flag in kernel dts config
Fix the VDD_ARM_IN / VDD_SOC_IN mismatch in U-boot (at least it's wrong for the evaluation board)
Rebuild and verify stability, enjoy the reduced heat and power consumption :-)

Detailled explanation:

How to Enable LDO Bypass Based on i.MX6 Android ICS

Set ldo-bypass flag your dtsi file: kernel//arch/arm/boot/dts/imx6dscm-freeX.dts

fsl,ldo-bypass = <1>;

There is an important change which is required in the U-Boot: Fix the inverted mapping of VDD_ARM_IN and VDD_SOC_IN to the PMIC outputs (double check with your custom hardware, at least for the evaluation board it's wrong): u-boot/board/freescale/mx6dqscmqwks/mx6dqscmqwks.c

Bug: The schematics of the evaluation board does not match the code (inverted)!

Schematics evaluation board:

PFUZE100_SW1ABVOL = VDD_SOC_IN

PFUZE100_SW1CVOL = VDD_ARM_IN

Code:

PFUZE100_SW1ABVOL = VDD_ARM_IN

PFUZE100_SW1CVOL = VDD_SOC_IN

We are currently running the following config and testing the stability (1.175V VDD_ARM_IN, 1.20V VDD_SOC_IN):

u-boot/board/freescale/mx6dqscmqwks/mx6dqscmqwks.c

void ldo_mode_set(int ldo_bypass) {
...
   /* set SW1C to 1.175V (VDD_ARM_IN) to compensate ripple */
   pmic_reg_read(pfuze, PFUZE100_SW1CVOL, &value);
   value &= ~0x3f;
   value |= 0x23;
   pmic_reg_write(pfuze, PFUZE100_SW1CVOL, value);

   /* set SW1AB to 1.20V (VDD_SOC_IN) to compensate ripple */
   pmic_reg_read(pfuze, PFUZE100_SW1ABVOL, &value);
   value &= ~0x3f;
   value |= 0x24;
   pmic_reg_write(pfuze, PFUZE100_SW1ABVOL, value);
...

FYI: depending on your hardware setup, you might have to increase SW1C and SW1AB to compensate for ripple (it should NEVER get below 1.15V)

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Fri, 13 Oct 2017 08:36:43 GMT

Hi michaelguntli Nice Work

Yes we are using internal LDO mode. We are not getting any issue with our 3 days of testing(apart from over heating issue).

Now we have started testing with LDO by-pass mode. Currently we have set CORE and SOC voltage to 1.3V(maximum value) in PMIC, will let you know the updates.

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Fri, 13 Oct 2017 14:01:37 GMT

Hi michaelguntli‌

We are seeing some issue with ldo-bypass mode which is similar to the one discussed in the below thread:

https://community.nxp.com/thread/391453

Any thoughts?

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Mon, 16 Oct 2017 05:44:54 GMT

Hi Partha

No sorry, we are using an "old" kernel L3.14.52-ga which is part of Android L5.1.1: https://community.nxp.com/docs/DOC-329594

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

parthasarathyr — Mon, 16 Oct 2017 14:53:33 GMT

Hi michaelguntli‌,

We found a patch to fix the issue.

Below is the patch link:

Patch "regulator: anatop: allow regulator to be in bypass mode" has been added to the 4.4-stable tree — Linux Stable Ker…

Now we have started testing the with ldo bypass enabled.

Will let you know results soon.

Thanks,

Partha

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Tue, 17 Oct 2017 06:22:31 GMT

Hi Partha

Good to know, thanks for the patch!

Important: Please measure the ripple on your custom hardware design, the voltages VDD_ARM_IN and VDD_SOC_IN should never get below the specified minimum voltages.

FYI: We noticed higher ripple on VDD_SOC_IN when the system was stressed (e.g. GPU performance test).

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

michaelguntli — Thu, 19 Oct 2017 12:04:22 GMT

Hi Partha

Any feedback regarding LDO bypass operation?

Which voltages are you currently operating? Is the system still stable?

LPDDR2
VDD_ARM_IN
VDD_SOC_IN

Re: Watchdog reboot on overloading GPU in SCM IMX6Q

taehyukkwon — Mon, 23 Oct 2017 20:58:49 GMT

Hi,

I am seeing the same issue on Android 4.3.1 build.

I tried to apply the same change to the kernel used in the build, but i cant find the files. Could you help me to have the equivalent changes for Android 4.3.1 kernel ?

Thanks,