i.MX6S Lock up in VPU on resume from Deep Sleep

[gennadiykiryukh]

I have a custom board based on i.MX6S that some times locks up when the system wake up from deep sleep.

Using pr_info(), the lock up was traced to one line in vpu_resume(struct device *dev) function.

Two pr_info() lines were added around pc = READ_REG(BIT_CUR_PC) located around line 1170 of mxc_vpu.c:

if (bitwork_mem.cpu_addr != 0) {
   u32 *p = (u32 *) bitwork_mem.cpu_addr;
   u32 data, pc;
   u16 data_hi;
   u16 data_lo;

   pr_info("Enabling vpu_clk\n");
   clk_prepare(vpu_clk);
   clk_enable(vpu_clk);

   pr_info("READ BIT_CUR_PC 0x%08X\n", (unsigned int)(vpu_base + BIT_CUR_PC));
   pc = READ_REG(BIT_CUR_PC);  // <<< - locks up here
   pr_info("READ BIT_CUR_PC end\n");
   if (pc) {

   ...

In the terminal window I see:

Enabling vpu_clk
READ BIT_CUR_PC 0x88A40018

and nothing after that. "READ BIT_CUR_PC end\n" is not printed.

It does not lock up every time it wake up from deep sleep. I have to constantly generate GPIO interrupts (push-button) to keep waking it up for the board to lock up on that line. The line seems to expand through a macro to one assembly instruction reading a memory location. this may point to some  deeper issue where the processor is unable to fetch a memory location (hardware register BIT Current PC (VPU_BitCurPc), section 69.7.6 of 6Solo/6DualLight Reference Manual).

clk_enable() function call two lines before returns 0 (success) every time (even when the system locks up).

What could be the cause and how can we go about fixing it?

Thank you.

[igorpadykov]

Hi Gennadiy

issue may be caused by memory errors and one can rerun ddr test and update

image with new ddr calibration coefficients

i.MX6/7 DDR Stress Test Tool V3.00 

also one can recheck low power drivers (like suspend-imx6.S) in

suspend-imx6.S\mach-imx\arm\arch - linux-imx - i.MX Linux kernel 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[gennadiykiryukh]

After more digging, I came to an interesting observation. According to imx6s/dl reference manual, power gating section 10.4.1.4.3.2 GPU3D, GPU2D and VPU, a delay between applying power and removing isolation is defined by SW count value (Figure 10-9 on page 512). However, section 28.8.2 Power Up Sequence Control Register says that delay between asserting power and negating isolation is defined by SW2ISO bits and SW bits control power-on delay.

It seems that figure 10-9 is misleading. But even then there are only 6 bits to define the delay. Which makes maximum delay between applying power and removing isolation 63 counts. At 66MHz PGC clock this is about 1 microsecond.

Below is the illustration of the rise time for VDDPU_CAP coming out of deep sleep mode.

As you can see, it takes 12 microseconds to get to 0.8V. After 1 microsecond, the voltage would be around 0.8/12 = 70mV. That means the isolation of GPU/VPU module will be negated long before the voltage rises to what is seems an acceptable level.

Could that cause the module to power up in some unknown state?

Gennadiy

[igorpadykov]

Hi Gennadiy

you can run Demo Images from below link on NXP Sabre SD board and

check signals with oscilloscope and compare them with custom board.

i.MX Software | NXP 

Best regards
igor

[gennadiykiryukh]

Below is  VDDPU_CAP at wake-up on the eval board (i.MX6DL). Rise time is the same but it does not have that step ramp.

I've have changed REG1_STEP_TIME in PMU_MISC2n from 512_CLOCKS down to 64_CLOCKS.

Now, my VDDPU_CAP looks the same as on eval board and the wake-up timing looks as follows:

However, the board is still locking up in the VPU driver when trying to access READ_REG(BIT_CUR_PC). What is my next step?

[igorpadykov]

Hi Gennadiy

one can check latest i.MX6 System Development User’s Guide

https://www.nxp.com/webapp/Download?colCode=IMX6DQ6SDLHDG 

For ramp times attached document.

Best regards
igor

[gennadiykiryukh]

Kernel version I'm running is 4.1.15. It is the same version as on the eval board. Patch you mentioned are for L3.0.35_3.0.0.

Section 28.8.1 PGC Control Register (PGC_GPU_CTRL) of the manual says that for register PGC_GPU_CTRL, "PCR must not change from power-down request (pdn_req) assertion until the target subsystem is completely powered up."

However, in the code for powering up VPU:

static void _imx6q_pm_pu_power_on(struct generic_pm_domain *genpd)
{
   struct pu_domain *pu = container_of(genpd, struct pu_domain, base);
   int i, sw, sw2iso;
   u32 val;

   /* Enable reset clocks for all devices in the PU domain */
   for (i = 0; i < pu->num_clks; i++)
   clk_prepare_enable(pu->clk[i]);

   /* Gate off PU domain when GPU/VPU when powered down */
   writel_relaxed(0x1, gpc_base + GPC_PGC_GPU_PDN);

   /* Read ISO and ISO2SW power down delays */
   val = readl_relaxed(gpc_base + GPC_PGC_GPU_PUPSCR);
   sw = val & 0x3f;
   sw2iso = (val >> 8) & 0x3f;

   /* Request GPC to power up GPU/VPU */
   val = readl_relaxed(gpc_base + GPC_CNTR);
val |= GPU_VPU_PUP_REQ;
writel_relaxed(val, gpc_base + GPC_CNTR);

/* Wait ISO + ISO2SW IPG clock cycles */
ndelay((sw + sw2iso) * 1000 / 66);
pr_warn("SW %d, SW2ISO %d", sw, sw2iso);

/* Disable reset clocks for all devices in the PU domain */
for (i = 0; i < pu->num_clks; i++)
clk_disable_unprepare(pu->clk[i]);
}

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Line number 12 "writel_relaxed(0x1, gpc_base + GPC_PGC_GPU_PDN);" in the code above attempts to write 1 to bit 0 of PGC_GPU_CTRL register before VPU is powered up. Does it contradict the requirement in the manual?

What conditions or state of the processor/peripheral would cause the processor to hang on reading  VPU register VPU_BitCurPc described in section 69.7.6 BIT Current PC (VPU_BitCurPc) of the manual? 

[gennadiykiryukh · Accepted solution]

Found it (sort of)...

It looks like a driver bug as there is a possibly unintended dependence of VPU on the USB driver (or possibly others). Since my board does not use USB, I had it removed from the Kernel and the device tree. I have not figured out the cause but adding it back fixed the lock-up on wake up from deep sleep in the VPU driver. I was able to reproduce the problem by loading my kernel onto the the eval. board. Removing modules from the Kernel one at a time, I has able to determine that presence of the USB driver in Kernel makes VPU come out of sleep every time. It probably has something to do with the order and how the peripherals/clocks are resumed after deep sleep.

[gennadiykiryukh]

As a side note, for some reason this question is marked as "assumed answered". It is not answered. The problem was not resolved.

[gennadiykiryukh]

I looked over suspend-imx6.S file. There was nothing out of ordinary.

Here are some imx6 power rails on wake up.

On the cycles when the processor locks up in VPU driver, the signals look exactly the same.

I tried putting VPU/GPU LDO regulator into LDO bypass mode, similar to what was mentioned in ERR005852. No change, VDDPU_CAP looked the same.

Is it how the diagram supposed to look like? For some reason VDDARM_CAP rises faster, where as VDDPU_CAP rises slower and after reaching around 800mV you can see a nice ramp all the way up to 1.25V. Is that how it is supposed to be?

Shouldn't VDDARM_CAP and VDDPU_CAP look similar? Shouldn't they rise at the same time?

Gennadiy

[gennadiykiryukh]

Hi Igor,

At the very beginning, I thought it would be a problem with initializing DDR. However, after setting no_console_suspend=1 (or whatever the exact flag is) in bootargs and adding pr_info(), I determined that that it called a bunch of resume callback functions on 25 drivers before freezing up. When it locks it is always while accessing VPU program counter register.

DDR was calibrated. I've checked the voltage on VDDPU_CAP and it is around 1.25V when the system locks up.

Important to note that I call vpu_Init() function (defined in VPU library) when my program starts up. Opening the device makes the driver run extra code in suspend/resume to save/restore VPU state. It acts like the VPU hardware  is in some weird state when reading VPU program counter causing the CPU to wait indefinitely for the fetch to complete.

I will be looking into suspend-imx6.S to see if there is anything happening.

In the meantime, any other ideas would be greatly appreciated.

Thanks,

Gennadiy