i.MX6 solo resume from suspend (DSM) problem

[billlau]

Hi all,

Background of hardware and software used.

1. Freescale or other standard reference board used = custom design board
2. Kernel version and/or BSP release used = based on 3.0.35
3. Any additional software/application or hardware used = i.MX6 Solo, 512MB DDR2 RAM (Micron MT42L128M32D1LF-18 WT:A @396MHz), 1GB NAND flash (Toshiba TC58NYG3S0FBAID)

Recently, we found about 1% of our production yield (an android phone based on i.MX6 solo) had a problem which fail to resume normally from suspend mode (DSM).

We checked the failure unit could work  normally before DSM. But once it went to DSM, it cannot resume anymore unless cold start or hard reset.

I have gone though some discussion in the forum, the resume procedure from DSM should be:

In a word, for exiting from DSM, the analog will take < 1mS to prepare normal working power(voltage up from standby mode to normal mode), then waiting timer to count down, about 0.5mS before re-power OSC, then OSC takes about 2mS to lock, but we wait for about 8ms to make sure OSC is enabled, then PLL takes about 0.45mS to re-lock, then wait for another 2mS to re-power the ARM core and another 2mS to reset ARM core. So it takes about 14mS to exit from STOP mode, here exit means ARM core restart to run.

Our observation :

1. During DSM, the internal regulator PU and ARM regulator are disabled  (VDD_ARM_CAP, VDD_PU_CAP dropped below 300mV),  VDD_SOC_CAP = VDD_SOC_IN (SoC regulator was by-pass), 24M OSC has been shutdown.

2. Pressing the button on the phone would trigger an wake up event to i.MX6 by an interrupt pin

3. Starting from the negative edge at the interrupt pin, VDD_SOC_CAP = 1.2V (SoC regulator was no longer by-pass)

4. And then after 1.1ms, PMIC_STBY_REQ was de-asserted.

5. And then after 1.6ms, the 24M OSC started to oscillate. and took additional 2.4ms to become stable

6. After further 5.3ms, the VDD_ARM_CAP was established.

7. Then here came a difference compare to normal units and failure units. After the VDD_ARM_CAP setup, a normal unit will successfully setup VDD_PU_CAP after 3ms.

But for failure units, VDD_PU_CAP will always keep  below 300mV and never rise to 1.2V.

8. When the resume was fail, the kernel was not running. current consumption kept at same level steadily

So, it seems that there was a failure after VDD_ARM_CAP has powered up. At that moment, the ARM core should be reset. My assumption is that it locked up at that ARM core reset  and  then VDD_PU_CAP could not be powered up normally.

-We also tried to supply VDD_ARM_CAP, VDD_PU_CAP externally. But the resume from DSM was still not successful.

-We also try to trigger the ONOFF pin in the iMX6 instead of interrupt pin to wake up from DSM. It had the same behavior mentioned above.

Question:

1. What possible causes could make the VDD_PU_CAP not able to setup?

2. In our custom board, VDD_PU_CAP has no external load. And we only added one 22u bulk capacitor stated in the requirement. And all power rail (VDD_ARM_IN, VDD_SOC_IN, VDD_HIGH_IN, VDD_SNVS_IN, DRAM VDD) were kept unchanged during DSM. Suppose the failure should not be a power source problem. Do you agree?

3. Is it related to DRAM problem? (fail to refresh during DSM or restore all setting after resume)

4. What else we can measure in hardware to secure a resume from DSM?

[Ajesh_Ametek1]

Hi @billlau 

Have you found solution for this issue? We are facing similar problem in our custom board.

It will be very helpful if you can post here the solution you found...

[igorpadykov]

Hi Bill

after reading problem I am inclined to think

that this may be caused by power supplies : spikes/surges

on power supplies rails when processor wakes up.

Please try to increase times between VDD_ARM_CAP  and VDD_PU_CAP

setups. In general VDD_PU_CAP may be turned on later, when processor

competely exited from DSM and fully running. One can decrease arm

core frequency before entering DSM, this may save overall power consumption

and effect in less noise after resuming.

Another check point: 32.768KHz  and 24MHz clocks, please check

that waveforms were smooth and stable during power-up (wake-up) process.

Noise, caused from charging power supplies capacitors may affect crystal stability

(both 32.768KHz  and 24MHz) - this may lead to hanging.

Best regards

chip

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------

[billlau]

Hi chipexpert,

thanks for your reply.

1. For the check point, increasing times between VDD_ARM_CAP  and VDD_PU_CAP, I will do that and come to you again later as this required some software changes.

but i have already tried to provide the above power rail externally (providing 1.2V even in DSM) . But it did no help.

2. For another check point, 32.768 is always on since the system was boot at the very beginning.

and 24MHz clock looks stable after 4ms from the falling edge of PMIC_STBY_REQ.

I have attached some waveform mesasured for

VDD_ARM_CAP,

VDD_SOC_CAP,

VDD_PU_CAP,

32KHz_OSC,

24MHz_OSC,

VDD_SOC_IN,

VDD_ARM_IN,

VDD_HIGH_IN

which they were referenced to PMIC_STBY_REQ.

it looks only VDD_PU_CAP was abnormal. It could never rise to correct logic level, 1.2V after gone to DSM.

[billlau]

Hi Chipexpert,

one more point, the reason why i am pointing to VDD_PU_CAP, is not meaning the VDD_PU_CAP create the failure. My idea is that, the resume failure happened just before VDD_PU_CAP setup. I am just thinking what activities in the chip was doing just before  VDD_PU_CAP setting up?

Best regards,

Bill

[igorpadykov]

Hi Bill

could you try to resume without enabling VDD_PU_CAP

and check ? Regarding activities, they are described in

sect.18.5.3.3.2 "Exiting STOP mode"

IMX6SDLRM i.MX 6Solo/6DualLite Applications Processor Reference Manual

Best regards

chip

[billlau]

Hi Chipexpert,

Thanks for your reply again. We are building software snapshot for disabling the VDD_PU_CAP.

but it requires some time.

at the same time, I am digesting the activities when exiting stop mode.

when i read 10.4.1.4.3.1 (Power Gating domain managment - cortex-A9 Core platform)

when arm core was power down during DSM, the isolation enable is logic low

does it mean ARM core signal is not isolated during DSM?

as i read 10.4.1.4.4, it mentioned isolation needs to be enable before power down.

also, when i read 28.6 figure, (power gating control)

It also shows, when domain was powered down, the isolation is enable (logic high)

the gpc:cpu_pcg:pgc_iso and gpc:cpu_iso would then pull low (disable) when exiting DSM.

Question:

1. i am confused should the isolation enable during DSM and disable during exiting DSM?

2. is the gpc:cpu_iso needed to be disable before enabling the src:arm_cpu_clk_en? is there any requirement for the isolation time delay [PGC_CPU_PUPSCR]? As  i plan to adjust a better delay margin  to secure a successful reset of arm core.

Best regards,

Bill

[igorpadykov]

Hi Bill

1. yes isolation is implemented, but this is internal (hidden from user) process

2. BSP have most optimal settings for these parameters, I do not think that changing these

   parameters will help.

I would suggest to concentrate on power noise casued by GPU (turn on/off).

Best regards

chip

[billlau]

Hi Chipexpert,

Ok, thanks for your suggestion, we will firstly concentrate on power noise. But for the experiment requiring software setting changes, we need some time to build a snapshot as most of our core software engineers are in summer vacation in July. I will come back once there are some more findings.

[igorpadykov]

Hi Bill

Ok, no problem.

Have a good vacation !

Best regards

chip

[billlau]

Hi Chipexpert,

for your information,

we are using imx_r13.4-ga, with patch for:

• bdata.bin variants for Wifi
• update GPU binaries to p12 for ICS 13.4-ga release

Bill

[igorpadykov]

Hi Bill

when exiting from DSM, turning ON GPU

may produce big power noise.

Best regards

chip

[msandersen]

Hi Chip

I've tried disabling the gpu_power_up() call when resuming from DSM in arch/arm/mach-mx6/pm.c. Problem seems to be the same. Shouldn't that be enough to stop the LDO from turning on?

We also have a LED attached so I've tried debugging a bit with this in arch/arm/mach-mx6/mx6_suspend.S. I have tried turning it on just before DSM is entered and also after it exits from DSM:

ldo_check_done1:
// MSA turn on led here works OK
/****************************************************************
execute a wfi instruction to let SOC go into stop mode.
****************************************************************/
    wfi

    nop
    nop
    nop
    nop

/****************************************************************
if go here, means there is a wakeup irq pending, we should resume
system immediately.
****************************************************************/
// MSA turn on led here doesn't work when used on platform that freezes on resume.

    /*restore it with 0x1f if use ldo bypass mode.*/
    ldr    r1, =ANATOP_BASE_ADDR
    add    r1, r1, #PERIPBASE_VIRT

For me it looks line no code is ever executed when returning from DSM.

Any suggestions?

Best regards,

Mikkel

[igorpadykov]

Hi Bill

probably you can just build image without gpu usage at all.

Other option reduce operating frequency (core, DDR).

Best regards

chip

[msandersen]

Hi Chip,

I've tried the following:

1. No GPU enabled at all
2. ARM clock root @396 MHz and DDR clock @198 MHz (this is half the speed we are normally running)

It changes nothing. Board still freezes on resume.

Any other ideas?

Best regards,

Mikkel

[igorpadykov]

Please also try next latest internal findings:

check tXPDLL (try to increase it) and

another suggestion is try to configure DDR chip to fast mode:

i.MX6 MMDCx_MDPDC:

Bit 7, SLOW_PD  Slow/fast power down.

NOTE: Memory should be configured the same.

0 Fast mode.

1 Slow mode.

A12 define as following info.

Accordingly bit 12 of MR0 DDR3 should be set:

3.4.2.6 Precharge PD DLL

MR0 (bit A12) is used to select the DLL usage during precharge power-down mode. When MR0 (A12 = 0),

or ‘slow-exit’, the DLL is frozen after entering precharge power-down (for potential power savings) and

upon exit requires tXPDLL to be met prior to the next valid command. When MR0 (A12 = 1), or ‘fast-exit’,

the DLL is maintained after entering precharge power-down and upon exiting power-down requires tXP to

be met prior to the next valid command.

Note, latest script (0.09) changed bit 12 MR0 to 1 compared with previous (0.08) version.

Mx6DQSDL DDR3 Script Aid V0.09.xlsx (87.5 K)

i.Mx6DQSDL DDR3 Script Aid

Another testing would be increasing CCM_ANALOG_MISC2n REG0_STEP_TIME to max value

and implement ERR005852 workaround IMX6DQCE .

~igor

[igorpadykov]

Hi Bill

if this is small number of boards, as it is written in

beginning: "about 1% of our production".

Then probably this is hardware issue, may be just poor soldering.

Please try to resolder processor.

Best regards

chip