IMX6SX Kernel 4.1.15 suspend/resume issue

daveebright · ‎03-30-2016

I decided to switch from the linux-rel_imx-3.14.52_ga kernel to the linux-rel_imx-4.1.15_ga kernel for use with the imx6sx sabre board. The 4.1.15 kernel built with no problems and the board boots up properly and runs as expected. However when I tried suspend and resume I ran into problems resuming. Note the suspend to mem (echo mem > /sys/power/state) also did not drop the current usage as low as it did in 3.14.52 either.

The problems looked similar to the problems I was having on the 3.14.52 kernel resuming (kernel would hang, the mmc card would have issues, or whenever I would access my encrypted partition errors would occur and need to reboot) before I did the following as documented in the i.MX Linux Reference Manual:

The full CAAM function is exclusive with the Mega/Fast mix off feature in DSM. If

CAAM is enabled, the Mega/Fast mix off feature needs to be disabled, and the user

should "echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2100000.caam/

2101000.jr0/power/wakeup" after the kernel boots up, and then Mega/Fast mix will

keep the power on in DSM.

With the 3.14.52 kernel everything was fine after I enabled the wakeup as suggested above.

I tried doing the same thing with the 4.1.15 kernel, but it has no impact any more. After some digging I found the following patch which appears to have changed this functionality by Marc Zyngier titled: ARM: imx6: convert GPC to stacked domains. It modifies the device tree interrupt structure to use stacked domains (not sure I totally understand this). Given these modifications I was wondering if anyone has a new workaround for this?

I did debug this and found that in arch/arm/mach-imx/gpc.c function imx_gpc_mf_mix_off() it no longer returns before turning off mega/fast mix in 4.1.15 like it did in 3.14.52 when the wake was enabled. I did hard code it to return before turning off mega/fast mix in 4.1.15 and that appears to fix the issue (current usage in suspend was lower and resumed properly, but not a real solution). Anyway I am going to keep working at this, but again I was wondering if anyone else had already encountered and fixed this problem. It is really noticeable if there is an encrypted file system partition and you attempt to access it after resuming. Also if you do the old 3.14.52 workaround, the resume code in 4.1.15 complains/warns about an "Unbalanced IRQ wake disable".

Also note that when you dump /proc/interrupts in 4.1.15 that the only GIC interrupts are now IRQ 309 2101000.jr0, IRQ 310 2102000.jr1, and IRQ 311 PCIe PME (almost all the rest are now GPC). In 3.14.52 almost all the interrupts were GIC (including those listed above, IRQ 137 2101000.jr0, IRQ 138 2102000.jr1, and IRQ 155 PCIe PME) and none were GPC. Looking at the IRQ masking registers 1-4 it looks like interrupts with IRQ numbers above 159 are not represented (maybe that is why the workaround no longer works because the IRQ is 309).

daveebright · ‎04-07-2016

Carlos,

The following are the patches that work for me while I am waiting for a more official response. These get around the issue I was having and no longer require the workaround of:

echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2100000.caam/2101000.jr0/power/wakeup"

If you only use the device tree patch, then the workaround is still required.

The call imx_gpc_mf_request_on(jrpriv->irq, 1); in jr.c caam_jr_suspend() sets the bit in the gpc_mf_request_on[] in arch/arm/mach-imx/gpc.c so that when imx_gpc_mix_off() is called during the suspend, then this function returns before turning off mega/fast mix (just like it used to in 3.14.52 with the workaround due to the wake IRQ being enabled). In 3.14.52 never saw the "Turn off M/F mix!" message in the system logs after the suspend/resume with the workaround, but in 4.1.15 without my patches you always see this message indicating the mega/fast mix was turned off (thus causing the problem where a reboot is needed to recover). The call to imx_gpc_mf_request_on(jrpriv->irq, 0); in jr.c caam_jr_resume() clears the bit.

Dave

--- linux-rel_imx_4.1.15_1.0.0_ga/arch/arm/boot/dts/imx6sx.dtsi.orig 2016-04-01 14:09:50.000000000 -0400

+++ linux-rel_imx_4.1.15_1.0.0_ga/arch/arm/boot/dts/imx6sx.dtsi 2016-04-01 14:10:34.000000000 -0400

@@ -928,14 +928,14 @@

sec_jr0: jr0@1000 {

compatible = "fsl,sec-v4.0-job-ring";

reg = <0x1000 0x1000>;

- interrupt-parent = <&intc>;

+ interrupt-parent = <&gpc>;

interrupts = <0 105 0x4>;

};

sec_jr1: jr1@2000 {

compatible = "fsl,sec-v4.0-job-ring";

reg = <0x2000 0x1000>;

- interrupt-parent = <&intc>;

+ interrupt-parent = <&gpc>;

interrupts = <0 106 0x4>;

};

--- linux-rel_imx_4.1.15_1.0.0_ga/drivers/crypto/caam/jr.c.orig 2016-04-01 12:34:12.000000000 -0400

+++ linux-rel_imx_4.1.15_1.0.0_ga/drivers/crypto/caam/jr.c 2016-04-01 15:09:58.000000000 -0400

@@ -7,6 +7,7 @@

#include <linux/of_irq.h>

#include <linux/of_address.h>

+#include <linux/imx_gpc.h>

#include "compat.h"

#include "regs.h"

@@ -543,6 +544,7 @@ static int caam_jr_suspend(struct device

struct platform_device *pdev = to_platform_device(dev);

struct caam_drv_private_jr *jrpriv = platform_get_drvdata(pdev);

+ imx_gpc_mf_request_on(jrpriv->irq, 1);

if (device_may_wakeup(&pdev->dev))

enable_irq_wake(jrpriv->irq);

@@ -554,6 +556,7 @@ static int caam_jr_resume(struct device

struct platform_device *pdev = to_platform_device(dev);

struct caam_drv_private_jr *jrpriv = platform_get_drvdata(pdev);

+ imx_gpc_mf_request_on(jrpriv->irq, 0);

if (device_may_wakeup(&pdev->dev))

disable_irq_wake(jrpriv->irq);

HectorPalacios · ‎02-09-2017

Hello,

Is there an official patch for this? I'm experiencing a similar problem in the i.MX6UL in kernel v4.1.

The workaround of doing echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2140000.caam/2141000.jr0/power/wakeup solves the issue but Dave's patch doesn't (at least not my problem on the i,MX6UL).

Thanks,

--

Héctor

CarlosCasillas · ‎04-07-2016

Hi Dave,

We are internally reviewing your case. In the meantime, you could post any additional information that you could consider useful.

Hope this will be useful for you.

Best regards!

/Carlos

daveebright · ‎04-01-2016

Given that in the device tree (imx6xsx.dtsi) for 4.1.15 the gpc entry still contains the line:

fsl,mf-mix-wakeup-irq = <0x7c00000 0x3d00 0x0 0x400240>;

where this represents all the MEGA domain wake IRQs which includes the CAAM job ring 0 interrupt:

0x400240

^

This bit is for CAAM job ring 0

However now with the 4.1.15 modifications, this interrupt was moved to the GIC domain which has no facility for wakeups. So this really breaks the mf-mix-wakeup-irq functionality. Therefore I suggest modifying the device tree entry minimally for sec_jr0 to change the interrupt-parent = <&intc>; to interrupt-parent = <&gpc>;. Thus now this interrupt will now be back in the GPC domain and the previous workaround will function properly. Could also make this change for sec_jr1, but currently the mf-mix-wakeup-irq mask does not reference it.

While I was looking into this, I also saw in drivers/crypto/caam/jr.c there are suspend and resume functions. It may be a good idea to add calls to imx_gpc_mf_request_on() to these functions which I believe could avoid the need for the workaround.

daveebright · ‎04-01-2016

After studying the modifications some more in 4.1.15 new code was added to translate all of the GPC interrupt values. For instance take for example the rtc alarm:

3.14.52 - 51: 0 GIC 51 rtc alarm

4.1.15 - 277: 0 GPC 19 Level rtc alarm

The GIC has the value 51 and to translate it to GPC simply subtract 32 which gets you to 19. This translation in 4.1.15 is done for all GPC domain interrupts, but not for the GIC domain. Thus this is why the work around no longer works.

daveebright · ‎04-01-2016

The following is some additional information which shows the contents of /proc/interrupts for each of the kernel versions. Notice that in the 3.14.52 kernel that the interrupt numbers correspond to the numbers described in the i.MX 6SoloX Applications Processor Reference Manual Chapter 3 Interrupts and DMA Events section 3.2 A.9 Interrupts (in the range from 32 to 159). For the 4.1.15 kernel this is no longer the case. I'm not sure yet how much this matters, but it is quite different. Like I previously said I believe that now for the 4.1.15 interrupts that only the GPC interrupts have the wake-up facility and GIC interrupts do not have the wake-up facility. That is why the previous work around no longer works.

Maybe it would make sense in the gpc.c file to check if CAAM is enabled (if that is possible) before changing/powering off the Mega/Fast Mix?

3.14.52 Kernel Interrupts

cat /proc/interrupts

CPU0

34: 2 GIC 34 sdma

36: 0 GIC 36 20cc000.snvs-pwrkey

38: 3 GIC 38 2224000.lcdif

39: 0 GIC 39 csi

40: 0 GIC 40 pxp-dmaengine

51: 0 GIC 51 rtc alarm

52: 0 GIC 52 snvs-secvio

55: 52 GIC 55 mmc1

56: 2276 GIC 56 mmc2

58: 118 GIC 58 2020000.serial

62: 0 GIC 62 21f4000.serial

67: 115 GIC 67 21f8000.i2c

68: 251 GIC 68 21a0000.i2c

69: 87 GIC 69 21a4000.i2c

70: 86 GIC 70 21a8000.i2c

73: 0 GIC 73 csi

74: 0 GIC 74 2184200.usb

75: 0 GIC 75 2184000.usb

79: 0 GIC 79 202c000.ssi

81: 0 GIC 81 imx_thermal

82: 0 GIC 82 asrc

87: 2713 GIC 87 i.MX Timer Tick

112: 0 GIC 112 20bc000.wdog

122: 0 GIC 122 imx-mu

132: 1 GIC 132 2280000.adc

133: 1 GIC 133 2284000.adc

134: 0 GIC 134 21b4000.ethernet

135: 0 GIC 135 21b4000.ethernet

137: 264 GIC 137 2101000.jr0

138: 3534 GIC 138 2102000.jr1

141: 8 GIC 141 21e4000.qspi

148: 0 GIC 148 imx6sx-sema4

150: 541 GIC 150 2188000.ethernet

151: 0 GIC 151 2188000.ethernet

155: 0 GIC 155 PCIe PME, ath9k

156: 0 GIC 156 220c000.dcic

157: 0 GIC 157 2210000.dcic

177: 0 gpio-mxc 17 headphone detect

178: 0 gpio-mxc 18 Volume Up

179: 0 gpio-mxc 19 Volume Down

202: 0 gpio-mxc 10 2198000.usdhc cd

325: 0 gpio-mxc 5 isl29023, mag3110

334: 0 gpio-mxc 14 abx8xx

IPI0: 0 CPU wakeup interrupts

IPI1: 0 Timer broadcast interrupts

IPI2: 0 Rescheduling interrupts

IPI3: 0 Function call interrupts

IPI4: 0 Single function call interrupts

IPI5: 0 CPU stop interrupts

IPI6: 148 IRQ work interrupts

IPI7: 0 completion interrupts

Err: 0

4.1.15 Kernel Interrupts

cat /proc/interrupts

CPU0

16: 45688372 GPC 55 Level i.MX Timer Tick

20: 0 GPC 20 Level snvs-secvio

22: 533 GPC 26 Level 2020000.serial

23: 0 GPC 47 Level 202c000.ssi

24: 0 GPC 50 Level 2034000.asrc

50: 0 gpio-mxc 17 Edge headphone detect

51: 0 gpio-mxc 18 Edge Volume Up

52: 0 gpio-mxc 19 Edge Volume Down

77: 0 gpio-mxc 10 Edge 2198000.usdhc cd

208: 0 gpio-mxc 5 Edge isl29023, mag3110

217: 0 gpio-mxc 14 Edge rtc-abx80x

269: 0 GPC 80 Level 20bc000.wdog

272: 0 GPC 49 Level imx_thermal

277: 0 GPC 19 Level rtc alarm

278: 2 GPC 4 Level 20cc000.snvs:snvs-powerkey

282: 261922 GPC 2 Level sdma

283: 0 GPC 43 Level 2184000.usb

284: 0 GPC 42 Level 2184200.usb

285: 338974 GPC 118 Level 2188000.ethernet

286: 0 GPC 119 Level 2188000.ethernet

287: 52 GPC 23 Level mmc1

288: 2095779 GPC 24 Level mmc2

289: 468174 GPC 36 Level 21a0000.i2c

290: 109 GPC 37 Level 21a4000.i2c

291: 116 GPC 38 Level 21a8000.i2c

292: 0 GPC 102 Level 21b4000.ethernet

293: 0 GPC 103 Level 21b4000.ethernet

295: 18 GPC 109 Level 21e4000.qspi

296: 2 GPC 30 Level 21f4000.serial

297: 115 GPC 35 Level 21f8000.i2c

298: 0 GPC 7 Level csi

299: 0 GPC 124 Level 220c000.dcic

300: 0 GPC 125 Level 2210000.dcic

301: 0 GPC 8 Level pxp-dmaengine

302: 0 GPC 41 Level csi

303: 6 GPC 6 Level 2224000.lcdif

304: 1 GPC 100 Level 2280000.adc

305: 1 GPC 101 Level 2284000.adc

306: 0 GPC 116 Level imx6sx-sema4

307: 0 GPC 90 Level imx-mu

309: 264 GIC 137 Level 2101000.jr0

310: 52326450 GIC 138 Level 2102000.jr1

311: 3861065 GIC 155 Level PCIe PME, ath9k

IPI0: 0 CPU wakeup interrupts

IPI1: 0 Timer broadcast interrupts

IPI2: 0 Rescheduling interrupts

IPI3: 0 Function call interrupts

IPI4: 0 Single function call interrupts

IPI5: 0 CPU stop interrupts

IPI6: 0 IRQ work interrupts

IPI7: 0 completion interrupts

Err: 0

IMX6SX Kernel 4.1.15 suspend/resume issue

IMX6SX Kernel 4.1.15 suspend/resume issue

i.MX6SoloX

Linux