I decided to switch from the linux-rel_imx-3.14.52_ga kernel to the linux-rel_imx-4.1.15_ga kernel for use with the imx6sx sabre board. The 4.1.15 kernel built with no problems and the board boots up properly and runs as expected. However when I tried suspend and resume I ran into problems resuming. Note the suspend to mem (echo mem > /sys/power/state) also did not drop the current usage as low as it did in 3.14.52 either.
The problems looked similar to the problems I was having on the 3.14.52 kernel resuming (kernel would hang, the mmc card would have issues, or whenever I would access my encrypted partition errors would occur and need to reboot) before I did the following as documented in the i.MX Linux Reference Manual:
The full CAAM function is exclusive with the Mega/Fast mix off feature in DSM. If
CAAM is enabled, the Mega/Fast mix off feature needs to be disabled, and the user
should "echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2100000.caam/
2101000.jr0/power/wakeup" after the kernel boots up, and then Mega/Fast mix will
keep the power on in DSM.
With the 3.14.52 kernel everything was fine after I enabled the wakeup as suggested above.
I tried doing the same thing with the 4.1.15 kernel, but it has no impact any more. After some digging I found the following patch which appears to have changed this functionality by Marc Zyngier titled: ARM: imx6: convert GPC to stacked domains. It modifies the device tree interrupt structure to use stacked domains (not sure I totally understand this). Given these modifications I was wondering if anyone has a new workaround for this?
I did debug this and found that in arch/arm/mach-imx/gpc.c function imx_gpc_mf_mix_off() it no longer returns before turning off mega/fast mix in 4.1.15 like it did in 3.14.52 when the wake was enabled. I did hard code it to return before turning off mega/fast mix in 4.1.15 and that appears to fix the issue (current usage in suspend was lower and resumed properly, but not a real solution). Anyway I am going to keep working at this, but again I was wondering if anyone else had already encountered and fixed this problem. It is really noticeable if there is an encrypted file system partition and you attempt to access it after resuming. Also if you do the old 3.14.52 workaround, the resume code in 4.1.15 complains/warns about an "Unbalanced IRQ wake disable".
Also note that when you dump /proc/interrupts in 4.1.15 that the only GIC interrupts are now IRQ 309 2101000.jr0, IRQ 310 2102000.jr1, and IRQ 311 PCIe PME (almost all the rest are now GPC). In 3.14.52 almost all the interrupts were GIC (including those listed above, IRQ 137 2101000.jr0, IRQ 138 2102000.jr1, and IRQ 155 PCIe PME) and none were GPC. Looking at the IRQ masking registers 1-4 it looks like interrupts with IRQ numbers above 159 are not represented (maybe that is why the workaround no longer works because the IRQ is 309).
Carlos,
The following are the patches that work for me while I am waiting for a more official response. These get around the issue I was having and no longer require the workaround of:
echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2100000.caam/2101000.jr0/power/wakeup"
If you only use the device tree patch, then the workaround is still required.
The call imx_gpc_mf_request_on(jrpriv->irq, 1); in jr.c caam_jr_suspend() sets the bit in the gpc_mf_request_on[] in arch/arm/mach-imx/gpc.c so that when imx_gpc_mix_off() is called during the suspend, then this function returns before turning off mega/fast mix (just like it used to in 3.14.52 with the workaround due to the wake IRQ being enabled). In 3.14.52 never saw the "Turn off M/F mix!" message in the system logs after the suspend/resume with the workaround, but in 4.1.15 without my patches you always see this message indicating the mega/fast mix was turned off (thus causing the problem where a reboot is needed to recover). The call to imx_gpc_mf_request_on(jrpriv->irq, 0); in jr.c caam_jr_resume() clears the bit.
Dave
--- linux-rel_imx_4.1.15_1.0.0_ga/arch/arm/boot/dts/imx6sx.dtsi.orig 2016-04-01 14:09:50.000000000 -0400
+++ linux-rel_imx_4.1.15_1.0.0_ga/arch/arm/boot/dts/imx6sx.dtsi 2016-04-01 14:10:34.000000000 -0400
@@ -928,14 +928,14 @@
sec_jr0: jr0@1000 {
compatible = "fsl,sec-v4.0-job-ring";
reg = <0x1000 0x1000>;
- interrupt-parent = <&intc>;
+ interrupt-parent = <&gpc>;
interrupts = <0 105 0x4>;
};
sec_jr1: jr1@2000 {
compatible = "fsl,sec-v4.0-job-ring";
reg = <0x2000 0x1000>;
- interrupt-parent = <&intc>;
+ interrupt-parent = <&gpc>;
interrupts = <0 106 0x4>;
};
};
--- linux-rel_imx_4.1.15_1.0.0_ga/drivers/crypto/caam/jr.c.orig 2016-04-01 12:34:12.000000000 -0400
+++ linux-rel_imx_4.1.15_1.0.0_ga/drivers/crypto/caam/jr.c 2016-04-01 15:09:58.000000000 -0400
@@ -7,6 +7,7 @@
#include <linux/of_irq.h>
#include <linux/of_address.h>
+#include <linux/imx_gpc.h>
#include "compat.h"
#include "regs.h"
@@ -543,6 +544,7 @@ static int caam_jr_suspend(struct device
struct platform_device *pdev = to_platform_device(dev);
struct caam_drv_private_jr *jrpriv = platform_get_drvdata(pdev);
+ imx_gpc_mf_request_on(jrpriv->irq, 1);
if (device_may_wakeup(&pdev->dev))
enable_irq_wake(jrpriv->irq);
@@ -554,6 +556,7 @@ static int caam_jr_resume(struct device
struct platform_device *pdev = to_platform_device(dev);
struct caam_drv_private_jr *jrpriv = platform_get_drvdata(pdev);
+ imx_gpc_mf_request_on(jrpriv->irq, 0);
if (device_may_wakeup(&pdev->dev))
disable_irq_wake(jrpriv->irq);
Hello,
Is there an official patch for this? I'm experiencing a similar problem in the i.MX6UL in kernel v4.1.
The workaround of doing echo enabled > /sys/bus/platform/devices/2100000.aips-bus/2140000.caam/2141000.jr0/power/wakeup solves the issue but Dave's patch doesn't (at least not my problem on the i,MX6UL).
Thanks,
--
Héctor
Hi Dave,
We are internally reviewing your case. In the meantime, you could post any additional information that you could consider useful.
Hope this will be useful for you.
Best regards!
/Carlos
Given that in the device tree (imx6xsx.dtsi) for 4.1.15 the gpc entry still contains the line:
fsl,mf-mix-wakeup-irq = <0x7c00000 0x3d00 0x0 0x400240>;
where this represents all the MEGA domain wake IRQs which includes the CAAM job ring 0 interrupt:
0x400240
^
This bit is for CAAM job ring 0
However now with the 4.1.15 modifications, this interrupt was moved to the GIC domain which has no facility for wakeups. So this really breaks the mf-mix-wakeup-irq functionality. Therefore I suggest modifying the device tree entry minimally for sec_jr0 to change the interrupt-parent = <&intc>; to interrupt-parent = <&gpc>;. Thus now this interrupt will now be back in the GPC domain and the previous workaround will function properly. Could also make this change for sec_jr1, but currently the mf-mix-wakeup-irq mask does not reference it.
While I was looking into this, I also saw in drivers/crypto/caam/jr.c there are suspend and resume functions. It may be a good idea to add calls to imx_gpc_mf_request_on() to these functions which I believe could avoid the need for the workaround.
After studying the modifications some more in 4.1.15 new code was added to translate all of the GPC interrupt values. For instance take for example the rtc alarm:
3.14.52 - 51: 0 GIC 51 rtc alarm
4.1.15 - 277: 0 GPC 19 Level rtc alarm
The GIC has the value 51 and to translate it to GPC simply subtract 32 which gets you to 19. This translation in 4.1.15 is done for all GPC domain interrupts, but not for the GIC domain. Thus this is why the work around no longer works.
The following is some additional information which shows the contents of /proc/interrupts for each of the kernel versions. Notice that in the 3.14.52 kernel that the interrupt numbers correspond to the numbers described in the i.MX 6SoloX Applications Processor Reference Manual Chapter 3 Interrupts and DMA Events section 3.2 A.9 Interrupts (in the range from 32 to 159). For the 4.1.15 kernel this is no longer the case. I'm not sure yet how much this matters, but it is quite different. Like I previously said I believe that now for the 4.1.15 interrupts that only the GPC interrupts have the wake-up facility and GIC interrupts do not have the wake-up facility. That is why the previous work around no longer works.
Maybe it would make sense in the gpc.c file to check if CAAM is enabled (if that is possible) before changing/powering off the Mega/Fast Mix?
3.14.52 Kernel Interrupts
cat /proc/interrupts
CPU0
34: 2 GIC 34 sdma
36: 0 GIC 36 20cc000.snvs-pwrkey
38: 3 GIC 38 2224000.lcdif
39: 0 GIC 39 csi
40: 0 GIC 40 pxp-dmaengine
51: 0 GIC 51 rtc alarm
52: 0 GIC 52 snvs-secvio
55: 52 GIC 55 mmc1
56: 2276 GIC 56 mmc2
58: 118 GIC 58 2020000.serial
62: 0 GIC 62 21f4000.serial
67: 115 GIC 67 21f8000.i2c
68: 251 GIC 68 21a0000.i2c
69: 87 GIC 69 21a4000.i2c
70: 86 GIC 70 21a8000.i2c
73: 0 GIC 73 csi
74: 0 GIC 74 2184200.usb
75: 0 GIC 75 2184000.usb
79: 0 GIC 79 202c000.ssi
81: 0 GIC 81 imx_thermal
82: 0 GIC 82 asrc
87: 2713 GIC 87 i.MX Timer Tick
112: 0 GIC 112 20bc000.wdog
122: 0 GIC 122 imx-mu
132: 1 GIC 132 2280000.adc
133: 1 GIC 133 2284000.adc
134: 0 GIC 134 21b4000.ethernet
135: 0 GIC 135 21b4000.ethernet
137: 264 GIC 137 2101000.jr0
138: 3534 GIC 138 2102000.jr1
141: 8 GIC 141 21e4000.qspi
148: 0 GIC 148 imx6sx-sema4
150: 541 GIC 150 2188000.ethernet
151: 0 GIC 151 2188000.ethernet
155: 0 GIC 155 PCIe PME, ath9k
156: 0 GIC 156 220c000.dcic
157: 0 GIC 157 2210000.dcic
177: 0 gpio-mxc 17 headphone detect
178: 0 gpio-mxc 18 Volume Up
179: 0 gpio-mxc 19 Volume Down
202: 0 gpio-mxc 10 2198000.usdhc cd
325: 0 gpio-mxc 5 isl29023, mag3110
334: 0 gpio-mxc 14 abx8xx
IPI0: 0 CPU wakeup interrupts
IPI1: 0 Timer broadcast interrupts
IPI2: 0 Rescheduling interrupts
IPI3: 0 Function call interrupts
IPI4: 0 Single function call interrupts
IPI5: 0 CPU stop interrupts
IPI6: 148 IRQ work interrupts
IPI7: 0 completion interrupts
Err: 0
4.1.15 Kernel Interrupts
cat /proc/interrupts
CPU0
16: 45688372 GPC 55 Level i.MX Timer Tick
20: 0 GPC 20 Level snvs-secvio
22: 533 GPC 26 Level 2020000.serial
23: 0 GPC 47 Level 202c000.ssi
24: 0 GPC 50 Level 2034000.asrc
50: 0 gpio-mxc 17 Edge headphone detect
51: 0 gpio-mxc 18 Edge Volume Up
52: 0 gpio-mxc 19 Edge Volume Down
77: 0 gpio-mxc 10 Edge 2198000.usdhc cd
208: 0 gpio-mxc 5 Edge isl29023, mag3110
217: 0 gpio-mxc 14 Edge rtc-abx80x
269: 0 GPC 80 Level 20bc000.wdog
272: 0 GPC 49 Level imx_thermal
277: 0 GPC 19 Level rtc alarm
278: 2 GPC 4 Level 20cc000.snvs:snvs-powerkey
282: 261922 GPC 2 Level sdma
283: 0 GPC 43 Level 2184000.usb
284: 0 GPC 42 Level 2184200.usb
285: 338974 GPC 118 Level 2188000.ethernet
286: 0 GPC 119 Level 2188000.ethernet
287: 52 GPC 23 Level mmc1
288: 2095779 GPC 24 Level mmc2
289: 468174 GPC 36 Level 21a0000.i2c
290: 109 GPC 37 Level 21a4000.i2c
291: 116 GPC 38 Level 21a8000.i2c
292: 0 GPC 102 Level 21b4000.ethernet
293: 0 GPC 103 Level 21b4000.ethernet
295: 18 GPC 109 Level 21e4000.qspi
296: 2 GPC 30 Level 21f4000.serial
297: 115 GPC 35 Level 21f8000.i2c
298: 0 GPC 7 Level csi
299: 0 GPC 124 Level 220c000.dcic
300: 0 GPC 125 Level 2210000.dcic
301: 0 GPC 8 Level pxp-dmaengine
302: 0 GPC 41 Level csi
303: 6 GPC 6 Level 2224000.lcdif
304: 1 GPC 100 Level 2280000.adc
305: 1 GPC 101 Level 2284000.adc
306: 0 GPC 116 Level imx6sx-sema4
307: 0 GPC 90 Level imx-mu
309: 264 GIC 137 Level 2101000.jr0
310: 52326450 GIC 138 Level 2102000.jr1
311: 3861065 GIC 155 Level PCIe PME, ath9k
IPI0: 0 CPU wakeup interrupts
IPI1: 0 Timer broadcast interrupts
IPI2: 0 Rescheduling interrupts
IPI3: 0 Function call interrupts
IPI4: 0 Single function call interrupts
IPI5: 0 CPU stop interrupts
IPI6: 0 IRQ work interrupts
IPI7: 0 completion interrupts
Err: 0