[imx8mp] cannot wake up from suspend if message sent to serial too early

[martinetd]

Hello,

Sometimes our imx8mp board does not wake up from suspend in regression testing.

It looks like when a message is sent to tty too early, the driver falls in a bad state and wake up from tty no longer works (wake up from another external source like gpio still works in this state, so suspend succeeded, only the wake up by tty stopped working)

This can be reproduced as follow, I've confirmed I can reproduce it on imx8mp-evk with multiple kernels from 5.10.72-2.2.2 to 6.6.3-1.0.0 so this doesn't look fixed.

root@imx8mpevk:~# echo enabled > /sys/class/tty/ttymxc1/power/wakeup
root@imx8mpevk:~# while true; do echo mem > /sys/power/state; done
(at this point, keep a key pressed -- this does not reproduce 100% of the time but after a dozen of suspend/resume the system will stop waking up; usually takes less than 30s. Logs below with the 'f' key pressed)
[...]

f[   39.292333] imx-dwmac 30bf0000.ethernet eth1: No Safety Features support found
[   39.299579] imx-dwmac 30bf0000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
[   39.307965] imx-dwmac 30bf0000.ethernet eth1: FPE workqueue start
f[   39.314271] xhci-hcd xhci-hcd.1.auto: xHC error in resume, USBSTS 0x401, Reinit
[   39.314504] caam 30900000.crypto: registering rng-caam
[   39.321582] usb usb1: root hub lost power or was reset
[   39.321586] usb usb2: root hub lost power or was reset
ff[   39.429609] PM: resume devices took 0.360 seconds
[   39.434367] OOM killer enabled.
f[   39.437515] Restarting tasks ... done.
[   39.442550] random: crng reseeded on system resumption
[   39.447806] PM: suspend exit
[   39.450892] PM: suspend entry (deep)
ff[   39.523722] Filesystems sync: 0.069 seconds
[   39.528483] Freezing user space processes
[   39.532967] Freezing user space processes completed (elapsed 0.004 seconds)
[   39.539965] OOM killer disabled.
[   39.543210] Freezing remaining freezable tasks
ff[   39.604257] Freezing remaining freezable tasks completed (elapsed 0.056 seconds)
ff[   39.681364] imx-dwmac 30bf0000.ethernet eth1: FPE workqueue stop
[   39.694694] PM: suspend devices took 0.084 seconds
[   39.702612] Disabling non-boot CPUs ...
[   39.708044] psci: CPU1 killed (polled 0 ms)
[   39.713999] psci: CPU2 killed (polled 0 ms)
[   39.718424] Wakeup pending. Abort CPU freeze
[   39.722733] Non-boot CPUs are not disabled
[   39.726841] Enabling non-boot CPUs ...
[   39.730964] Detected VIPT I-cache on CPU1
[   39.730991] GICv3: CPU1: found redistributor 1 region 0:0x00000000388a0000
[   39.731020] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[   39.731491] CPU1 is up
[   39.751641] Detected VIPT I-cache on CPU2
[   39.751662] GICv3: CPU2: found redistributor 2 region 0:0x00000000388c0000
[   39.751683] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[   39.752156] CPU2 is up
ffffff[   39.952174] imx-dwmac 30bf0000.ethernet eth1: configuring for phy/rgmii-id link mode
f[   39.995983] imx-dwmac 30bf0000.ethernet eth1: No Safety Features support found
f[   40.003231] imx-dwmac 30bf0000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
[   40.012256] imx-dwmac 30bf0000.ethernet eth1: FPE workqueue start
[   40.018553] xhci-hcd xhci-hcd.1.auto: xHC error in resume, USBSTS 0x401, Reinit
[   40.018705] caam 30900000.crypto: registering rng-caam
[   40.025864] usb usb1: root hub lost power or was reset
[   40.025867] usb usb2: root hub lost power or was reset
fff[   40.132990] PM: resume devices took 0.360 seconds
[   40.137748] OOM killer enabled.
[   40.140896] Restarting tasks ... done.
[   40.145896] random: crng reseeded on system resumption
[   40.151162] PM: suspend exit
-sh: echo: write error: Device or[   40.154279] PM: suspend entry (deep)
 resource busy
ff[   40.225654] Filesystems sync: 0.064 seconds
[   40.230469] Freezing user space processes
[   40.235950] Freezing user space processes completed (elapsed 0.001 seconds)
f[   40.242979] OOM killer disabled.
[   40.246544] Freezing remaining freezable tasks
f[   40.295917] Freezing remaining freezable tasks completed (elapsed 0.044 seconds)
fff[   40.422151] imx-dwmac 30bf0000.ethernet eth1: FPE workqueue stop
[   40.435887] PM: suspend devices took 0.136 seconds
[   40.443055] Disabling non-boot CPUs ...
[   40.447724] psci: CPU1 killed (polled 4 ms)
[   40.453729] psci: CPU2 killed (polled 0 ms)
[   40.459617] psci: CPU3 killed (polled 0 ms)
(hung here, no more feedback from tty)

The "keep key pressed" example is quite silly, but it's possible that the system decided to sleep after a timeout and at that precise timing an external wakeup event comes and gets stuck (this is what happens in our test), so it'd be great to fix this.

It's likely that a key event comes between enabling wakeup in drivers/tty/serial/imx.c's imx_uart_suspend() and the actual suspend and the state machine gets somehow incorrectly stuck; perhaps not disabling irq and catching these (calling pm_system_wakeup() from the irq if one came after suspend started) would work better?

Thanks.

[Bio_TICFSL]

Hello,

Something is bad with the current driver, I will report immediately thanks for the catch. could you telll me what pins are you using?

Regards

[martinetd]

Hi @Bio_TICFSL

Have you heard back from the developers?
Would be happy to test a workaround or anything you could provide.

(The linux-imx lf-6.6.23-2.0.0 tag was pushed last week and I've quickly checked the content but there doesn't seem to have been a fix this fast -- I'm not in a very big hurry here, but our platform test failed again on this problem earlier today so I got curious and checked)

Thanks,
Dominique

[Bio_TICFSL]

Hello,

The issue is not reproducible with latest kernel, please provide a test code to check it.

Regards

[martinetd]

Thanks for the quick reply!

We're using the same pins as the imx8mp-evk, MX8MP_IOMUXC_UART2_RXD__UART2_DCE_RX / MX8MP_IOMUXC_UART2_TXD__UART2_DCE_TX

Thanks,
Dominique