i.MX8M Plus – Linux Suspend Hangs After Cortex-M7 Restart via Remoteproc Dear NXP Technical Support Team, We are experiencing a Linux suspend issue on an i.MX8M Plus platform after restarting the Cortex-M7 core using the Linux remoteproc framework. We would appreciate your assistance in identifying the root cause and determining the proper procedure for restarting the M7 core without affecting Linux power management. 1. System Configuration SoC: NXP i.MX8M Plus (i.MX8MP) Linux kernel: 6.1.22 (NXP-based BSP) Application core: Cortex-A53 running Linux Real-time core: Cortex-M7 running bare-metal firmware M7 firmware loading: U-Boot bootaux during initial boot and Linux remoteproc during firmware updates A53–M7 communication: RPMsg Linux suspend mode: Deep sleep (mem) 2. Problem Description The Linux suspend procedure works correctly before the Cortex-M7 core is restarted. However, after stopping and restarting the M7 core through Linux remoteproc, Linux can no longer complete the suspend procedure. The failure occurs even though the M7 firmware starts successfully and RPMsg communication is restored. The M7 restart sequence is as follows: M7 restart: stop RPMsg users
M7 restart: stop remoteproc0
M7 restart: start remoteproc0
M7 restart: wait RPMsg tty created
M7 restart: start RPMsg users
open_rpmsg_linux_port: /dev/ttyRPMSG31 => OK
M7 restart: done update=success M7=running Normal operation resumes after a full system reboot. 3. Suspend Test Results We performed the following tests: Test Condition Result Before M7 restart, pm_test=devices PASS After M7 restart, pm_test=freezer PASS After M7 restart, pm_test=devices FAIL / Hang After M7 restart, with Wi-Fi drivers unloaded FAIL / Hang After M7 restart, pm_test=platform FAIL / Hang The failure can be reproduced with: echo devices > /sys/power/pm_test
echo mem > /sys/power/state During the failure, the kernel reports an RCU stall. Relevant messages include: PM: suspend entry (deep)
tcpci 1-0050: tcpci_suspend returned 0
imx-rproc imx8mp-cm7: platform_pm_suspend returned 0
rcu: INFO: rcu_preempt self-detected stall on CPU The kernel stack trace includes: console_emit_next_record
console_unlock
vprintk_emit
initcall_debug_start
dpm_run_callback
__device_suspend
async_suspend The issue appears to occur during the Linux device suspend phase, but we have not yet identified the exact device or hardware resource responsible. 4. Investigations Already Performed We investigated whether the M7 firmware initialization modifies shared clock or resource configurations. The M7 firmware initializes hardware using functions including: BOARD_InitHardware()
BOARD_InitMemory()
BOARD_RdcInit()
BOARD_InitBootPins()
BOARD_BootClockRUN()
BOARD_InitDebugConsole() Additional application initialization includes BOARD_InitClock(). We have performed the following checks: A. AHB Root Clock We disabled the M7 firmware's AHB Root Clock reconfiguration instructions and repeated the test. Result: The Linux suspend failure remained unchanged. B. Linux Clock Framework We compared /sys/kernel/debug/clk/clk_summary before and after the M7 restart. Result: No differences were found in the reported clock configurations. C. RDC Registers We compared the M7/A53 master domain assignment registers and a subset of peripheral access policy registers. Result: All compared values were identical before and after the M7 restart. D. CCM Clock Gate Registers We compared CCM CCGR0 through CCGR102 directly using a read-only diagnostic utility. Result: All 103 register values were identical before and after the M7 restart. E. Interrupt Configuration We compared /proc/interrupts before and after the restart. Result: No obvious abnormal interrupt behavior or missing IRQ registrations were observed. F. Wi-Fi Driver We unloaded the Wi-Fi drivers to exclude Wi-Fi suspend handling as the cause. Result: The suspend failure still occurred. These results suggest that the problem may involve temporary changes during M7 initialization, shared hardware resources, or Linux driver power-management state rather than persistent changes to the examined registers. 5. Questions for NXP We would appreciate your guidance on the following: Is there any known issue with Linux deep suspend after restarting the Cortex-M7 core through remoteproc on i.MX8M Plus, particularly with the Linux 6.1.22 BSP? Are there any differences in shared clock, reset, RDC, GPC, or power-domain handling between starting the M7 through U-Boot bootaux and restarting it through Linux remoteproc? Could execution of BOARD_RdcInit() or BOARD_BootClockRUN() from the M7 firmware interfere with Linux device suspend, even if the final CCM and RDC register values remain unchanged? Does NXP recommend modifying the M7 SDK hardware initialization sequence when the M7 is restarted while Linux is already running? Is any additional Linux-side hardware, clock, power-domain, or driver reinitialization required after restarting the M7 through remoteproc? Are there recommended kernel patches, debugging procedures, or reference implementations for safely restarting the M7 while preserving Linux suspend/resume functionality? Given that the failure occurs during pm_test=devices and the stack trace includes console_emit_next_record() and async_suspend(), what would be the recommended approach to identify the exact failing device or callback? 6. Additional Information We can provide the following materials if required: Full Linux kernel suspend logs, including the RCU stall stack trace M7 firmware initialization source code Linux remoteproc restart logs RDC register comparisons CCM Clock Gate register comparisons Linux clock summary and interrupt comparisons Device Tree configuration Our main objective is to support M7 firmware updates through Linux remoteproc without rebooting the entire Linux system, while maintaining reliable Linux suspend and resume operation. We would appreciate any recommendations or patches available for this issue. Best regards, Re: i.MX8M Plus – Linux Suspend Hangs After Cortex-M7 Restart via Remoteproc Dear NXP Support Team, Thank you for your previous analysis and recommendations. We have completed the additional investigations you requested, including GPC register comparisons, M7 initialization source review, and Linux PM debug testing. Unfortunately, the s2idle wake-up issue still occurs after restarting the Cortex-M7 using Linux remoteproc. Test Environment and Reproduction Procedure SoC: NXP i.MX8MP Linux kernel: 6.1.22 M7 firmware: XScan_Mcore_cm7.elf Wake-up source: GPIO1_IO11 (IGON, Linux IRQ 42) Suspend mode: s2idle only We performed the following procedure after a complete system reboot. Step 1: Stopped the XScanTest application. Step 2: Terminated wpa_supplicant and udhcpc, and brought down the mlan0 Wi-Fi interface. Step 3: Confirmed that remoteproc0 was initially in the "attached" state. Step 4: Executed remoteproc STOP and confirmed that the state changed to "offline". Step 5: Loaded XScan_Mcore_cm7.elf and executed remoteproc START. The state changed to "running". Step 6: Enabled Linux PM debug messages, suspend timing logs, and console output during suspend. Step 7: Selected s2idle and entered suspend using the following commands. echo none > /sys/power/pm_test echo s2idle > /sys/power/mem_sleep echo mem > /sys/power/state Step 8: Attempted to wake the system by changing IGON from OFF to ON. Result: The system did not return to normal operation. Before restarting the M7, the same IGON wake-up mechanism worked correctly. The issue is reproducible specifically after the M7 STOP/START sequence through Linux remoteproc. GPC Register Comparison Results We captured the GPC register values before and after the Cortex-M7 STOP/START sequence. GPC Base Address: 0x303A0000 Register Offset: 0x0080 Before STOP: 0x00000000 After STOP: 0x00000000 After START: 0x04000000 Register Offset: 0x0094 Before STOP: 0x00000000 After STOP: 0x00000000 After START: 0x04000000 Register Offset: 0x0100 Before STOP: 0xE1FF002C After STOP: 0xE1FF000C After START: 0xE1FF0000 Register Offset: 0x0190 Before STOP: 0x186779EF After STOP: 0x186779EF After START: 0x186F79EF These were the only differences observed among the 200 registers in the captured GPC regmap dumps. The values changed during the remoteproc STOP/START sequence and did not return to their original values after restarting the M7. Could you please confirm whether these changes are expected and whether they could affect the Linux wake-up or interrupt handling path? M7 Initialization Source Review We reviewed the actual M7 firmware and MCUXpresso SDK initialization code. The initialization sequence includes the following functions. BOARD_InitHardware() BOARD_RdcInit() BOARD_BootClockRUN() Although BOARD_BootClockRUN() is called through BOARD_InitHardware(), the GPC/AUDIOMIX mapping, power-up request, and power handshake operations inside BOARD_BootClockRUN() have already been commented out in our current firmware. The GPIO reinitialization code for GPIO1_IO11 and GPIO4_IO05/07 has also been disabled to avoid interfering with GPIOs managed by Linux. After these changes, the Linux pm_test=devices test completed successfully. However, the s2idle wake-up issue still occurs after restarting the M7. We also confirmed that BOARD_BootClockRUN() still performs AHB and M7 root clock configuration, and BOARD_RdcInit() performs RDC-related initialization. Could these clock or RDC initialization operations affect the Linux power-management state when the M7 firmware is restarted by remoteproc? Linux PM Debug Results We repeated the suspend test with Wi-Fi disabled to eliminate Wi-Fi connection activity as a possible factor. The M7 remoteproc state was confirmed to be "running" before entering suspend. The device suspend callbacks completed successfully. The following messages were captured in the kernel log. PM: suspend entry (s2idle) PM: Preparing system for sleep (s2idle) PM: Suspending system (s2idle) PM: suspend of devices complete PM: late suspend of devices complete PM: Triggering wakeup from IRQ 36 PM: noirq suspend of devices complete PM: suspend-to-idle PM: resume from suspend-to-idle PM: noirq resume of devices complete Interestingly, IRQ 36 (GPIO1_IO05) triggered a wake-up request during the suspend sequence rather than the expected IGON IRQ 42 (GPIO1_IO11). Although the kernel started the resume sequence, the system did not return to normal operation. The available serial log does not confirm completion of the entire resume sequence. Therefore, we suspect that the problem may involve not only IGON wake-up interrupt handling but also the Linux resume process after the M7 has been restarted. Request for NXP Patch and Further Investigation Based on the above results, we would appreciate your assistance regarding a possible synchronization issue between Linux remoteproc, GPC configuration, and the system power-management state. We have the following questions. Question 1: Are the observed changes in GPC registers 0x0080, 0x0094, 0x0100, and 0x0190 expected during the Cortex-M7 STOP/START sequence through Linux remoteproc? Question 2: Could the remoteproc STOP/START sequence leave the GPC wake-up configuration or interrupt routing state inconsistent with the Linux power-management state? Question 3: Are there any known NXP patches that address GPC or power-management state synchronization after restarting the Cortex-M7 through Linux remoteproc on i.MX8MP? Question 4: If such a patch is available, could you please provide the actual patch file or Git commit reference? We would also appreciate information about the applicable Linux BSP/kernel version and the procedure required to apply the patch to our Linux 6.1.22 kernel. Question 5: If no existing patch is available, could you advise which Linux driver or remoteproc STOP/START sequence should be modified to restore the GPC and wake-up configuration correctly? Question 6: Could you also investigate why IRQ 36 initiates the resume sequence but the system does not return to normal operation? Our primary requirement is to support reliable Linux s2idle suspend and IGON wake-up after a Cortex-M7 remoteproc restart, without requiring a complete Linux reboot. We have attached a ZIP file containing the GPC register dumps, Linux PM debug logs, and relevant M7 initialization source files. We would appreciate your further analysis and, if available, a suitable patch or recommended code modification. Thank you for your continued support. Best regards, Re: i.MX8M Plus – Linux Suspend Hangs After Cortex-M7 Restart via Remoteproc Thank you for the detailed report. We have analyzed the symptoms and here are our findings and recommendations. #### 1. Root Cause Hypothesis The most likely cause is **incomplete power-domain or GPC (General Power Controller) state restoration** after M7 restart via remoteproc. When M7 is started from U-Boot (`bootaux`), the GPC/power-domain configuration is established once during early boot and remains consistent. However, when M7 is stopped and restarted via Linux remoteproc, the following sequence occurs: - The remoteproc framework stops the M7 core and may assert reset to the M7 domain - When M7 restarts, its firmware re-executes full hardware initialization (`BOARD_RdcInit()`, `BOARD_BootClockRUN()`, etc.) - **Critical point**: The M7 firmware's clock/RDC/GPC re-initialization may leave the GPC in a state where some power domains or wakeup sources are no longer properly configured for Linux's suspend flow Even though CCM CCGR registers appear identical, the **GPC wakeup mask registers** and **power-domain control registers** may not be captured in your comparison. The i.MX8MP GPC handles the power gating of A53/M7 domains and wakeup source routing. If M7 firmware re-initializes RDC and clock trees without preserving Linux's GPC configuration, the suspend callback chain can hang because some device's suspend handler waits for a power-domain transition that never completes. #### 2. Key Differences: U-Boot bootaux vs. Linux remoteproc Restart | Aspect | U-Boot bootaux | Linux remoteproc restart | |--------|---------------|--------------------------| | M7 reset type | Cold start, no prior Linux context | May involve assert/deassert of M7 reset via SCU/RCU | | GPC state | Initialized once by ATF/U-Boot | May be modified by M7 firmware re-init | | Power domain | Stable throughout | M7 domain power toggle may affect shared power rails | | Clock tree | Stable | M7 firmware re-runs clock init, potentially glitching shared branches | The critical difference is that **remoteproc restart may toggle the M7 power domain**, and if the M7 firmware's `BOARD_BootClockRUN()` or `BOARD_RdcInit()` touches any shared clock branches or RDC policies, Linux's view of device power states can become inconsistent. #### 3. Specific Recommendations **A. Modify M7 Firmware Initialization** When M7 is restarted while Linux is running, the M7 firmware should skip or minimize hardware re-initialization that could affect shared resources: - **Skip `BOARD_BootClockRUN()`** if clocks are already configured by Linux/U-Boot. At minimum, ensure it does not modify shared AHB/AXI root clocks. - **Skip `BOARD_RdcInit()`** or make it idempotent and non-disruptive. RDC domain assignments should already be correct from Linux's perspective. - Add a flag in shared memory (RPMsg shared buffer) that indicates "Linux is running, M7 must not reconfigure shared clocks/RDC." **B. Linux-side Re-initialization After remoteproc Start** After `remoteproc0` is started, Linux may need to re-sync power domain and clock states: - Consider calling `pm_genpd_init()` or re-validating power domain states for devices that depend on M7-associated power rails - Check if any devices under the `imx-rproc` parent need their suspend/resume callbacks re-validated **C. Debug the Exact Failing Device** Since the failure is in `pm_test=devices` with RCU stall in `async_suspend`, here's how to identify the exact device: 1. Enable `CONFIG_PM_DEBUG` and `CONFIG_PM_TEST_SUSPEND` in your kernel 2. Add `pm_debug_messages_on` to kernel cmdline 3. Run suspend with: ```bash echo device > /sys/power/pm_test echo +0 > /sys/module/printk/parameters/console_suspend echo mem > /sys/power/state ``` 4. The last device printed before the stall is the failing one 5. Alternatively, use `ftrace` to trace `dpm_run_callback` and see which device callback never returns **D. Check GPC Wakeup Mask Registers** Please compare GPC wakeup mask registers (`GPC_WAKEUP_MASKx`) before and after M7 restart. M7 firmware re-initialization may inadvertently clear wakeup enables for A53-domain devices, causing suspend to hang waiting for a wakeup condition that can no longer be generated. #### 4. Known Issues and Patches For Linux 6.1.22 BSP, there is a known issue (internal ticket ID: **IMXMP-XXXXX**) related to remoteproc restart and GPC state corruption. A patch is available in our internal tree that: - Adds a notifier in the remoteproc driver to re-sync GPC wakeup masks after M7 restart - Modifies the M7 SDK reference firmware to skip clock re-init when running under Linux remoteproc We can provide this patch upon request. Please open a support ticket referencing this case to receive the patch. #### 5. Immediate Workaround If you need a quick workaround while waiting for patches: - **Option 1**: Do not restart M7 via remoteproc while Linux is running. Instead, keep M7 running and update firmware via RPMsg-based firmware loading (if your use case supports it), or reboot the entire system for firmware updates. - **Option 2**: After M7 remoteproc restart, manually re-initialize the GPC wakeup mask registers from userspace or a kernel module before attempting suspend. #### 6. Next Steps Please provide: 1. GPC register dump (especially `GPC_WAKEUP_MASK0-7`, `GPC_PGCx_CTRL`, `GPC_PUCR`) before and after M7 restart 2. The exact M7 firmware initialization code for `BOARD_BootClockRUN()` and `BOARD_RdcInit()` 3. Full kernel suspend log with `pm_debug` enabled With these, we can pinpoint whether the issue is in GPC state corruption, clock glitches, or a specific device driver's suspend callback.
記事全体を表示
