topic Re: kernel panic: imx-sdma causes scheduling while atomic in i.MX Processors

kernel panic: imx-sdma causes scheduling while atomic

mikoli — Tue, 10 Sep 2024 08:46:55 GMT

We have a custom i.MX8M Plus board running NXP Yocto i.MX, Linux, LF5.15.71_2.2.0

We are experiencing intermittent kernel panics that seems to be related to audio playback using ALSA aplay. After studying the call trace it seems that the imx-sdma driver causes scheduling while in atomic context (see below)

soc_pcm_trigger() is within atomic context and sdma_prep_dma_cyclic() allows the power manager to resume and finally locks a mutex causing a scheduling.

This seems very similar to what is described (and solved) here:
linux-yocto@lists.yoctoproject.org | [v6.1/standard/nxp-sdk-6.1/nxp-soc & v6.1/standard/preempt-rt/nxp-sdk-6.1/nxp-soc][PATCH] dmaengine: imx-sdma: Make runtime PM irq safe

Its very difficult to reproduce the issue as it only happens twice a week on a live system.

Call trace:

dump_backtrace+0x0/0x19c

show_stack+0x18/0x70

dump_stack_lvl+0x68/0x84

dump_stack+0x18/0x34

__schedule_bug+0x60/0x80

__schedule+0x658/0x710

schedule+0x88/0x100

schedule_preempt_disabled+0x24/0x40

__mutex_lock.constprop.0+0x18c/0x56c

__mutex_lock_slowpath+0x14/0x20

mutex_lock+0x48/0x54

clk_prepare_lock+0x44/0xa0

clk_unprepare+0x24/0x44

clk_bulk_unprepare+0x3c/0x60

genpd_runtime_resume+0x26c/0x2b0

__rpm_callback+0x48/0x1a0

rpm_callback+0x6c/0x80

rpm_resume+0x378/0x614

__pm_runtime_resume+0x3c/0x8c

sdma_prep_dma_cyclic+0x58/0x294

snd_dmaengine_pcm_trigger+0xec/0x1c0

dmaengine_pcm_trigger+0x18/0x24

snd_soc_pcm_component_trigger+0x164/0x230

soc_pcm_trigger+0xbc/0x1c0

Re: kernel panic: imx-sdma causes scheduling while atomic

Bio_TICFSL — Tue, 10 Sep 2024 14:25:03 GMT

Hello,

The SDMA firmware is provided by both meta-freescale and meta-imx layers, the latter provides a newer version of it.
The NXP Yocto user guide IMX Yocto User Guide, suggests to use the layers referenced in the nxp release manifests imx-manifests which include the meta-imx layer as well.
The meta-imx is not strictly necessary for machine support since it should only add new patches which should later be merged into the meta-freescale-distro layer. However, also the fsl-image-machine-test image, which is the basic official image with no GUI or multimedia packages provided by the meta-freescale layer, adds the firmwared daemon package to load the firmware from userspace via udev signaling.

Besides that, I also confirm that after configuring the imx-sdma driver as module , the firmware gets loaded in direct mode immediately after the module gets loaded, in my case it corresponds to about 10 seconds since boot start.

# dmesg | grep sdma
[ 10.459118] imx-sdma 20ec000.sdma: TEST DEBUG: sdma_probe
[ 10.541432] imx-sdma 20ec000.sdma: firmware found.
[ 10.576034] imx-sdma 20ec000.sdma: loaded firmware 3.6

I agree this is a better option, even without the synchronous loading patch, than relying on firmwared daemon because hardly a sdma transfer would take place between the module loading and firmware loading. I am not sure if this applies to all cases, I think it has been opted for a user space loader in the NXP image to support lazy firmware loading for all the possible drivers so that it is not necessary to configure them as module. I hope that usually a safe driver checks whether or not the firmware has been loaded before starting a transfer, or otherwise fails safely without panic.

Regards

Re: kernel panic: imx-sdma causes scheduling while atomic

mikoli — Wed, 11 Sep 2024 11:12:48 GMT

@Bio_TICFSL I've read your reply multiple times and I do not understand how this is connected to my question? Anyway, in my build firmware-imx-8m v8.18 from meta-imx is used.

My issue seems to be somewhere in the region of how the i.MX ALSA SoC Platform driver uses sDMA while its in atomic context.

Embedded in the log I attached I also see:
tlv320aic3x 2-0018: ASoC: error at snd_soc_component_update_bits on tlv320aic3x.2-0018: -110

tlv320aic3x is our ASoC Codec driver, not sure if this is the cause or the symptom

Re: kernel panic: imx-sdma causes scheduling while atomic

mikoli — Wed, 18 Sep 2024 10:03:16 GMT

We found a workaround by disabling runtime pm for the imx-sdma driver which strengthens our beliefs that this is an issue with the NXP imx-sdma driver.
This is the workaround:
echo on > /sys/devices/platform/soc@0/30c00000.bus/30e10000.dma-controller/power/control

Re: kernel panic: imx-sdma causes scheduling while atomic

Bio_TICFSL — Wed, 18 Sep 2024 13:52:21 GMT

Got it! will escalated this issue hope it works now thanks.

Regards

Re: kernel panic: imx-sdma causes scheduling while atomic

mikoli — Wed, 16 Oct 2024 07:16:57 GMT

Any update on this?

Re: kernel panic: imx-sdma causes scheduling while atomic

Bio_TICFSL — Thu, 17 Oct 2024 12:35:45 GMT

Hello,

Have you update the kernel to see if this happens? I could not reproduce it with latest BSP. however the case is in hand of the developer if it fixed i will appear fixed until next bsp release.

Regards

Re: kernel panic: imx-sdma causes scheduling while atomic

Bio_TICFSL — Wed, 06 Nov 2024 14:35:32 GMT

Hello,

We already reproduced this issue and this is going to appear public a fixed in next Q1 2025 release.

Thanks

Re: kernel panic: imx-sdma causes scheduling while atomic

mikoli — Wed, 06 Nov 2024 14:57:33 GMT

This is great to hear. As we have a workaround (by disabling PM for imx-sdma) we can manage to wait for the proper solution.

Re: kernel panic: imx-sdma causes scheduling while atomic

ggpr — Thu, 19 Jun 2025 08:03:36 GMT

Hello,

I'm encountering the same issue on the iMX8MP running NXP Yocto i.MX, Linux, LF5.10.72_2.2.3 (hardknott).
@Bio_TICFSL , you mentioned that a fix was planned for Q1 2025—has it already been released? Where has it been applied, and how can I make use of it?

Regards,

Re: kernel panic: imx-sdma causes scheduling while atomic

ggpr — Thu, 19 Jun 2025 08:06:27 GMT

Hello,

Regards,

Re: kernel panic: imx-sdma causes scheduling while atomic

user123412 — Wed, 18 Mar 2026 21:10:05 GMT

I know this thread is old, but has this issue been addressed in a patch? I am seeing the same kernel panic periodically on linux-imx 5.10