kernel panic: imx-sdma causes scheduling while atomic

[mikoli]

We have a custom i.MX8M Plus board running NXP Yocto i.MX, Linux, LF5.15.71_2.2.0

We are experiencing intermittent kernel panics that seems to be related to audio playback using ALSA aplay. After studying the call trace it seems that the imx-sdma driver causes scheduling while in atomic context (see below)

soc_pcm_trigger() is within atomic context and sdma_prep_dma_cyclic() allows the power manager to resume and finally locks a mutex causing a scheduling.

This seems very similar to what is described (and solved) here:
linux-yocto@lists.yoctoproject.org | [v6.1/standard/nxp-sdk-6.1/nxp-soc & v6.1/standard/preempt-rt/n...

Its very difficult to reproduce the issue as it only happens twice a week on a live system.

Call trace:

  dump_backtrace+0x0/0x19c

  show_stack+0x18/0x70

  dump_stack_lvl+0x68/0x84

  dump_stack+0x18/0x34

  __schedule_bug+0x60/0x80

  __schedule+0x658/0x710

  schedule+0x88/0x100

  schedule_preempt_disabled+0x24/0x40

  __mutex_lock.constprop.0+0x18c/0x56c

  __mutex_lock_slowpath+0x14/0x20

  mutex_lock+0x48/0x54

  clk_prepare_lock+0x44/0xa0

  clk_unprepare+0x24/0x44

  clk_bulk_unprepare+0x3c/0x60

  genpd_runtime_resume+0x26c/0x2b0

  __rpm_callback+0x48/0x1a0

  rpm_callback+0x6c/0x80

  rpm_resume+0x378/0x614

  __pm_runtime_resume+0x3c/0x8c

  sdma_prep_dma_cyclic+0x58/0x294

  snd_dmaengine_pcm_trigger+0xec/0x1c0

  dmaengine_pcm_trigger+0x18/0x24

  snd_soc_pcm_component_trigger+0x164/0x230

  soc_pcm_trigger+0xbc/0x1c0

[mikoli]

We found a workaround by disabling runtime pm for the imx-sdma driver which strengthens our beliefs that this is an issue with the NXP imx-sdma driver.
This is the workaround:
echo on > /sys/devices/platform/soc@0/30c00000.bus/30e10000.dma-controller/power/control

[Bio_TICFSL]

Got it! will escalated this issue hope it works now thanks.

Regards

[mikoli]

Any update on this?

[Bio_TICFSL]

Hello,

Have you update the kernel to see if this happens? I could not reproduce it with latest BSP. however the case is in hand of the developer if it fixed i will appear fixed until next bsp release.

Regards

[Bio_TICFSL]

Hello,

The SDMA firmware is provided by both meta-freescale and meta-imx layers, the latter provides a newer version of it.
The NXP Yocto user guide IMX Yocto User Guide, suggests to use the layers referenced in the nxp release manifests imx-manifests which include the meta-imx layer as well.
The meta-imx is not strictly necessary for machine support since it should only add new patches which should later be merged into the meta-freescale-distro layer. However, also the fsl-image-machine-test image, which is the basic official image with no GUI or multimedia packages provided by the meta-freescale layer, adds the firmwared daemon package to load the firmware from userspace via udev signaling.

Besides that, I also confirm that after configuring the imx-sdma driver as module , the firmware gets loaded in direct mode immediately after the module gets loaded, in my case it corresponds to about 10 seconds since boot start.

# dmesg | grep sdma
[ 10.459118] imx-sdma 20ec000.sdma: TEST DEBUG: sdma_probe
[ 10.541432] imx-sdma 20ec000.sdma: firmware found.
[ 10.576034] imx-sdma 20ec000.sdma: loaded firmware 3.6

I agree this is a better option, even without the synchronous loading patch, than relying on firmwared daemon because hardly a sdma transfer would take place between the module loading and firmware loading. I am not sure if this applies to all cases, I think it has been opted for a user space loader in the NXP image to support lazy firmware loading for all the possible drivers so that it is not necessary to configure them as module. I hope that usually a safe driver checks whether or not the firmware has been loaded before starting a transfer, or otherwise fails safely without panic.

Regards

[mikoli]

@Bio_TICFSL I've read your reply multiple times and I do not understand how this is connected to my question? Anyway, in my build firmware-imx-8m v8.18 from meta-imx is used.

My issue seems to be somewhere in the region of how the i.MX ALSA SoC Platform driver uses sDMA while its in atomic context. 

Embedded in the log I attached I also see:
tlv320aic3x 2-0018: ASoC: error at snd_soc_component_update_bits on tlv320aic3x.2-0018: -110

tlv320aic3x is our ASoC Codec driver, not sure if this is the cause or the symptom

[Bio_TICFSL]

Hello,

We already reproduced this issue and this is going to appear public a fixed in next Q1 2025 release.

Thanks

[ggpr]

Hello,

I'm encountering the same issue on the iMX8MP running NXP Yocto i.MX, Linux, LF5.10.72_2.2.3 (hardknott).
@Bio_TICFSL , you mentioned that a fix was planned for Q1 2025—has it already been released? Where has it been applied, and how can I make use of it?

Regards,

[ggpr]

Hello,

I'm encountering the same issue on the iMX8MP running NXP Yocto i.MX, Linux, LF5.10.72_2.2.3 (hardknott).
@Bio_TICFSL , you mentioned that a fix was planned for Q1 2025—has it already been released? Where has it been applied, and how can I make use of it?

Regards,

[mikoli]

This is great to hear. As we have a workaround (by disabling PM for imx-sdma) we can manage to wait for the proper solution.