dmaengine imx-sdma fails to start transfer to audio peripheral (descriptor allocation fails)

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

dmaengine imx-sdma fails to start transfer to audio peripheral (descriptor allocation fails)

1,667 Views
dbrig
Contributor II

Hello

We are designing a product using an imx8mm SoM and encountered an issue when starting audio playback. Sometimes trying to play will result in:

# speaker-test
[...]
0 - Front Left
Write error: -5,Input/output error
xrun_recovery failed: -5,Input/output error
Transfer failed: Input/output error
# aplay test.wav
[...]
aplay: pcm_write:2011: write error: Cannot allocate memory

Sometimes it occurs immediately after boot, sometimes after a while. After the first failure all attempts to playback audio fail with the same error until reboot. We also found this issue to occur with multiple kernel versions: we tried 4.14.98, 5.4.24, 5.4.47 from https://source.codeaurora.org/external/imx/linux-imx

I tracked the error down to sdma_alloc_bd(). dma_alloc_coherent() fails to allocate descriptors here (bd_size is 48 bytes at the time of the call):

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/dma/imx-sdma.c?h=imx_5.4.47_2.2.0#...

Using a kernel built with CMA, DMA and dmaengine debugging enabled we do not see errors/warnings; dma_alloc_coherent() simply returns a failure even if the CMA pool and dam-api debug information indicate that there is plenty of pages to allocate from.

[ 0.000000][ T0] Reserved memory: created CMA memory pool at 0x000000009c000000, size 64 MiB
[ 0.000000][ T0] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
# cat dma-api/num_free_entries
63902
# cat dma-api/min_free_entries
63202
# cat dma-api/nr_total_entries
65536
# cat dma-api/dump | grep sdma
302c0000.dma-controller imx-sdma coherent idx 1 P=9c003000 N=9c003 D=9c003000 L=1000 DMA_BIDIRECTIONAL dma map error check not applicable
302b0000.dma-controller imx-sdma coherent idx 2 P=9c004000 N=9c004 D=9c004000 L=1000 DMA_BIDIRECTIONAL dma map error check not applicable
30bd0000.dma-controller imx-sdma coherent idx 2 P=9c005000 N=9c005 D=9c005000 L=1000 DMA_BIDIRECTIONAL dma map error check not applicable
302c0000.dma-controller imx-sdma coherent idx 112 P=9e0e0000 N=9e0e0 D=9e0e0000 L=1200 DMA_BIDIRECTIONAL dma map error check not applicable
302b0000.dma-controller imx-sdma coherent idx 113 P=9e0e2000 N=9e0e2 D=9e0e2000 L=1200 DMA_BIDIRECTIONAL dma map error check not applicable
30bd0000.dma-controller imx-sdma coherent idx 114 P=9e0e4000 N=9e0e4 D=9e0e4000 L=1200 DMA_BIDIRECTIONAL dma map error check not applicable
# cat cma/cma-linux,cma/bitmap
4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 4294967295 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
# cat cma/cma-linux,cma/used
8422
# cat cma/cma-linux,cma/order_per_bit
0
# cat cma/cma-linux,cma/count
16384

The issue still occurs when pre-allocating a CMA pool of 640 MiB instead of 64 MiB.

Has anyone else seen this issue before?

I may have found a workaround that involves setting up a reserved memory region and associate it with the sdma controller used by the SAI.

sdma2_reserved: sdma2_res@0xb4000000 {
compatible = "shared-dma-pool";
no-map;
reg = <0 0xb4000000 0 0x4000000>;
};

[...]

&sdma2 {
memory-region = <&sdma2_reserved>;
};

After setting up the above and patching the imx-sdma driver to setup a dedicated DMA pool using of_reserved_mem_device_init() I haven't been abe to reproduce the issue for a few hours.

diff --git a/drivers/dma/imx-sdma.c b/drivers/dma/imx-sdma.c
index 387c842efc13..b540a01dec57 100644
--- a/drivers/dma/imx-sdma.c
+++ b/drivers/dma/imx-sdma.c
@@ -43,6 +43,7 @@
#include <linux/regmap.h>
#include <linux/mfd/syscon.h>
#include <linux/mfd/syscon/imx6q-iomuxc-gpr.h>
+#include <linux/of_reserved_mem.h>

#include "dmaengine.h"
#include "virt-dma.h"
@@ -2446,6 +2447,9 @@ static int sdma_probe(struct platform_device *pdev)
dev_info(&pdev->dev, "alloc bd from iram. \n");
}

+ if (of_reserved_mem_device_init(&pdev->dev))
+ dev_warn(&pdev->dev, "device does not have specific CMA pool\n");
+
if (pdata) {
sdma->fw_name = pdata->fw_name;
} else {

However the global CMA pool should work just fine for DMA, what could be the problem?

Labels (2)
8 Replies

1,624 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello dbrig,

I use the 5.4.47 release image and run on the 8mm EVK board, run command aplay wavname for more than 900 times and can't reproduce your issue, so there should be different on your system, could you clarify if you run any other sdma task when you run the test, the others please check what setting change compare with our release code.

Regards

 

0 Kudos

1,615 Views
dbrig
Contributor II

Hi,

Thank you for looking into the issue.

could you clarify if you run any other sdma task when you run the test

No other sdma tasks are running as far as I know and the result of `grep sdma` on the `dma-api/dump` debugfs entry seems to confirm that. However, IMO the issue is not related to sdma but rather to DMA buffer allocation and from memory there are indeed other pheripherals allocating from the global CMA pool (the FEC ethernet driver accounts for most allocations).

Regards

0 Kudos

1,597 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

right, The memory buffer size maybe change by customer, you can compare the memory size, such as the total size, CMA size, and so on.

 

0 Kudos

1,594 Views
dbrig
Contributor II
The memory buffer size maybe change by customer, you can compare the memory size, such as the total size, CMA size, and so on.

As mentioned in my opening comment the issue still occurs when pre-allocating a CMA pool of 640 MiB instead of 64 MiB. I also reported contents of dma-api and CMA debugfs entries which indicate that the CMA pool is only half-full (and and not fragmented) but the allocation of a sdma descriptor (< 64 bytes) still fails.

DMA-API and CMA kernel debug and leak detection options are turned ON and nothing is reported in the logs. How can I debug this further?

 

0 Kudos

1,572 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hi,

 

Can this issue be reproduced on the EVK your side? Frankly, we can't reproduce it, then can't do more deep investigation, So that's why I ask if there is hardware/software diff between your test platform and EVK, by the way I'm asking internal team to see if they have any suggestion, from their feedback, they never meet such issue.

Regards

 

0 Kudos

1,566 Views
dbrig
Contributor II

 

First of all thank you for looking into this. Our board is simpler than the EVK: no QSPI, no camera, no display, no wireless connectivity. Audio is AK4458 DAC connected to SAI2 and controlled via I2C.

The kernel is https://source.codeaurora.org/external/imx/linux-imx/log/?h=imx_5.4.47_2.2.0 built using buildroot. I understand you cannot help much unless the issue can be reproduced and I would welcome any advice on how to debug this at my end beyond the debugfs information I already provided.

Thank you for your support!

Regards

0 Kudos

1,299 Views
John_C
Contributor I

Hello dbrig,

Did your solution with the reserved memory solve your problem? We are experiencing the same exact error on the iMX8 with Kernel 5.4.85 using an ADS1278 to bring in audio data over SAI2 with SDMA2. I also have not been able to pin-point what is causing our allocation failure.

Thank you!

0 Kudos

1,284 Views
dbrig
Contributor II

Hello @John_C 

> Did your solution with the reserved memory solve your problem?

Yes, it did work.

0 Kudos