iMX6ULL UART SDMA kernel panic

[qba]

There is a bug inside imx-sdma driver.

I am using linux-imx kernel from YOCTO kirkstone release:

Linux version 5.15.52+g102538749583

My device has multiple uart ports with configuration in device-tree like below:

&uart2 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_uart2>;
    dma-names = "rx","tx";
    uart-has-rtscts;
    linux,rs485-enabled-at-boot-time;
    rs485-term-gpios = <&gpio1 23 GPIO_ACTIVE_HIGH>;
    status = "okay";
};

There also is a userspace daemon program started by systemd multiuser.target that is using uart for communication.

If it is started before loading sdma firmware:

[   64.542915] imx-sdma 20ec000.sdma: firmware found.
[   64.561653] imx-sdma 20ec000.sdma: loaded firmware 3.6

Kernel panic occurs:

[   18.668855] Unable to handle kernel NULL pointer dereference at virtual address 00000003
[   18.676958] pgd = b81ad877
[   18.679679] [00000003] *pgd=00000000
[   18.683274] Internal error: Oops: 805 [#1] PREEMPT SMP ARM
[   18.688770] Modules linked in: imx_rngc rng_core 8188eu(O) option usb_wwan usbserial secvio error evbug
[   18.698233] CPU: 0 PID: 255 Comm: sunspecd Tainted: G           O      5.15.52+g53dc604710e6 #1
[   18.706944] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[   18.713132] PC is at sdma_transfer_init+0x120/0x23c
[   18.718039] LR is at 0x0
[   18.720581] pc : [<805e46e8>]    lr : [<00000000>]    psr: 80000193
[   18.726856] sp : 835d5dd8  ip : 8a040280  fp : 822faca0
[   18.732089] r10: 822f8040  r9 : 822fa040  r8 : 00000002
[   18.737320] r7 : 8a040200  r6 : 00000000  r5 : 831cde80  r4 : 822f82d4
[   18.743858] r3 : 00000002  r2 : 00000008  r1 : 00000001  r0 : 8a040200
[   18.750395] Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   18.757631] Control: 10c5387d  Table: 835fc06a  DAC: 00000051
[   18.763383] Register r0 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[   18.773936] Register r1 information: non-paged memory
[   18.779000] Register r2 information: non-paged memory
[   18.784062] Register r3 information: non-paged memory
[   18.789124] Register r4 information: non-slab/vmalloc memory
[   18.794797] Register r5 information: slab kmalloc-128 start 831cde80 pointer offset 0 size 128
[   18.803451] Register r6 information: NULL pointer
[   18.808164] Register r7 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[   18.818710] Register r8 information: non-paged memory
[   18.823771] Register r9 information: non-slab/vmalloc memory
[   18.829442] Register r10 information: non-slab/vmalloc memory
[   18.835199] Register r11 information: non-slab/vmalloc memory
[   18.840955] Register r12 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[   18.851584] Process sunspecd (pid: 255, stack limit = 0xc9724682)
[   18.857692] Stack: (0x835d5dd8 to 0x835d6000)
[   18.862058] 5dc0:                                                       00000000 00000000
[   18.870249] 5de0: 00000002 20000193 e2720104 8218b5dc 00000001 822f82d4 00000001 822fa040
[   18.878440] 5e00: 822f8040 00000000 20000193 805e5320 8218b5dc 82132010 00004000 00000002
[   18.886632] 5e20: 20000193 8218b440 00000000 822f82d4 8218b5dc 82132010 805e52b4 00000004
[   18.894822] 5e40: 20000193 80632d98 00000001 00000000 00000008 20000193 8218b440 9ff347c0
[   18.903011] 5e60: 9ff347c0 9ff34814 80635944 80635984 8218b658 9ff34800 9ff347c0 8019d834
[   18.911202] 5e80: 00000000 57e8f28a 00000006 57e8f28a 00000004 8144e1f4 00000008 9ff347c0
[   18.919392] 5ea0: 20000193 00000003 57e8f28a 9ff347cc 9ff34848 9ff34870 9ff34898 8019e910
[   18.927582] 5ec0: 20000193 0000000f 806147fc 9ff34910 9ff348f0 9ff34940 57e8f28a 00000004
[   18.935772] 5ee0: 835d5f80 8200c140 8200c26c 00000000 835d5f48 00000019 8200c200 8144e16b
[   18.943961] 5f00: 80fe94cc 80931cf4 82023800 8017f3e8 7ee1c49c 00000008 00010000 80fe94f4
[   18.952152] 5f20: 00000000 8200c200 8200c26c 8200c26c 00000000 a080200c 835d5f90 835d5fb0
[   18.960343] 5f40: 00000000 8017f5a0 00000000 8be7beb0 8200c200 8200c26c 8130cac0 80183e98
[   18.968532] 5f60: 81278430 00000000 00000057 00000000 a080200c 8017ed50 81305474 813e76ec
[   18.976724] 5f80: a0802000 8127843c a080200c 80539b74 76dd8fcc 00000030 ffffffff 10c5387d
[   18.984914] 5fa0: 10c5387d 76ed5c4c 7ee1c3dc 80100f50 00000005 7ee1c3e4 00000000 00000000
[   18.993105] 5fc0: 00000004 7ee1c3e4 01c690e0 7ee1c3dc 76ee6edc 76ed5c4c 7ee1c3dc 00000000
[   19.001296] 5fe0: 76ee6f18 7ee1c398 76ed3f7b 76dd8fcc 00000030 ffffffff 00000000 00000000
[   19.009489] [<805e46e8>] (sdma_transfer_init) from [<805e5320>] (sdma_prep_slave_sg+0x6c/0x284)
[   19.018216] [<805e5320>] (sdma_prep_slave_sg) from [<80632d98>] (imx_uart_dma_tx+0xfc/0x230)
[   19.026686] [<80632d98>] (imx_uart_dma_tx) from [<80635984>] (imx_trigger_start_tx+0x40/0x44)
[   19.035238] [<80635984>] (imx_trigger_start_tx) from [<8019d834>] (__hrtimer_run_queues+0x15c/0x218)
[   19.044400] [<8019d834>] (__hrtimer_run_queues) from [<8019e910>] (hrtimer_interrupt+0x124/0x2b0)
[   19.053295] [<8019e910>] (hrtimer_interrupt) from [<80931cf4>] (mxc_timer_interrupt+0x34/0x3c)
[   19.061935] [<80931cf4>] (mxc_timer_interrupt) from [<8017f3e8>] (__handle_irq_event_percpu+0x50/0x130)
[   19.071358] [<8017f3e8>] (__handle_irq_event_percpu) from [<8017f5a0>] (handle_irq_event+0x58/0xc4)
[   19.080431] [<8017f5a0>] (handle_irq_event) from [<80183e98>] (handle_fasteoi_irq+0x9c/0x204)
[   19.088983] [<80183e98>] (handle_fasteoi_irq) from [<8017ed50>] (handle_domain_irq+0x5c/0x78)
[   19.097534] [<8017ed50>] (handle_domain_irq) from [<80539b74>] (gic_handle_irq+0x7c/0x90)
[   19.105741] [<80539b74>] (gic_handle_irq) from [<80100f50>] (__irq_usr+0x50/0x80)
[   19.113251] Exception stack(0x835d5fb0 to 0x835d5ff8)
[   19.118314] 5fa0:                                     00000005 7ee1c3e4 00000000 00000000
[   19.126504] 5fc0: 00000004 7ee1c3e4 01c690e0 7ee1c3dc 76ee6edc 76ed5c4c 7ee1c3dc 00000000
[   19.134692] 5fe0: 76ee6f18 7ee1c398 76ed3f7b 76dd8fcc 00000030 ffffffff
[   19.141323] Code: e5942108 e5872024 e59d3008 e3a01001 (e5c61003)
[   19.147430] ---[ end trace aa80ea8cab929004 ]---
[   19.152055] Kernel panic - not syncing: Fatal exception in interrupt
[   19.158422] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

No matter if CONFIG_IMX_SDMA is configured as module or built-in:

CONFIG_IMX_SDMA=m

CONFIG_IMX_SDMA=y

Kernel panic occurs if communication is started too early. My current workaround is to delay start my service by add

ExecStartPre=/bin/sleep 60

[mattmart]

With a new Kirkstone image, I am facing the same issue which however I was not encountering with a previous Gatesgarth image. What differs is the loading time which increased from about 10 seconds to about 60 seconds. [EDIT]: I mean how much time passes since boot start before the sdma firmware is found and loaded.
I found out that before the meta-imx hardknott-5.10.52-2.1.0 branch, the firmware-imx recipe was installing a script in /etc/sdma that together with additional udev rules (imx-rules recipe) was early triggering the firmware loading.
Since hardknott-5.10.52-2.1.0 branch, the firmware loader scripts have been substituted with the firmwared daemon [ref]. After adding the firmwared package to my image the sdma firmware loading time decreased back to 15 seconds. The firmwared package comes with a systemd-unit which could be tweaked to be activated before the mutli-users.target, however it still would not guarantee the firmware to be loaded before your UART communication: this means you would likely still require the synchronous loading patch but at least it would be loaded sooner.

[qba]

You have linked meta-imx layer - this is "iMX release layer"

Is that layer(and sublayers) necessary when I am creating firmware for my own device?

meta-imx/meta-bsp
meta-imx/meta-sdk
meta-imx/meta-ml
meta-imx/meta-v2x

Maybe I do not understand NXP layers philosophy enough. I have thought that this layer is for demo purpose to provide reference images with proper imx features support.

Also isn't easy to use external kernel defconfig when meta-imx/meta-bsp is enabled

My curent layer set is:

BBLAYERS += "${BSPDIR}/sources/poky/meta"
BBLAYERS += "${BSPDIR}/sources/poky/meta-poky"
BBLAYERS += "${BSPDIR}/sources/meta-openembedded/meta-oe"
BBLAYERS += "${BSPDIR}/sources/meta-openembedded/meta-filesystems"
BBLAYERS += "${BSPDIR}/sources/meta-openembedded/meta-multimedia"
BBLAYERS += "${BSPDIR}/sources/meta-openembedded/meta-networking"
BBLAYERS += "${BSPDIR}/sources/meta-openembedded/meta-python"
BBLAYERS += "${BSPDIR}/sources/meta-freescale"
BBLAYERS += "${BSPDIR}/sources/meta-freescale-distro"
BBLAYERS += "${BSPDIR}/sources/meta-somlabs" - board vendor layer
BBLAYERS += "${BSPDIR}/sources/meta-swupdate" - updates over the air
BBLAYERS += "${BSPDIR}/sources/meta-elsta/meta-elsta-bsp" - my company custom layer
BBLAYERS += "${BSPDIR}/sources/meta-elsta/meta-elsta-distro" - my company custom layer
BBLAYERS += "${BSPDIR}/sources/meta-elsta/meta-elsta-esb" - my company custom layer

[ceggers]

Small is beautiful!

In my project, I have no need for any Freescale layers. Maybe that Freescale layers are required for multimedia stuff. I don't know from which layer the SDMA firmware is normally taken, as I have created a custom one.

[mattmart · Accepted solution]

The SDMA firmware is provided by both meta-freescale and meta-imx layers, the latter provides a newer version of it.
The NXP Yocto user guide IMX Yocto User Guide, suggests to use the layers referenced in the nxp release manifests imx-manifests which include the meta-imx layer as well.
Anyway, @qba I think you are right, the meta-imx is not strictly necessary for machine support since it should only add new patches which should later be merged into the meta-freescale-distro layer. However, also the fsl-image-machine-test image, which is the basic official image with no GUI or multimedia packages provided by the meta-freescale layer, adds the firmwared daemon package to load the firmware from userspace via udev signaling.

Besides that, I also confirm that after configuring the imx-sdma driver as module as suggested by @ceggers , the firmware gets loaded in direct mode immediately after the module gets loaded, in my case it corresponds to about 10 seconds since boot start.

# dmesg | grep sdma
[ 10.459118] imx-sdma 20ec000.sdma: TEST DEBUG: sdma_probe
[ 10.541432] imx-sdma 20ec000.sdma: firmware found.
[ 10.576034] imx-sdma 20ec000.sdma: loaded firmware 3.6

I agree this is a better option, even without the synchronous loading patch, than relying on firmwared daemon because hardly a sdma transfer would take place between the module loading and firmware loading. I am not sure if this applies to all cases, I think it has been opted for a user space loader in the NXP image to support lazy firmware loading for all the possible drivers so that it is not necessary to configure them as module. I hope that usually a safe driver checks whether or not the firmware has been loaded before starting a transfer, or otherwise fails safely without panic.

[qba]

I did full rebuild, and now I am sure that sdma driver is build as module.

CONFIG_IMX_SDMA=y

Now I am also sure that I am not using any recipe from meta-imx which modifies linux-firmware recipe:

https://github.com/nxp-imx/meta-imx/blob/kirkstone-5.15.71-2.2.1/meta-bsp/recipes-kernel/linux-firmw...

by removing sdma-firmware packages as stands there:

# Use the latest version of sdma firmware in firmware-imx
PACKAGES:remove = "${PN}-imx-sdma-license ${PN}-imx-sdma-imx6q ${PN}-imx-sdma-imx7d"

I am not sure if fact that firmware-imx provided sdma firmware was the reason of late firmware load, but without meta-imx layer firmware is loaded 10 seconds after boot:

[ 10.207536] imx-sdma 20ec000.sdma: firmware found.
[ 10.257309] imx-sdma 20ec000.sdma: loaded firmware 3.5

[mattmart]

I think it's just the fact you built the driver as module (plus the full rebuild) that solved the issue since I am in fact using the firmware-imx-sdma-imx6q package provided by the meta-imx layer and resulting in your same behavior.

[ceggers]

As I have never used any NXP kernels, I don't know how SDMA firmware loading has been done there in the past. On my system, I do not require any userspace daemon for firmware loading, as the kernel is able to this itself.

the sdma firmware loading time decreased back to 15 seconds

Do you mean that it takes 15 seconds to load the SDMA firmware?!?

this means you would likely still require the synchronous loading patch but at least it would be loaded sooner.

Does this mean that synchronous SDMA firmware loading solves your problem?

regards,
Christian

[mattmart]

Sorry, I meant that the SDMA firmware is found and loaded after 15 second the boot started (instead of 64 seconds reported by @qba), and about 10 seconds after the rootfs is mounted.

# dmesg | grep sdma
[ 0.560423] imx-sdma 20ec000.sdma: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[ 0.560489] imx-sdma 20ec000.sdma: Falling back to sysfs fallback for: imx/sdma/sdma-imx6q.bin
[ 15.375231] imx-sdma 20ec000.sdma: firmware found.
[ 15.396199] imx-sdma 20ec000.sdma: loaded firmware 3.6

To my understanding the driver signals to the user-space, through uevents (u-dev), that it is waiting for a firmware to be loaded. At this point the firmwared daemon captures the u-dev event and loads the firmware. It looks like that without any user-space loader the kernel loads the firmware anyway but much later, I am not sure how it does it as I did not dig deeper to that part.

I also tested the synchronous loading patch but it looks like the serial output (or the whole kernel) freezes until the kernel firmware is actually loaded. For my current situation, using firmwared is likely enough since most of the user space applications start later.

I guess the proper solution at the driver level would be to avoid starting sdma tx/rx if the driver has not yet been loaded. Haven't checked but maybe such a solution is already in later versions of the driver?

[ceggers1]

# dmesg | grep sdma
[ 0.560423] imx-sdma 20ec000.sdma: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[ 0.560489] imx-sdma 20ec000.sdma: Falling back to sysfs fallback for: imx/sdma/sdma-imx6q.bin
[ 15.375231] imx-sdma 20ec000.sdma: firmware found.
[ 15.396199] imx-sdma 20ec000.sdma: loaded firmware 3.6

At least if the SDMA driver is built as a kernel module (and so it is loaded after userspace is available), there shouldn't be large delay between driver and firmware loading. Also the error message (first line) should disappear.

I load the SDMA driver "manually" as the first step in my boot sequence (I use busybox inittab, so I cannot tell how to do this with systemd). So I can ensure that ..

1. the firmware (in user space) is accessible by the kernel when the driver is loaded
2. the SDMA is available before any other kernel modules are loaded

Some device drivers require that the SDMA is available at probe time, others require the SDMA only when the device is actually used (e.g. opened).

To my understanding the driver signals to the user-space, through uevents (u-dev), that it is waiting for a firmware to be loaded.

I think this is in general correct. But for specific (simple) cases, the kernel is able to fetch the firmware from file system without the help of any userspace software. I think that this is are more recent feature, but I cannot provide any details.

but it looks like the serial output (or the whole kernel) freezes until the kernel is actually loaded

I never had such problems on my system (I have used several vanilla kernels since 4.14). Please also note that the kernel doesn't (or shouldn't) use the SDMA for the UART which is used as serial console.

Haven't checked but maybe such a solution is already in later versions of the driver?

I think that every kernel since 5.4 should be recent enough... It also looks that boot time on your system is quite long. My best time (power on to hello world) on i.MX6LL was < 3 seconds (but with many optimizations and without systemd).

regards,
Christian

[mattmart]

At least if the SDMA driver is built as a kernel module (and so it is loaded after userspace is available), there shouldn't be large delay between driver and firmware loading. Also the error message (first line) should disappear.

I confirm this, see my my latest comment.

I think this is in general correct. But for specific (simple) cases, the kernel is able to fetch the firmware from file system without the help of any userspace software.

True, but if it fails to load the firmware, e.g. due to rootfs not mounted yet, it falls back to wait for userspace loading by exposing a sysfs interface (see firmware-fallback-sysfs ). Then I think that the kernel tries again to load it by itself after a certain timeout if no user-space loader is in place.

[ceggers]

My device has multiple uart ports with configuration in device-tree like below:

&uart2 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_uart2>;
    dma-names = "rx","tx";
    uart-has-rtscts;
    linux,rs485-enabled-at-boot-time;
    rs485-term-gpios = <&gpio1 23 GPIO_ACTIVE_HIGH>;
    status = "okay";
};

The "dmas" statement is missing:

dmas = <&sdma 27 4 2>, <&sdma 28 4 2>;

The first number (27/28) is the DMA channel number (see IMX6ULLRM.pdf, page 189). The 2nd number (4) selects the SDMA script ("MCU domain UART" in this case) and the last number (2) is the SDMA channel priority (0=highest, 2=lowest).

regards,
Christian

[qba]

Thanks for pointing out missing "dmas" statement. I've checked kernel imx6ul.dtsi file and sdma channels are already configured for all UART ports. Also my "dma-names" wasn't needed

			uart2: serial@21e8000 {
				compatible = "fsl,imx6ul-uart",
					     "fsl,imx6q-uart";
				reg = <0x021e8000 0x4000>;
				interrupts = <GIC_SPI 27 IRQ_TYPE_LEVEL_HIGH>;
				clocks = <&clks IMX6UL_CLK_UART2_IPG>,
					 <&clks IMX6UL_CLK_UART2_SERIAL>;
				clock-names = "ipg", "per";
				dmas = <&sdma 27 4 0>, <&sdma 28 4 0>;
				dma-names = "rx", "tx";
				status = "disabled";
			};

[ceggers]

As far as I remember, the SDMA firmware is loaded asynchronously by the Linux SDMA driver. In my current project, I switched to the "blocking" firmware loading api in order to avoid such problems...

Please let me know whether the attached patch helps in your case.

regards,
Christian

[qba]

After applying your patch the following things changed:

After loading kernel to RAM:

## Flattened Device Tree blob at 83000000
   Booting using the fdt blob at 0x83000000
   Loading Device Tree to 9fb4f000, end 9fb5ab35 ... OK

Starting kernel ...

 Nothing happens for about 60 seconds - until SDMA firmware should be loaded. Then kernel starts printing boot messages

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.15.52+ge140f98d5bd3 (oe-user@oe-host) (arm-fsl-linux-gnueabi-gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38.20220623) #1 SMP PREEMPT Wed Mar 8 20:16:20 UTC 2023
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: CBDM01
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Reserved memory: created CMA memory pool at 0x8a000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000] percpu: Embedded 12 pages/cpu s17164 r8192 d23796 u49152
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129920
[    0.000000] Kernel command line: console=ttymxc0,115200 root=/dev/mmcblk1p3 rootwait rw
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 170548K/524288K available (12288K kernel code, 1338K rwdata, 4300K rodata, 1024K init, 433K bss, 26060K reserved, 327680K cma-reserved, 0K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] Switching to timer-based delay loop, resolution 41ns
[    0.000003] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[    0.000056] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.003201] Console: colour dummy device 80x30
[    0.003319] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=240000)
[    0.003376] pid_max: default: 32768 minimum: 301
[    0.003889] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.003956] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.005794] CPU: Testing write buffer coherency: ok
[    0.006508] CPU0: update cpu_capacity 1024
[    0.006561] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.008981] Setting up static identity map for 0x80100000 - 0x80100060
[    0.009403] rcu: Hierarchical SRCU implementation.
[    0.010733] smp: Bringing up secondary CPUs ...
[    0.010782] smp: Brought up 1 node, 1 CPU
[    0.010817] SMP: Total of 1 processors activated (48.00 BogoMIPS).
[    0.010847] CPU: All CPU(s) started in SVC mode.
[    0.011938] devtmpfs: initialized
[    0.028313] VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
[    0.028896] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.028975] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.053396] pinctrl core: initialized pinctrl subsystem
[    0.057295] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.075131] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.078101] thermal_sys: Registered thermal governor 'step_wise'
[    0.078510] cpuidle: using governor menu
[    0.078863] CPU identified as i.MX6ULL, silicon rev 1.1
[    0.105887] vdd3p0: supplied by regulator-dummy
[    0.106944] cpu: supplied by regulator-dummy
[    0.108460] vddsoc: supplied by regulator-dummy
[    0.147032] failed to find ocotp node
[    0.147516] failed to find ocotp node
[    0.147937] No ATAGs?
[    0.148077] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.148120] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.151091] imx6ul-pinctrl 20e0000.pinctrl: initialized IMX pinctrl driver
[    0.152671] imx6ul-pinctrl 2290000.iomuxc-snvs: initialized IMX pinctrl driver
[    0.155807] imx mu driver is registered.
[    0.156831] imx rpmsg driver is registered.
[    0.192903] Kprobes globally optimized
[    0.218813] gpio-65 (can_enable): hogged as output/high
[    0.223441] gpio-123 (lte_enable): hogged as output/high
[    0.227364] gpio-134 (hub_reset): hogged as output/high
[    0.241594] vgaarb: loaded
[    0.243618] SCSI subsystem initialized
[    0.244866] usbcore: registered new interface driver usbfs
[    0.245039] usbcore: registered new interface driver hub
[    0.245172] usbcore: registered new device driver usb
[    0.249589] i2c i2c-0: IMX I2C adapter registered
[    0.252433] i2c i2c-1: IMX I2C adapter registered
[    0.253313] mc: Linux media interface: v0.10
[    0.253443] videodev: Linux video capture interface: v2.00
[    0.253707] pps_core: LinuxPPS API ver. 1 registered
[    0.253740] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.253803] PTP clock support registered
[    0.256962] MIPI CSI2 driver module loaded
[    0.257130] Advanced Linux Sound Architecture Driver Initialized.
[    0.259054] Bluetooth: Core ver 2.22
[    0.259232] NET: Registered PF_BLUETOOTH protocol family
[    0.259266] Bluetooth: HCI device and connection manager initialized
[    0.259313] Bluetooth: HCI socket layer initialized
[    0.259346] Bluetooth: L2CAP socket layer initialized
[    0.259407] Bluetooth: SCO socket layer initialized
[    0.260718] clocksource: Switched to clocksource mxc_timer1
[    0.261146] VFS: Disk quotas dquot_6.6.0
[    0.261310] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.282331] NET: Registered PF_INET protocol family
[    0.282909] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.285171] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    0.285287] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.285339] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.285458] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.285633] TCP: Hash tables configured (established 4096 bind 4096)
[    0.285852] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.285946] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.286391] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.288076] RPC: Registered named UNIX socket transport module.
[    0.288139] RPC: Registered udp transport module.
[    0.288164] RPC: Registered tcp transport module.
[    0.288185] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.290504] PCI: CLS 0 bytes, default 64
[    0.291790] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[    0.296270] Bus freq driver module loaded
[    0.298495] Initialise system trusted keyrings
[    0.299824] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[    0.315293] NFS: Registering the id_resolver key type
[    0.315604] Key type id_resolver registered
[    0.315637] Key type id_legacy registered
[    0.315902] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.315940] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.316041] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    0.317299] fuse: init (API version 7.34)
[    0.558313] Key type asymmetric registered
[    0.558376] Asymmetric key parser 'x509' registered
[    0.558571] io scheduler mq-deadline registered
[    0.558608] io scheduler kyber registered

Here kernel tries to load firmware:

[    0.581201] imx-sdma 20ec000.sdma: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[    0.581277] imx-sdma 20ec000.sdma: Falling back to sysfs fallback for: imx/sdma/sdma-imx6q.bin
[   64.497374] imx-sdma 20ec000.sdma: failed to get firmware.

but without success. Rootfs is not ready yet

[   64.606310] mxs-dma 1804000.dma-apbh: initialized
[   64.803768] 2018000.serial: ttymxc6 at MMIO 0x2018000 (irq = 28, base_baud = 5000000) is a IMX
[   64.857364] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 29, base_baud = 5000000) is a IMX
[   65.714919] printk: console [ttymxc0] enabled
[   65.784830] 21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 66, base_baud = 5000000) is a IMX
[   65.849489] 21ec000.serial: ttymxc2 at MMIO 0x21ec000 (irq = 67, base_baud = 5000000) is a IMX
[   65.917209] 21f0000.serial: ttymxc3 at MMIO 0x21f0000 (irq = 68, base_baud = 5000000) is a IMX
[   65.986190] 21f4000.serial: ttymxc4 at MMIO 0x21f4000 (irq = 69, base_baud = 5000000) is a IMX

....

and then about 2 mins after system boot(my service is started after delay) kernel panic occurs:

  144.918973] imx-uart 21f0000.serial: We cannot prepare for the RX slave dma!
[  144.962052] 8<--- cut here ---
[  144.965154] Unable to handle kernel NULL pointer dereference at virtual address 00000003
[  144.973256] pgd = 239a35ca
[  144.975979] [00000003] *pgd=00000000
[  144.979573] Internal error: Oops: 805 [#1] PREEMPT SMP ARM
[  144.985070] Modules linked in: imx_rngc rng_core 8188eu(O) option usb_wwan usbserial secvio error evbug
[  144.994528] CPU: 0 PID: 569 Comm: sunspecd Tainted: G           O      5.15.52+ge140f98d5bd3 #1
[  145.003239] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[  145.009424] PC is at sdma_transfer_init+0x120/0x23c
[  145.014328] LR is at 0x0
[  145.016867] pc : [<805e46d8>]    lr : [<00000000>]    psr: 80000193
[  145.023141] sp : 82b8fdd8  ip : 8a040280  fp : 822faca0
[  145.028374] r10: 822f8040  r9 : 822fa040  r8 : 00000002
[  145.033604] r7 : 8a040200  r6 : 00000000  r5 : 83b21600  r4 : 822f82d4
[  145.040139] r3 : 00000002  r2 : 00000008  r1 : 00000001  r0 : 8a040200
[  145.046674] Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  145.053908] Control: 10c5387d  Table: 83ae006a  DAC: 00000051
[  145.059658] Register r0 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[  145.070213] Register r1 information: non-paged memory
[  145.075274] Register r2 information: non-paged memory
[  145.080333] Register r3 information: non-paged memory
[  145.085395] Register r4 information: non-slab/vmalloc memory
[  145.091064] Register r5 information: slab kmalloc-128 start 83b21600 pointer offset 0 size 128
[  145.099714] Register r6 information: NULL pointer
[  145.104426] Register r7 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[  145.114969] Register r8 information: non-paged memory
[  145.120030] Register r9 information: non-slab/vmalloc memory
[  145.125698] Register r10 information: non-slab/vmalloc memory
[  145.131453] Register r11 information: non-slab/vmalloc memory
[  145.137209] Register r12 information: 0-page vmalloc region starting at 0x8a000000 allocated at iotable_init+0x0/0xec
[  145.147838] Process sunspecd (pid: 569, stack limit = 0x0f604475)
[  145.153943] Stack: (0x82b8fdd8 to 0x82b90000)
[  145.158308] fdc0:                                                       00000000 00000000
[  145.166499] fde0: 00000002 20000193 f2720306 8218b5dc 00000001 822f82d4 00000001 822fa040
[  145.174687] fe00: 822f8040 00000000 20000193 805e5310 8218b5dc 82132010 00004000 00000002
[  145.182876] fe20: 20000193 8218b440 00000000 822f82d4 8218b5dc 82132010 805e52a4 00000021
[  145.191065] fe40: 20000193 80632d88 00000001 00000000 00000008 20000193 8218b440 9ff347c0
[  145.199253] fe60: 9ff347c0 9ff34814 80635934 80635974 8218b658 9ff34800 9ff347c0 8019d834
[  145.207441] fe80: 00000000 bfc5a69b 00000006 bfc5a69b 00000021 8144e1f4 00000008 9ff347c0
[  145.215629] fea0: 20000193 00000003 bfc5a69b 9ff347cc 9ff34848 9ff34870 9ff34898 8019e910
[  145.223816] fec0: 20000193 0000000f 806147ec 9ff34910 9ff348f0 9ff34940 bfc5a69b 00000021
[  145.232004] fee0: 82b8ff80 8200c140 8200c26c 00000000 82b8ff48 00000019 8200c200 8144e16b
[  145.240192] ff00: 80fe94cc 80931ce4 82023800 8017f3e8 7e96949c 00000008 00010000 80fe94f4
[  145.248380] ff20: 00000000 8200c200 8200c26c 8200c26c 00000000 a080200c 82b8ff90 82b8ffb0
[  145.256568] ff40: 00000000 8017f5a0 00000000 fa0dd3da 8200c200 8200c26c 8130cac0 80183e98
[  145.264756] ff60: 81278430 00000000 00000057 00000000 a080200c 8017ed50 81305474 813e76ec
[  145.272944] ff80: a0802000 8127843c a080200c 80539b74 76e58d88 40000030 ffffffff 10c5387d
[  145.281132] ffa0: 10c5387d 76f55c4c 7e9693dc 80100f50 00000005 7e9693e4 00000000 00000000
[  145.289320] ffc0: 7e9693dc 7e969378 01f55da8 7e9693dc 76f66edc 76f55c4c 7e9693dc 00000000
[  145.297508] ffe0: 76f66f18 7e969370 76e58fed 76e58d88 40000030 ffffffff 00000000 00000000
[  145.305699] [<805e46d8>] (sdma_transfer_init) from [<805e5310>] (sdma_prep_slave_sg+0x6c/0x284)
[  145.314425] [<805e5310>] (sdma_prep_slave_sg) from [<80632d88>] (imx_uart_dma_tx+0xfc/0x230)
[  145.322889] [<80632d88>] (imx_uart_dma_tx) from [<80635974>] (imx_trigger_start_tx+0x40/0x44)
[  145.331440] [<80635974>] (imx_trigger_start_tx) from [<8019d834>] (__hrtimer_run_queues+0x15c/0x218)
[  145.340599] [<8019d834>] (__hrtimer_run_queues) from [<8019e910>] (hrtimer_interrupt+0x124/0x2b0)
[  145.349494] [<8019e910>] (hrtimer_interrupt) from [<80931ce4>] (mxc_timer_interrupt+0x34/0x3c)
[  145.358132] [<80931ce4>] (mxc_timer_interrupt) from [<8017f3e8>] (__handle_irq_event_percpu+0x50/0x130)
[  145.367553] [<8017f3e8>] (__handle_irq_event_percpu) from [<8017f5a0>] (handle_irq_event+0x58/0xc4)
[  145.376623] [<8017f5a0>] (handle_irq_event) from [<80183e98>] (handle_fasteoi_irq+0x9c/0x204)
[  145.385171] [<80183e98>] (handle_fasteoi_irq) from [<8017ed50>] (handle_domain_irq+0x5c/0x78)
[  145.393717] [<8017ed50>] (handle_domain_irq) from [<80539b74>] (gic_handle_irq+0x7c/0x90)
[  145.401922] [<80539b74>] (gic_handle_irq) from [<80100f50>] (__irq_usr+0x50/0x80)
[  145.409427] Exception stack(0x82b8ffb0 to 0x82b8fff8)
[  145.414488] ffa0:                                     00000005 7e9693e4 00000000 00000000
[  145.422677] ffc0: 7e9693dc 7e969378 01f55da8 7e9693dc 76f66edc 76f55c4c 7e9693dc 00000000
[  145.430864] ffe0: 76f66f18 7e969370 76e58fed 76e58d88 40000030 ffffffff
[  145.437494] Code: e5942108 e5872024 e59d3008 e3a01001 (e5c61003)
[  145.443600] ---[ end trace 0274ce7a37069a08 ]---
[  145.448224] Kernel panic - not syncing: Fatal exception in interrupt
[  145.454594] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

It happens because no firmware is loaded, and it was not possible right after start because rootfs isn't ready.

SDMA driver was configured as module:

CONFIG_IMX_SDMA=m

[ceggers]

The log output doesn't really look like CONFIG_IMX_SDMA is set to 'm:

[    0.581201] imx-sdma 20ec000.sdma: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[    0.581277] imx-sdma 20ec000.sdma: Falling back to sysfs fallback for: imx/sdma/sdma-imx6q.bin
[   64.497374] imx-sdma 20ec000.sdma: failed to get firmware.

Usually one can see messages like "Freeing initrd memory: 976K" or "Freeing unused kernel image (initmem) memory: 1024K" when the kernel passes control to userspace (init process). So I would expect seeing the SDMA messages after that.

I rechecked my own DTS configuration. I also use ttymxc0 as console and have also set "dmas" and "dma-names" for this UART. But I know from the i.MX serial driver, that no SDMA is used in this case. So there shouldn't be a "circular dependency" between the console output and the SDMA firmware loading.

[Bio_TICFSL]

(y)