iMX8 mini UART kernel panic

LorenzoSpadaro · ‎02-12-2024

Hi,

i have a imx8 mini processor. I am using all the 4 uart, one of these seems to trigger a Kernel Panic apparently randomly after some time (also a lot of hour of work). These are some examples of the error log:

"""
[53664.843877] Internal error: synchronous external abort: 96000210 [#1] PREEMPT SMP
[53664.851375] Modules linked in: xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 GobiSerial(O) libcrc32c GobiNet(O) iptable_filter ip_tables x_tables uvcvideo crct10dif_ce
[53664.867220] CPU: 0 PID: 1194 Comm: weston Tainted: G W O 5.4.3+g3acc432cc #1
[53664.875405] Hardware name: Alma # DeviceTree:1.1.0 Kernel:1.2.0 (DT)
[53664.881766] pstate: 40000085 (nZcv daIf -PAN -UAO)
[53664.886565] pc : imx_uart_rxint+0x74/0x288
[53664.890664] lr : imx_uart_rxint+0x3c/0x288
[53664.894759] sp : ffff800010003de0
[53664.898073] x29: ffff800010003de0 x28: 0000000000000060
[53664.903387] x27: 0000000000000001 x26: ffff800016a02dd3
[53664.908700] x25: 0000000000000002 x24: 0000000000000000
[53664.914013] x23: ffff8000168d6000 x22: 0000000000000004
[53664.919326] x21: ffff00004e02c340 x20: ffff8000174c0000
[53664.924639] x19: ffff000049d1e080 x18: 0000000000000000
[53664.929951] x17: 0000000000000000 x16: 0000000000000000
[53664.935263] x15: 0000000000000000 x14: 0000000000000000
[53664.940574] x13: 0000000000000000 x12: 0000000000000000
[53664.945886] x11: 0000000000000000 x10: 0000000000000040
[53664.951199] x9 : ffff8000168f15d8 x8 : ffff8000168f15d0
[53664.956511] x7 : ffff0000487d8000 x6 : 0000000000000000
[53664.961823] x5 : ffff00004c000248 x4 : 0000000000000001
[53664.967135] x3 : ffff000049d1e080 x2 : 0000000000000000
[53664.972447] x1 : 0000000000000000 x0 : 000000000018aa5e
[53664.977760] Call trace:
[53664.980208] imx_uart_rxint+0x74/0x288
[53664.983957] imx_uart_int+0x2ac/0x310
[53664.987625] __handle_irq_event_percpu+0x64/0x170
[53664.992332] handle_irq_event_percpu+0x30/0x88
[53664.996777] handle_irq_event+0x44/0xc8
[53665.000615] handle_fasteoi_irq+0xb4/0x160
[53665.004713] generic_handle_irq+0x24/0x38
[53665.008724] __handle_domain_irq+0x60/0xb8
[53665.012822] gic_handle_irq+0x5c/0x148
[53665.016572] el1_irq+0xb8/0x180
[53665.019714] ___sys_recvmsg+0x0/0xd8
[53665.023291] __arm64_sys_recvmsg+0x20/0x28
[53665.027391] el0_svc_common.constprop.0+0x68/0x160
[53665.032183] el0_svc_handler+0x20/0x80
[53665.035932] el0_svc+0x8/0xc
[53665.038817] Code: b940e260 f9400a74 11000400 b900e260 (b9400294)
[53665.044919] ---[ end trace 44d8f1d376146734 ]---
[53665.049536] Kernel panic - not syncing: Fatal exception in interrupt
[53665.055891] SMP: stopping secondary CPUs
[53666.059815] SMP: failed to stop secondary CPUs 0,2
[53666.064606] Kernel Offset: disabled
[53666.068094] CPU features: 0x0002,2000200c
[53666.072100] Memory Limit: none
[53666.075158] Rebooting in 3 seconds..
[53669.150633] SMP: stopping secondary CPUs
[53670.154554] SMP: failed to stop secondary CPUs 0,2
"""

"""
[30424.088555] Internal error: synchronous external abort: 96000210 [#1] PREEMPT SMP
[30424.096049] Modules linked in: uvcvideo crct10dif_ce xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c GobiSerial(O) GobiNet(O) iptable_filter ip_tables x_tables
[30424.111893] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 5.4.3+g3acc432cc #1
[30424.120070] Hardware name: Alma # DeviceTree:1.1.0 Kernel:1.2.0 (DT)
[30424.126425] pstate: 40000085 (nZcv daIf -PAN -UAO)
[30424.131224] pc : imx_uart_rxint+0x74/0x288
[30424.135322] lr : imx_uart_rxint+0x3c/0x288
[30424.139417] sp : ffff800010003de0
[30424.142731] x29: ffff800010003de0 x28: 0000000000000060
[30424.148044] x27: 0000000000000001 x26: ffff800016a02dd3
[30424.153357] x25: 0000000000000002 x24: 0000000000000000
[30424.158669] x23: ffff8000168d6000 x22: 0000000000000004
[30424.163980] x21: ffff00004e2e8340 x20: ffff8000174f0000
[30424.169293] x19: ffff000049cd0080 x18: 0000000000000000
[30424.174605] x17: 0000000000000000 x16: 0000000000000000
[30424.179918] x15: 0000000000000000 x14: 0000000000000000
[30424.185231] x13: 00c8094e84a88a82 x12: 0000000000000001
[30424.190543] x11: 0000000000000001 x10: 0000000000000040
[30424.195856] x9 : ffff8000168f15d8 x8 : ffff8000168f15d0
[30424.201169] x7 : ffff8000168e1f00 x6 : 0000000000000000
[30424.206481] x5 : ffff00004c000248 x4 : 0000000000000001
[30424.211793] x3 : ffff000049cd0080 x2 : 0000000000000000
[30424.217106] x1 : 0000000000000000 x0 : 0000000000167662
[30424.222419] Call trace:
[30424.224869] imx_uart_rxint+0x74/0x288
[30424.228619] imx_uart_int+0x2ac/0x310
[30424.232285] __handle_irq_event_percpu+0x64/0x170
[30424.236992] handle_irq_event_percpu+0x30/0x88
[30424.241437] handle_irq_event+0x44/0xc8
[30424.245274] handle_fasteoi_irq+0xb4/0x160
[30424.249371] generic_handle_irq+0x24/0x38
[30424.253383] __handle_domain_irq+0x60/0xb8
[30424.257481] gic_handle_irq+0x5c/0x148
[30424.261230] el1_irq+0xb8/0x180
[30424.264373] sched_ttwu_pending+0x0/0xc0
[30424.268298] cpu_startup_entry+0x24/0x40
[30424.272225] rest_init+0xd4/0xe0
[30424.275456] arch_call_rest_init+0xc/0x14
[30424.279466] start_kernel+0x420/0x44c
[30424.283133] Code: b940e260 f9400a74 11000400 b900e260 (b9400294)
[30424.289236] ---[ end trace 61ee38b101d7830d ]---
[30424.293854] Kernel panic - not syncing: Fatal exception in interrupt
[30424.300209] SMP: stopping secondary CPUs
[30425.304133] SMP: failed to stop secondary CPUs 0-1
[30425.308923] Kernel Offset: disabled
[30425.312410] CPU features: 0x0002,2000200c
[30425.316417] Memory Limit: none
[30425.319475] Rebooting in 3 seconds..
[30428.395158] SMP: stopping secondary CPUs
[30429.399080] SMP: failed to stop secondary CPUs 0-1
"""

"""
[ 9034.374177] Internal error: synchronous external abort: 96000210 [#1] PREEMPT SMP
[ 9034.381670] Modules linked in: xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c GobiSerial(O) GobiNet(O) iptable_filter ip_tables x_tables uvcvideo crct10dif_ce
[ 9034.397511] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 5.4.3+g3acc432cc #1
[ 9034.405687] Hardware name: Engicam i.Core MX8MM intelliTower (DT)
[ 9034.411782] pstate: 40000085 (nZcv daIf -PAN -UAO)
[ 9034.416580] pc : imx_uart_rxint+0x74/0x288
[ 9034.420677] lr : imx_uart_rxint+0x3c/0x288
[ 9034.424773] sp : ffff800010003de0
[ 9034.428087] x29: ffff800010003de0 x28: 0000000000000060
[ 9034.433399] x27: 0000000000000001 x26: ffff800016983453
[ 9034.438712] x25: 0000000000000002 x24: 0000000000000000
[ 9034.444023] x23: ffff800016856000 x22: 0000000000000004
[ 9034.449336] x21: ffff00004e2d0340 x20: ffff800017430000
[ 9034.454647] x19: ffff00004e4c5080 x18: 0000000000000000
[ 9034.459960] x17: 0000000000000000 x16: 0000000000000000
[ 9034.465272] x15: 0000000000000000 x14: 0000000000000000
[ 9034.470584] x13: 0000000000000000 x12: 0000000000001a8f
[ 9034.475896] x11: 0000000000000258 x10: 0000000000000040
[ 9034.481208] x9 : ffff8000168715d8 x8 : ffff8000168715d0
[ 9034.486520] x7 : ffff800016861f00 x6 : 0000000000000000
[ 9034.491832] x5 : ffff00004c000248 x4 : 0000000000000001
[ 9034.497144] x3 : ffff00004e4c5080 x2 : 0000000000000000
[ 9034.502456] x1 : 0000000000000000 x0 : 000000000017327b
[ 9034.507769] Call trace:
[ 9034.510219] imx_uart_rxint+0x74/0x288
[ 9034.513968] imx_uart_int+0x2ac/0x310
[ 9034.517635] __handle_irq_event_percpu+0x64/0x170
[ 9034.522339] handle_irq_event_percpu+0x30/0x88
[ 9034.526784] handle_irq_event+0x44/0xc8
[ 9034.530620] handle_fasteoi_irq+0xb4/0x160
[ 9034.534716] generic_handle_irq+0x24/0x38
[ 9034.538727] __handle_domain_irq+0x60/0xb8
[ 9034.542825] gic_handle_irq+0x5c/0x148
[ 9034.546574] el1_irq+0xb8/0x180
[ 9034.549717] cpuidle_enter_state+0x84/0x360
[ 9034.553901] cpuidle_enter+0x34/0x48
[ 9034.557477] call_cpuidle+0x18/0x38
[ 9034.560966] do_idle+0x1e0/0x280
[ 9034.564195] cpu_startup_entry+0x20/0x40
[ 9034.568119] rest_init+0xd4/0xe0
[ 9034.571349] arch_call_rest_init+0xc/0x14
[ 9034.575360] start_kernel+0x420/0x44c
[ 9034.579026] Code: b940e260 f9400a74 11000400 b900e260 (b9400294)
[ 9034.585128] ---[ end trace 202c87084a341874 ]---
[ 9034.589745] Kernel panic - not syncing: Fatal exception in interrupt
[ 9034.596100] SMP: stopping secondary CPUs
[ 9034.600358] ------------[ cut here ]------------
[ 9034.604990] WARNING: CPU: 2 PID: 0 at kernel/time/tick-broadcast.c:655 tick_handle_oneshot_broadcast+0x1e4/0x200
[ 9034.615160] Modules linked in: xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c GobiSerial(O) GobiNet(O) iptable_filter ip_tables x_tables uvcvideo crct10dif_ce
[ 9034.630994] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D W O 5.4.3+g3acc432cc #1
[ 9034.639167] Hardware name: Engicam i.Core MX8MM intelliTower (DT)
[ 9034.645259] pstate: 00000085 (nzcv daIf -PAN -UAO)
[ 9034.650050] pc : tick_handle_oneshot_broadcast+0x1e4/0x200
[ 9034.655535] lr : tick_handle_oneshot_broadcast+0x124/0x200
[ 9034.661017] sp : ffff800010013e10
[ 9034.664329] x29: ffff800010013e10 x28: 0000000000000060
[ 9034.669640] x27: ffff8000168592b4 x26: ffff8000169d1010
[ 9034.674951] x25: 7fffffffffffffff x24: ffff800016934b00
[ 9034.680262] x23: ffff800016856a00 x22: ffff8000169d10a0
[ 9034.685573] x21: ffff800016859118 x20: ffff8000169d1050
[ 9034.690883] x19: 0000000000000000 x18: 0000000000000000
[ 9034.696194] x17: 0000000000000000 x16: 0000000000000000
[ 9034.701504] x15: 0000000000000000 x14: ffff00004e0aaa00
[ 9034.706815] x13: ffff800067549000 x12: 0000000034d4d91d
[ 9034.712126] x11: 0000000000000000 x10: 0000000000000040
[ 9034.717437] x9 : ffff8000168715d8 x8 : ffff8000168715d0
[ 9034.722747] x7 : 0000000000000001 x6 : ffff8000169d1050
[ 9034.728058] x5 : 000000000000000d x4 : 0000000000000006
[ 9034.733369] x3 : 0000000000000001 x2 : 0000000000000100
[ 9034.738679] x1 : ffff800016859118 x0 : 0000000000000000
[ 9034.743990] Call trace:
[ 9034.746438] tick_handle_oneshot_broadcast+0x1e4/0x200
[ 9034.751576] sysctr_timer_interrupt+0x34/0x40
[ 9034.755934] __handle_irq_event_percpu+0x64/0x170
[ 9034.760637] handle_irq_event_percpu+0x30/0x88
[ 9034.765080] handle_irq_event+0x44/0xc8
[ 9034.768916] handle_fasteoi_irq+0xb4/0x160
[ 9034.773012] generic_handle_irq+0x24/0x38
[ 9034.777021] __handle_domain_irq+0x60/0xb8
[ 9034.781116] gic_handle_irq+0x5c/0x148
[ 9034.784864] el1_irq+0xb8/0x180
[ 9034.788005] cpuidle_enter_state+0x84/0x360
[ 9034.792187] cpuidle_enter+0x34/0x48
[ 9034.795762] call_cpuidle+0x18/0x38
[ 9034.799250] do_idle+0x1e0/0x280
[ 9034.802477] cpu_startup_entry+0x20/0x40
[ 9034.806401] secondary_start_kernel+0x154/0x190
[ 9034.810930] ---[ end trace 202c87084a341875 ]---
[ 9035.600023] SMP: failed to stop secondary CPUs 0,3
[ 9035.604812] Kernel Offset: disabled
[ 9035.608300] CPU features: 0x0002,2000200c
[ 9035.612306] Memory Limit: none
[ 9035.615365] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
"""

I am using kernel version 5.4.3, builded with Zeus version of Yocto.
This is the device tree for the uarts:

"""

&uart1 { /* microcontroller */

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_uart1>;

assigned-clocks = <&clk IMX8MM_CLK_UART1>;

assigned-clock-parents = <&clk IMX8MM_SYS_PLL1_80M>;

status = "okay";

};

&uart2 { /* console */

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_uart2>;

status = "okay";

};

&uart3 {

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_uart3>;

assigned-clocks = <&clk IMX8MM_CLK_UART3>;

assigned-clock-parents = <&clk IMX8MM_SYS_PLL1_80M>;

fsl,uart-has-rtscts; // zeus

control-gpios = <&gpio5 8 GPIO_ACTIVE_HIGH>;

linux,rs485-enabled-at-boot-time; /* rs485 enabled at boot */

rs485-mode = <1>; /* rs485 mode */

/*cts-gpios = <&gpio5 8 GPIO_ACTIVE_HIGH>; */

/*rts-gpios = <&gpio5 9 GPIO_ACTIVE_LOW>; */

off_levels = <0>;

rs485_levels = <0x1>;

rs485_txen_levels = <1>;

rs485_txen_mask = <1>;

status = "okay";

};

&uart4 { /* */

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_uart4>;

status = "okay";

};
"""

I am quite sure the problem is in uart3, but i can also be uart1. The other 2 i think it can't be.
Which can be the problem?

Chavira · ‎02-12-2024

Hi @LorenzoSpadaro!
Thank you for contacting NXP Support!

What board are you using?

Please check this post on how to configure the UART4 in IMX8MM and try that configuration.

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-use-UART4-on-iMX8M-from-Linux-Use...

Do you modify the BL13_setup.c?

Best Regards!

Chavira

LorenzoSpadaro · ‎02-13-2024

Hi Chavira, thank you for the answer!

I am using a custom board with a com module equipped with iMX8M.

Yesterday i replicated the problem sending a lot of bytes to the UART3, so i think the problem is on UART3, not UART4. The UART4 is working, so i think it is configured correctly.

Best regards

Lorenzo

Chavira · ‎02-13-2024

Hi @LorenzoSpadaro!

I compiled a simple program printing "Hello World" every second on UART3, UART2, and UART4 for 4 hours and I didn't get any errors.

Try to use the same configuration for UART3 like the iMX8MM-EVK.

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/arch/arm64/boot/dts/freescale/imx8mm-evk.dtsi#L72...

Best Regards!
Chavira