topic Re: Kernel panic - not syncing: Asynchronous SError Interrupt in Layerscape

Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Tue, 07 Sep 2021 08:02:42 GMT

lx2160a custom board。ddr: 4GB.

=> pri bootcmd
bootcmd=sf probe 0:0; sf read 0x806c0000 0x6c0000 0x40000; env exists mcinitcmd && env exists secureboot && esbc_validate 0x806c0000; sf read 0x80d00000 0xd00000 0x100000; env exists mcinitcmd && fsl_mc lazyapply dpl 0x80d00000; run xspi_bootcmd;run distro_bootcmd; env exists secureboot && esbc_halt;

If i removed the string "fsl_mc lazyapply dpl 0x80d00000" of bootcmd ,the board can start up,but mc function can not be used。if i not removed that string, The following Memory Limit error occurred during kernel startup, the start log is attatchment. Can you provide any debugging ideas.

thanks

[ 3.618783] fsl_dpaa2_eth dpni.4: Adding to iommu group 2
[ 3.703332] fsl_dpaa2_eth dpni.3: Adding to iommu group 2
[ 3.787859] fsl_dpaa2_eth dpni.2: Adding to iommu group 2
[ 3.872372] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.956879] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 4.041417] dpaa2_mac dpmac.18: Adding to iommu group 2
[ 4.049862] SError Interrupt on CPU4, code 0xbf000000 -- SError
[ 4.049865] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049866] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049867] Workqueue: events deferred_probe_work_func
[ 4.049869] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 4.049871] pc : lpi_write_config+0xbc/0xd8
[ 4.049872] lr : lpi_update_config+0x30/0x80
[ 4.049873] sp : ffff8000101c3520
[ 4.049874] x29: ffff8000101c3520 x28: ffffa1171eabb1a8
[ 4.049876] x27: 0000000000000000 x26: 0000000000000000
[ 4.049879] x25: ffff0000210f2ca4 x24: ffff0000210f2d58
[ 4.049881] x23: 0000000000000000 x22: 0000000000000000
[ 4.049883] x21: ffff000021154b00 x20: ffff00002130a500
[ 4.049886] x19: ffffa1171f0788c8 x18: ffffffffffffffff
[ 4.049888] x17: 000000001a00eb9e x16: 000000005679f78c
[ 4.049891] x15: ffffa1171f0788c8 x14: ffff8000101c35b8
[ 4.049893] x13: ffff8000101c35af x12: 0000000000000000
[ 4.049895] x11: 0000000000000000 x10: ffff000067979900
[ 4.049897] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 4.049900] x7 : 0000000006030040 x6 : ffffa1171f07b8c8
[ 4.049902] x5 : 0000000000000000 x4 : ffff800010090000
[ 4.049904] x3 : 0000000000000000 x2 : 0000000000000003
[ 4.049906] x1 : 0000000000000000 x0 : ffff800010090000
[ 4.049909] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 4.049911] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049912] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049913] Workqueue: events deferred_probe_work_func
[ 4.049914] Call trace:
[ 4.049915] dump_backtrace+0x0/0x150
[ 4.049916] show_stack+0x14/0x20
[ 4.049917] dump_stack+0xbc/0x100
[ 4.049918] panic+0x16c/0x37c
[ 4.049919] __stack_chk_fail+0x0/0x18
[ 4.049920] arm64_serror_panic+0x74/0x88
[ 4.049921] do_serror+0x70/0x138
[ 4.049922] el1_error+0x84/0xf8
[ 4.049923] lpi_write_config+0xbc/0xd8
[ 4.049924] lpi_update_config+0x30/0x80
[ 4.049925] its_unmask_irq+0x34/0x40
[ 4.049926] irq_chip_unmask_parent+0x1c/0x28
[ 4.049927] unmask_irq.part.40+0x24/0x48
[ 4.049928] irq_enable+0x6c/0x88
[ 4.049929] __irq_startup+0x7c/0x88
[ 4.049929] irq_startup+0x5c/0x110
[ 4.049930] __setup_irq+0x770/0x7f8
[ 4.049931] request_threaded_irq+0xe0/0x198
[ 4.049932] devm_request_threaded_irq+0x78/0xf8
[ 4.049933] dpaa2_mac_probe+0x2e0/0x698
[ 4.049934] fsl_mc_driver_probe+0x20/0x68
[ 4.049935] really_probe+0x108/0x360
[ 4.049936] driver_probe_device+0x58/0x100
[ 4.049937] __device_attach_driver+0x90/0xb0
[ 4.049938] bus_for_each_drv+0x64/0xc8
[ 4.049939] __device_attach+0xd8/0x138
[ 4.049940] device_initial_probe+0x10/0x18
[ 4.049941] bus_probe_device+0x90/0x98
[ 4.049942] device_add+0x4ec/0x690
[ 4.049943] fsl_mc_device_add+0x1f4/0x520
[ 4.049944] fsl_mc_obj_device_add+0xc8/0x100
[ 4.049945] dprc_scan_objects+0x224/0x370
[ 4.049946] dprc_probe+0x138/0x3c0
[ 4.049947] fsl_mc_driver_probe+0x20/0x68
[ 4.049948] really_probe+0x108/0x360
[ 4.049949] driver_probe_device+0x58/0x100
[ 4.049950] __device_attach_driver+0x90/0xb0
[ 4.049951] bus_for_each_drv+0x64/0xc8
[ 4.049952] __device_attach+0xd8/0x138
[ 4.049953] device_initial_probe+0x10/0x18
[ 4.049954] bus_probe_device+0x90/0x98
[ 4.049955] deferred_probe_work_func+0x6c/0xa8
[ 4.049956] process_one_work+0x1e0/0x358
[ 4.049957] worker_thread+0x208/0x488
[ 4.049957] kthread+0x118/0x120
[ 4.049959] ret_from_fork+0x10/0x18
[ 4.049982] SMP: stopping secondary CPUs
[ 4.049983] Kernel Offset: 0x21170de00000 from 0xffff800010000000
[ 4.049985] PHYS_OFFSET: 0xfffff60040000000
[ 4.049986] CPU features: 0x0002,20806008
[ 4.049987] Memory Limit: none

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Thu, 09 Sep 2021 02:14:13 GMT

Please check whether this problem remains with DPL file flexbuild_lsdk2012/packages/firmware/mc_utils/config/lx2160a/RDB/dpl-2dpni.dtb.

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Thu, 09 Sep 2021 05:19:22 GMT

the problem is still remains:

[ 3.331092] fsl_mc_allocator dpmcp.5: Adding to iommu group 2
[ 3.337986] fsl_mc_allocator dpmcp.4: Adding to iommu group 2
[ 3.344857] fsl_mc_allocator dpmcp.3: Adding to iommu group 2
[ 3.351732] fsl_mc_allocator dpmcp.2: Adding to iommu group 2
[ 3.358616] fsl_mc_allocator dpmcp.1: Adding to iommu group 2
[ 3.364441] fsl_mc_allocator dpcon.1: Adding to iommu group 2
[ 3.370260] fsl_mc_allocator dpcon.0: Adding to iommu group 2
[ 3.376085] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.428339] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 3.483515] fsl_mc_dpio dpio.7: Adding to iommu group 2
[ 3.493459] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 3.493461] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493463] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493464] Workqueue: events deferred_probe_work_func
[ 3.493466] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 3.493467] pc : lpi_write_config+0xbc/0xd8
[ 3.493468] lr : lpi_update_config+0x30/0x80
[ 3.493469] sp : ffff8000101734d0
[ 3.493470] x29: ffff8000101734d0 x28: ffffa13314e703d8
[ 3.493473] x27: 0000000000000000 x26: 0000000000000000
[ 3.493475] x25: ffff000021a83cd4 x24: ffff000021a83d88
[ 3.493478] x23: 0000000000000000 x22: 0000000000000000
[ 3.493480] x21: ffff0000216b5200 x20: ffff000021054800
[ 3.493482] x19: ffffa13314e388c8 x18: ffffffffffffffff
[ 3.493485] x17: 00000000c12a7d13 x16: 00000000f7e81e37
[ 3.493487] x15: ffffa13314e388c8 x14: 000000081c091000
[ 3.493489] x13: ffff80001008d000 x12: ffffa13314839000
[ 3.493492] x11: 0000000000000020 x10: ffff5ecd52ee2000
[ 3.493494] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 3.493496] x7 : 00000000fffff040 x6 : ffffa13314e3c0b8
[ 3.493498] x5 : 0000000000000000 x4 : ffff800010090000
[ 3.493501] x3 : 0000000000000000 x2 : 0000000000000003
[ 3.493503] x1 : 0000000000000000 x0 : ffff800010090000
[ 3.493506] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 3.493507] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493508] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493509] Workqueue: events deferred_probe_work_func
[ 3.493511] Call trace:
[ 3.493512] dump_backtrace+0x0/0x150
[ 3.493513] show_stack+0x14/0x20
[ 3.493514] dump_stack+0xbc/0x100
[ 3.493515] panic+0x16c/0x37c
[ 3.493516] __stack_chk_fail+0x0/0x18
[ 3.493517] arm64_serror_panic+0x74/0x88
[ 3.493517] do_serror+0x70/0x138
[ 3.493518] el1_error+0x84/0xf8
[ 3.493519] lpi_write_config+0xbc/0xd8
[ 3.493520] lpi_update_config+0x30/0x80
[ 3.493521] its_unmask_irq+0x34/0x40
[ 3.493522] irq_chip_unmask_parent+0x1c/0x28
[ 3.493523] unmask_irq.part.40+0x24/0x48
[ 3.493524] irq_enable+0x6c/0x88
[ 3.493525] __irq_startup+0x7c/0x88
[ 3.493526] irq_startup+0x5c/0x110
[ 3.493527] __setup_irq+0x770/0x7f8
[ 3.493528] request_threaded_irq+0xe0/0x198
[ 3.493529] devm_request_threaded_irq+0x78/0xf8
[ 3.493530] dpaa2_dpio_probe+0x280/0x508
[ 3.493531] fsl_mc_driver_probe+0x20/0x68
[ 3.493532] really_probe+0x108/0x360
[ 3.493532] driver_probe_device+0x58/0x100
[ 3.493533] __device_attach_driver+0x90/0xb0
[ 3.493534] bus_for_each_drv+0x64/0xc8
[ 3.493535] __device_attach+0xd8/0x138
[ 3.493536] device_initial_probe+0x10/0x18
[ 3.493537] bus_probe_device+0x90/0x98
[ 3.493538] device_add+0x4ec/0x690
[ 3.493539] fsl_mc_device_add+0x1f4/0x520
[ 3.493540] fsl_mc_obj_device_add+0xc8/0x100
[ 3.493541] dprc_scan_objects+0x224/0x370
[ 3.493542] dprc_probe+0x138/0x3c0
[ 3.493543] fsl_mc_driver_probe+0x20/0x68
[ 3.493544] really_probe+0x108/0x360
[ 3.493545] driver_probe_device+0x58/0x100
[ 3.493546] __device_attach_driver+0x90/0xb0
[ 3.493547] bus_for_each_drv+0x64/0xc8
[ 3.493548] __device_attach+0xd8/0x138
[ 3.493549] device_initial_probe+0x10/0x18
[ 3.493550] bus_probe_device+0x90/0x98
[ 3.493551] deferred_probe_work_func+0x6c/0xa8
[ 3.493552] process_one_work+0x1e0/0x358
[ 3.493553] worker_thread+0x208/0x488
[ 3.493554] kthread+0x118/0x120
[ 3.493554] ret_from_fork+0x10/0x18
[ 3.493582] SMP: stopping secondary CPUs
[ 3.493583] Kernel Offset: 0x213303000000 from 0xffff800010000000
[ 3.493584] PHYS_OFFSET: 0xffffcefac0000000
[ 3.493585] CPU features: 0x0002,20806008
[ 3.493586] Memory Limit: none

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Fri, 10 Sep 2021 07:01:57 GMT

From your logs it looks like you are using LSDK 1012, but your MC version is 10.29.100.
When you use default MC version which is - 10.24.0, do you still get this issue ?

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Fri, 10 Sep 2021 07:20:00 GMT

I suspect it was an MC problem, so I upgraded to 10.29,but problem still remain

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Fri, 10 Sep 2021 07:55:07 GMT

Can you send the full logs, so far you have shared the logs from u-boot shell

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Fri, 10 Sep 2021 08:34:16 GMT

all log is attatched

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Mon, 13 Sep 2021 07:33:49 GMT

Looking at your SError dump and objdump of vmlinux it seems that the issue happens while accessing -

if (gic_rdists->flags & RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING)

Now gic_rdists is allocated within gic subsystem and 'flags' member also doesn't seem to be affected by DPAA2 MAC.

Though i am still checking with the DPAA2 driver/MC team but my initial doubt is on the way GIC node is written in the dts file

So Can you share your dts/dtsi files where you have made changes related to both GIC and ethernet

Also i can see following log -
mdio@0x8B97000 node not found in dts
fsl-mc: Deploying data path layout ... SUCCESS
mdio@0x8B97000 node not found in dts

So are you not using EMI2 ?

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Mon, 13 Sep 2021 09:13:24 GMT

hi , yiping Wang:

I can guarantee that GIC in the kernel device tree has not been changed. Attached is the device tree file i'm using。

this is a Asynchronous SError, The place where the error occurred is not necessarily the function that caused the error。

the print "mdio@0x8B97000 node not found in dts" is the part of uboot log.

mdio2 we used it,but the IN112525 is not used, so that prints, but that prints have no effect

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Wed, 15 Sep 2021 05:56:54 GMT

I would want to see if this is a problem only in the dpaa2-mac driver.
So could you please remove all the objects from the DPL file under dprc.1.
I suppose you will be using following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Thu, 16 Sep 2021 02:21:26 GMT

it looks like a interupt problem.

remove all the objects from the DPL file under dprc.1, the log is attached.

yes,i used the following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts.

panic log:

[ 2.923007] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 2.923009] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923010] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923011] Workqueue: events deferred_probe_work_func
[ 2.923013] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 2.923014] pc : lpi_write_config+0xbc/0xd8
[ 2.923015] lr : lpi_update_config+0x30/0x80
[ 2.923015] sp : ffff800010173950
[ 2.923016] x29: ffff800010173950 x28: 0000000000000000
[ 2.923018] x27: 0000000000000000 x26: 0000000000000000
[ 2.923020] x25: ffff0000216bfad4 x24: ffff0000216bfb88
[ 2.923022] x23: 0000000000000000 x22: 0000000000000000
[ 2.923024] x21: ffff0000213e4200 x20: ffff0000213e0780
[ 2.923026] x19: ffffc9f2693788c8 x18: ffffffffffffffff
[ 2.923027] x17: 0000000024934c66 x16: 0000000025ffb868
[ 2.923029] x15: ffffc9f2693788c8 x14: ffff8000101739e8
[ 2.923031] x13: ffff8000101739df x12: 0000000000000000
[ 2.923033] x11: 0000000000000000 x10: ffff00006794bd00
[ 2.923034] x9 : 0000000000000000 x8 : 0000000000000000
[ 2.923036] x7 : ffff000065009200 x6 : ffffc9f26937c0a8
[ 2.923038] x5 : 0000000000000000 x4 : ffff800010090000
[ 2.923040] x3 : 0000000000000000 x2 : 0000000000000003
[ 2.923041] x1 : 0000000000000000 x0 : ffff800010090000
[ 2.923044] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2.923045] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923046] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923047] Workqueue: events deferred_probe_work_func
[ 2.923048] Call trace:
[ 2.923049] dump_backtrace+0x0/0x150
[ 2.923049] show_stack+0x14/0x20
[ 2.923050] dump_stack+0xbc/0x100
[ 2.923051] panic+0x16c/0x37c
[ 2.923052] __stack_chk_fail+0x0/0x18
[ 2.923052] arm64_serror_panic+0x74/0x88
[ 2.923053] do_serror+0x70/0x138
[ 2.923054] el1_error+0x84/0xf8
[ 2.923054] lpi_write_config+0xbc/0xd8
[ 2.923055] lpi_update_config+0x30/0x80
[ 2.923056] its_unmask_irq+0x34/0x40
[ 2.923057] irq_chip_unmask_parent+0x1c/0x28
[ 2.923058] unmask_irq.part.40+0x24/0x48
[ 2.923058] irq_enable+0x6c/0x88
[ 2.923059] __irq_startup+0x7c/0x88
[ 2.923060] irq_startup+0x5c/0x110
[ 2.923061] __setup_irq+0x770/0x7f8
[ 2.923061] request_threaded_irq+0xe0/0x198
[ 2.923062] devm_request_threaded_irq+0x78/0xf8
[ 2.923063] dprc_probe+0x1a0/0x3c0
[ 2.923064] fsl_mc_driver_probe+0x20/0x68
[ 2.923064] really_probe+0x108/0x360
[ 2.923065] driver_probe_device+0x58/0x100
[ 2.923066] __device_attach_driver+0x90/0xb0
[ 2.923067] bus_for_each_drv+0x64/0xc8
[ 2.923068] __device_attach+0xd8/0x138
[ 2.923068] device_initial_probe+0x10/0x18
[ 2.923069] bus_probe_device+0x90/0x98
[ 2.923070] deferred_probe_work_func+0x6c/0xa8
[ 2.923071] process_one_work+0x1e0/0x358
[ 2.923072] worker_thread+0x208/0x488
[ 2.923072] kthread+0x118/0x120
[ 2.923073] ret_from_fork+0x10/0x18
[ 2.923097] SMP: stopping secondary CPUs
[ 2.923098] Kernel Offset: 0x49f258000000 from 0xffff800010000000
[ 2.923099] PHYS_OFFSET: 0xffff81e880000000
[ 2.923100] CPU features: 0x0002,20806008
[ 2.923101] Memory Limit: none

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

yipingwang — Fri, 17 Sep 2021 08:14:21 GMT

The board is based on the LX2080ACE SoC which tells me that there are 8 A72 cores.

But the DTS file seems to be of LX2160A which has 16 cores described there.

Would you please try correcting the CPU entries in your dtsi ?

As there is a good chance that incorrect CPU entries can hit GIC operations.

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Sat, 18 Sep 2021 02:18:17 GMT

We also have a custom board for 2160A, same problem，Attached is the log of 2160A。

Re: Kernel panic - not syncing: Asynchronous SError Interrupt

muaxi8 — Tue, 21 Sep 2021 10:21:21 GMT

Change the environment variable mcmemsize, the default value is 0x70000000, I will change it to 0x20000000, I don't know if it is because the DDR is too small, our DDR is 4GB

root@TinyLinux:~# ls-listmac
dprc.1/dpmac.18 (end point: dpni.3)
dprc.1/dpmac.17 (end point: dpni.2)
dprc.1/dpmac.16
dprc.1/dpmac.14 (end point: dpni.1)
dprc.1/dpmac.13 (end point: dpni.0)
dprc.1/dpmac.12