Kernel panic - not syncing: Asynchronous SError Interrupt

[muaxi8]

lx2160a custom board。ddr: 4GB.

 => pri bootcmd
bootcmd=sf probe 0:0; sf read 0x806c0000 0x6c0000 0x40000; env exists mcinitcmd && env exists secureboot && esbc_validate 0x806c0000; sf read 0x80d00000 0xd00000 0x100000; env exists mcinitcmd && fsl_mc lazyapply dpl 0x80d00000; run xspi_bootcmd;run distro_bootcmd; env exists secureboot && esbc_halt;

If i removed the string "fsl_mc lazyapply dpl 0x80d00000" of bootcmd ,the board can start up,but mc function can not be used。if i not removed that string, The following Memory Limit error occurred during kernel startup, the start log is attatchment. Can you provide any debugging ideas.

thanks

[ 3.618783] fsl_dpaa2_eth dpni.4: Adding to iommu group 2
[ 3.703332] fsl_dpaa2_eth dpni.3: Adding to iommu group 2
[ 3.787859] fsl_dpaa2_eth dpni.2: Adding to iommu group 2
[ 3.872372] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.956879] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 4.041417] dpaa2_mac dpmac.18: Adding to iommu group 2
[ 4.049862] SError Interrupt on CPU4, code 0xbf000000 -- SError
[ 4.049865] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049866] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049867] Workqueue: events deferred_probe_work_func
[ 4.049869] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 4.049871] pc : lpi_write_config+0xbc/0xd8
[ 4.049872] lr : lpi_update_config+0x30/0x80
[ 4.049873] sp : ffff8000101c3520
[ 4.049874] x29: ffff8000101c3520 x28: ffffa1171eabb1a8
[ 4.049876] x27: 0000000000000000 x26: 0000000000000000
[ 4.049879] x25: ffff0000210f2ca4 x24: ffff0000210f2d58
[ 4.049881] x23: 0000000000000000 x22: 0000000000000000
[ 4.049883] x21: ffff000021154b00 x20: ffff00002130a500
[ 4.049886] x19: ffffa1171f0788c8 x18: ffffffffffffffff
[ 4.049888] x17: 000000001a00eb9e x16: 000000005679f78c
[ 4.049891] x15: ffffa1171f0788c8 x14: ffff8000101c35b8
[ 4.049893] x13: ffff8000101c35af x12: 0000000000000000
[ 4.049895] x11: 0000000000000000 x10: ffff000067979900
[ 4.049897] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 4.049900] x7 : 0000000006030040 x6 : ffffa1171f07b8c8
[ 4.049902] x5 : 0000000000000000 x4 : ffff800010090000
[ 4.049904] x3 : 0000000000000000 x2 : 0000000000000003
[ 4.049906] x1 : 0000000000000000 x0 : ffff800010090000
[ 4.049909] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 4.049911] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049912] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049913] Workqueue: events deferred_probe_work_func
[ 4.049914] Call trace:
[ 4.049915] dump_backtrace+0x0/0x150
[ 4.049916] show_stack+0x14/0x20
[ 4.049917] dump_stack+0xbc/0x100
[ 4.049918] panic+0x16c/0x37c
[ 4.049919] __stack_chk_fail+0x0/0x18
[ 4.049920] arm64_serror_panic+0x74/0x88
[ 4.049921] do_serror+0x70/0x138
[ 4.049922] el1_error+0x84/0xf8
[ 4.049923] lpi_write_config+0xbc/0xd8
[ 4.049924] lpi_update_config+0x30/0x80
[ 4.049925] its_unmask_irq+0x34/0x40
[ 4.049926] irq_chip_unmask_parent+0x1c/0x28
[ 4.049927] unmask_irq.part.40+0x24/0x48
[ 4.049928] irq_enable+0x6c/0x88
[ 4.049929] __irq_startup+0x7c/0x88
[ 4.049929] irq_startup+0x5c/0x110
[ 4.049930] __setup_irq+0x770/0x7f8
[ 4.049931] request_threaded_irq+0xe0/0x198
[ 4.049932] devm_request_threaded_irq+0x78/0xf8
[ 4.049933] dpaa2_mac_probe+0x2e0/0x698
[ 4.049934] fsl_mc_driver_probe+0x20/0x68
[ 4.049935] really_probe+0x108/0x360
[ 4.049936] driver_probe_device+0x58/0x100
[ 4.049937] __device_attach_driver+0x90/0xb0
[ 4.049938] bus_for_each_drv+0x64/0xc8
[ 4.049939] __device_attach+0xd8/0x138
[ 4.049940] device_initial_probe+0x10/0x18
[ 4.049941] bus_probe_device+0x90/0x98
[ 4.049942] device_add+0x4ec/0x690
[ 4.049943] fsl_mc_device_add+0x1f4/0x520
[ 4.049944] fsl_mc_obj_device_add+0xc8/0x100
[ 4.049945] dprc_scan_objects+0x224/0x370
[ 4.049946] dprc_probe+0x138/0x3c0
[ 4.049947] fsl_mc_driver_probe+0x20/0x68
[ 4.049948] really_probe+0x108/0x360
[ 4.049949] driver_probe_device+0x58/0x100
[ 4.049950] __device_attach_driver+0x90/0xb0
[ 4.049951] bus_for_each_drv+0x64/0xc8
[ 4.049952] __device_attach+0xd8/0x138
[ 4.049953] device_initial_probe+0x10/0x18
[ 4.049954] bus_probe_device+0x90/0x98
[ 4.049955] deferred_probe_work_func+0x6c/0xa8
[ 4.049956] process_one_work+0x1e0/0x358
[ 4.049957] worker_thread+0x208/0x488
[ 4.049957] kthread+0x118/0x120
[ 4.049959] ret_from_fork+0x10/0x18
[ 4.049982] SMP: stopping secondary CPUs
[ 4.049983] Kernel Offset: 0x21170de00000 from 0xffff800010000000
[ 4.049985] PHYS_OFFSET: 0xfffff60040000000
[ 4.049986] CPU features: 0x0002,20806008
[ 4.049987] Memory Limit: none

[yipingwang]

Please check whether this problem remains with DPL file flexbuild_lsdk2012/packages/firmware/mc_utils/config/lx2160a/RDB/dpl-2dpni.dtb.

[muaxi8]

the problem is still remains:

[ 3.331092] fsl_mc_allocator dpmcp.5: Adding to iommu group 2
[ 3.337986] fsl_mc_allocator dpmcp.4: Adding to iommu group 2
[ 3.344857] fsl_mc_allocator dpmcp.3: Adding to iommu group 2
[ 3.351732] fsl_mc_allocator dpmcp.2: Adding to iommu group 2
[ 3.358616] fsl_mc_allocator dpmcp.1: Adding to iommu group 2
[ 3.364441] fsl_mc_allocator dpcon.1: Adding to iommu group 2
[ 3.370260] fsl_mc_allocator dpcon.0: Adding to iommu group 2
[ 3.376085] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.428339] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 3.483515] fsl_mc_dpio dpio.7: Adding to iommu group 2
[ 3.493459] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 3.493461] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493463] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493464] Workqueue: events deferred_probe_work_func
[ 3.493466] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 3.493467] pc : lpi_write_config+0xbc/0xd8
[ 3.493468] lr : lpi_update_config+0x30/0x80
[ 3.493469] sp : ffff8000101734d0
[ 3.493470] x29: ffff8000101734d0 x28: ffffa13314e703d8
[ 3.493473] x27: 0000000000000000 x26: 0000000000000000
[ 3.493475] x25: ffff000021a83cd4 x24: ffff000021a83d88
[ 3.493478] x23: 0000000000000000 x22: 0000000000000000
[ 3.493480] x21: ffff0000216b5200 x20: ffff000021054800
[ 3.493482] x19: ffffa13314e388c8 x18: ffffffffffffffff
[ 3.493485] x17: 00000000c12a7d13 x16: 00000000f7e81e37
[ 3.493487] x15: ffffa13314e388c8 x14: 000000081c091000
[ 3.493489] x13: ffff80001008d000 x12: ffffa13314839000
[ 3.493492] x11: 0000000000000020 x10: ffff5ecd52ee2000
[ 3.493494] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 3.493496] x7 : 00000000fffff040 x6 : ffffa13314e3c0b8
[ 3.493498] x5 : 0000000000000000 x4 : ffff800010090000
[ 3.493501] x3 : 0000000000000000 x2 : 0000000000000003
[ 3.493503] x1 : 0000000000000000 x0 : ffff800010090000
[ 3.493506] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 3.493507] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493508] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493509] Workqueue: events deferred_probe_work_func
[ 3.493511] Call trace:
[ 3.493512] dump_backtrace+0x0/0x150
[ 3.493513] show_stack+0x14/0x20
[ 3.493514] dump_stack+0xbc/0x100
[ 3.493515] panic+0x16c/0x37c
[ 3.493516] __stack_chk_fail+0x0/0x18
[ 3.493517] arm64_serror_panic+0x74/0x88
[ 3.493517] do_serror+0x70/0x138
[ 3.493518] el1_error+0x84/0xf8
[ 3.493519] lpi_write_config+0xbc/0xd8
[ 3.493520] lpi_update_config+0x30/0x80
[ 3.493521] its_unmask_irq+0x34/0x40
[ 3.493522] irq_chip_unmask_parent+0x1c/0x28
[ 3.493523] unmask_irq.part.40+0x24/0x48
[ 3.493524] irq_enable+0x6c/0x88
[ 3.493525] __irq_startup+0x7c/0x88
[ 3.493526] irq_startup+0x5c/0x110
[ 3.493527] __setup_irq+0x770/0x7f8
[ 3.493528] request_threaded_irq+0xe0/0x198
[ 3.493529] devm_request_threaded_irq+0x78/0xf8
[ 3.493530] dpaa2_dpio_probe+0x280/0x508
[ 3.493531] fsl_mc_driver_probe+0x20/0x68
[ 3.493532] really_probe+0x108/0x360
[ 3.493532] driver_probe_device+0x58/0x100
[ 3.493533] __device_attach_driver+0x90/0xb0
[ 3.493534] bus_for_each_drv+0x64/0xc8
[ 3.493535] __device_attach+0xd8/0x138
[ 3.493536] device_initial_probe+0x10/0x18
[ 3.493537] bus_probe_device+0x90/0x98
[ 3.493538] device_add+0x4ec/0x690
[ 3.493539] fsl_mc_device_add+0x1f4/0x520
[ 3.493540] fsl_mc_obj_device_add+0xc8/0x100
[ 3.493541] dprc_scan_objects+0x224/0x370
[ 3.493542] dprc_probe+0x138/0x3c0
[ 3.493543] fsl_mc_driver_probe+0x20/0x68
[ 3.493544] really_probe+0x108/0x360
[ 3.493545] driver_probe_device+0x58/0x100
[ 3.493546] __device_attach_driver+0x90/0xb0
[ 3.493547] bus_for_each_drv+0x64/0xc8
[ 3.493548] __device_attach+0xd8/0x138
[ 3.493549] device_initial_probe+0x10/0x18
[ 3.493550] bus_probe_device+0x90/0x98
[ 3.493551] deferred_probe_work_func+0x6c/0xa8
[ 3.493552] process_one_work+0x1e0/0x358
[ 3.493553] worker_thread+0x208/0x488
[ 3.493554] kthread+0x118/0x120
[ 3.493554] ret_from_fork+0x10/0x18
[ 3.493582] SMP: stopping secondary CPUs
[ 3.493583] Kernel Offset: 0x213303000000 from 0xffff800010000000
[ 3.493584] PHYS_OFFSET: 0xffffcefac0000000
[ 3.493585] CPU features: 0x0002,20806008
[ 3.493586] Memory Limit: none

[yipingwang]

From your logs it looks like you are using LSDK 1012, but your MC version is 10.29.100.
When you use default MC version which is - 10.24.0, do you still get this issue ?

[muaxi8]

I suspect it was an MC problem, so I upgraded to 10.29,but problem still remain

[yipingwang]

Can you send the full logs, so far you have shared the logs from u-boot shell

[muaxi8]

all log is attatched

[yipingwang]

Looking at your SError dump and objdump of vmlinux it seems that the issue happens while accessing -

if (gic_rdists->flags & RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING)

Now gic_rdists is allocated within gic subsystem and 'flags' member also doesn't seem to be affected by DPAA2 MAC.

Though i am still checking with the DPAA2 driver/MC team but my initial doubt is on the way GIC node is written in the dts file

So Can you share your dts/dtsi files where you have made changes related to both GIC and ethernet

Also i can see following log -
mdio@0x8B97000 node not found in dts
fsl-mc: Deploying data path layout ... SUCCESS
mdio@0x8B97000 node not found in dts

So are you not using EMI2 ?

[muaxi8]

hi , yiping Wang:

I can guarantee that GIC in the kernel device tree has not been changed. Attached is the device tree file i'm using。

this is a Asynchronous SError, The place where the error occurred is not necessarily the function that caused the error。

the print "mdio@0x8B97000 node not found in dts" is the part of uboot log.

mdio2 we used it,but the IN112525 is not used, so that  prints,  but that prints have no effect 

[yipingwang]

I would want to see if this is a problem only in the dpaa2-mac driver.
So could you please remove all the objects from the DPL file under dprc.1.
I suppose you will be using following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts

[muaxi8]

it looks like a interupt problem.

remove all the objects from the DPL file under dprc.1,  the log is attached.

yes,i used the following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts.

panic log:

[ 2.923007] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 2.923009] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923010] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923011] Workqueue: events deferred_probe_work_func
[ 2.923013] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 2.923014] pc : lpi_write_config+0xbc/0xd8
[ 2.923015] lr : lpi_update_config+0x30/0x80
[ 2.923015] sp : ffff800010173950
[ 2.923016] x29: ffff800010173950 x28: 0000000000000000
[ 2.923018] x27: 0000000000000000 x26: 0000000000000000
[ 2.923020] x25: ffff0000216bfad4 x24: ffff0000216bfb88
[ 2.923022] x23: 0000000000000000 x22: 0000000000000000
[ 2.923024] x21: ffff0000213e4200 x20: ffff0000213e0780
[ 2.923026] x19: ffffc9f2693788c8 x18: ffffffffffffffff
[ 2.923027] x17: 0000000024934c66 x16: 0000000025ffb868
[ 2.923029] x15: ffffc9f2693788c8 x14: ffff8000101739e8
[ 2.923031] x13: ffff8000101739df x12: 0000000000000000
[ 2.923033] x11: 0000000000000000 x10: ffff00006794bd00
[ 2.923034] x9 : 0000000000000000 x8 : 0000000000000000
[ 2.923036] x7 : ffff000065009200 x6 : ffffc9f26937c0a8
[ 2.923038] x5 : 0000000000000000 x4 : ffff800010090000
[ 2.923040] x3 : 0000000000000000 x2 : 0000000000000003
[ 2.923041] x1 : 0000000000000000 x0 : ffff800010090000
[ 2.923044] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2.923045] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923046] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923047] Workqueue: events deferred_probe_work_func
[ 2.923048] Call trace:
[ 2.923049] dump_backtrace+0x0/0x150
[ 2.923049] show_stack+0x14/0x20
[ 2.923050] dump_stack+0xbc/0x100
[ 2.923051] panic+0x16c/0x37c
[ 2.923052] __stack_chk_fail+0x0/0x18
[ 2.923052] arm64_serror_panic+0x74/0x88
[ 2.923053] do_serror+0x70/0x138
[ 2.923054] el1_error+0x84/0xf8
[ 2.923054] lpi_write_config+0xbc/0xd8
[ 2.923055] lpi_update_config+0x30/0x80
[ 2.923056] its_unmask_irq+0x34/0x40
[ 2.923057] irq_chip_unmask_parent+0x1c/0x28
[ 2.923058] unmask_irq.part.40+0x24/0x48
[ 2.923058] irq_enable+0x6c/0x88
[ 2.923059] __irq_startup+0x7c/0x88
[ 2.923060] irq_startup+0x5c/0x110
[ 2.923061] __setup_irq+0x770/0x7f8
[ 2.923061] request_threaded_irq+0xe0/0x198
[ 2.923062] devm_request_threaded_irq+0x78/0xf8
[ 2.923063] dprc_probe+0x1a0/0x3c0
[ 2.923064] fsl_mc_driver_probe+0x20/0x68
[ 2.923064] really_probe+0x108/0x360
[ 2.923065] driver_probe_device+0x58/0x100
[ 2.923066] __device_attach_driver+0x90/0xb0
[ 2.923067] bus_for_each_drv+0x64/0xc8
[ 2.923068] __device_attach+0xd8/0x138
[ 2.923068] device_initial_probe+0x10/0x18
[ 2.923069] bus_probe_device+0x90/0x98
[ 2.923070] deferred_probe_work_func+0x6c/0xa8
[ 2.923071] process_one_work+0x1e0/0x358
[ 2.923072] worker_thread+0x208/0x488
[ 2.923072] kthread+0x118/0x120
[ 2.923073] ret_from_fork+0x10/0x18
[ 2.923097] SMP: stopping secondary CPUs
[ 2.923098] Kernel Offset: 0x49f258000000 from 0xffff800010000000
[ 2.923099] PHYS_OFFSET: 0xffff81e880000000
[ 2.923100] CPU features: 0x0002,20806008
[ 2.923101] Memory Limit: none

[yipingwang]

The board is based on the LX2080ACE SoC which tells me that there are 8 A72 cores.

But the DTS file seems to be of LX2160A which has 16 cores described there.

Would you please try correcting the CPU entries in your dtsi ?

As there is a good chance that incorrect CPU entries can hit GIC operations.

[muaxi8 · Accepted solution]

Change the environment variable mcmemsize, the default value is 0x70000000, I will change it to 0x20000000, I don't know if it is because the DDR is too small, our DDR is 4GB

root@TinyLinux:~# ls-listmac
dprc.1/dpmac.18 (end point: dpni.3)
dprc.1/dpmac.17 (end point: dpni.2)
dprc.1/dpmac.16
dprc.1/dpmac.14 (end point: dpni.1)
dprc.1/dpmac.13 (end point: dpni.0)
dprc.1/dpmac.12

[muaxi8]

We also have a custom  board for 2160A, same problem，Attached is the log of 2160A。