Kernel panic - not syncing: Asynchronous SError Interrupt

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Kernel panic - not syncing: Asynchronous SError Interrupt

Jump to solution
11,758 Views
muaxi8
Contributor V

lx2160a custom board。ddr: 4GB.

 => pri bootcmd
bootcmd=sf probe 0:0; sf read 0x806c0000 0x6c0000 0x40000; env exists mcinitcmd && env exists secureboot && esbc_validate 0x806c0000; sf read 0x80d00000 0xd00000 0x100000; env exists mcinitcmd && fsl_mc lazyapply dpl 0x80d00000; run xspi_bootcmd;run distro_bootcmd; env exists secureboot && esbc_halt;

If i removed the string "fsl_mc lazyapply dpl 0x80d00000" of bootcmd ,the board can start up,but mc function can not be used。if i not removed that string, The following Memory Limit error occurred during kernel startup, the start log is attatchment. Can you provide any debugging ideas.

thanks

[ 3.618783] fsl_dpaa2_eth dpni.4: Adding to iommu group 2
[ 3.703332] fsl_dpaa2_eth dpni.3: Adding to iommu group 2
[ 3.787859] fsl_dpaa2_eth dpni.2: Adding to iommu group 2
[ 3.872372] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.956879] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 4.041417] dpaa2_mac dpmac.18: Adding to iommu group 2
[ 4.049862] SError Interrupt on CPU4, code 0xbf000000 -- SError
[ 4.049865] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049866] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049867] Workqueue: events deferred_probe_work_func
[ 4.049869] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 4.049871] pc : lpi_write_config+0xbc/0xd8
[ 4.049872] lr : lpi_update_config+0x30/0x80
[ 4.049873] sp : ffff8000101c3520
[ 4.049874] x29: ffff8000101c3520 x28: ffffa1171eabb1a8
[ 4.049876] x27: 0000000000000000 x26: 0000000000000000
[ 4.049879] x25: ffff0000210f2ca4 x24: ffff0000210f2d58
[ 4.049881] x23: 0000000000000000 x22: 0000000000000000
[ 4.049883] x21: ffff000021154b00 x20: ffff00002130a500
[ 4.049886] x19: ffffa1171f0788c8 x18: ffffffffffffffff
[ 4.049888] x17: 000000001a00eb9e x16: 000000005679f78c
[ 4.049891] x15: ffffa1171f0788c8 x14: ffff8000101c35b8
[ 4.049893] x13: ffff8000101c35af x12: 0000000000000000
[ 4.049895] x11: 0000000000000000 x10: ffff000067979900
[ 4.049897] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 4.049900] x7 : 0000000006030040 x6 : ffffa1171f07b8c8
[ 4.049902] x5 : 0000000000000000 x4 : ffff800010090000
[ 4.049904] x3 : 0000000000000000 x2 : 0000000000000003
[ 4.049906] x1 : 0000000000000000 x0 : ffff800010090000
[ 4.049909] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 4.049911] CPU: 4 PID: 31 Comm: kworker/4:0 Not tainted 5.4.47-dirty #36
[ 4.049912] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 4.049913] Workqueue: events deferred_probe_work_func
[ 4.049914] Call trace:
[ 4.049915] dump_backtrace+0x0/0x150
[ 4.049916] show_stack+0x14/0x20
[ 4.049917] dump_stack+0xbc/0x100
[ 4.049918] panic+0x16c/0x37c
[ 4.049919] __stack_chk_fail+0x0/0x18
[ 4.049920] arm64_serror_panic+0x74/0x88
[ 4.049921] do_serror+0x70/0x138
[ 4.049922] el1_error+0x84/0xf8
[ 4.049923] lpi_write_config+0xbc/0xd8
[ 4.049924] lpi_update_config+0x30/0x80
[ 4.049925] its_unmask_irq+0x34/0x40
[ 4.049926] irq_chip_unmask_parent+0x1c/0x28
[ 4.049927] unmask_irq.part.40+0x24/0x48
[ 4.049928] irq_enable+0x6c/0x88
[ 4.049929] __irq_startup+0x7c/0x88
[ 4.049929] irq_startup+0x5c/0x110
[ 4.049930] __setup_irq+0x770/0x7f8
[ 4.049931] request_threaded_irq+0xe0/0x198
[ 4.049932] devm_request_threaded_irq+0x78/0xf8
[ 4.049933] dpaa2_mac_probe+0x2e0/0x698
[ 4.049934] fsl_mc_driver_probe+0x20/0x68
[ 4.049935] really_probe+0x108/0x360
[ 4.049936] driver_probe_device+0x58/0x100
[ 4.049937] __device_attach_driver+0x90/0xb0
[ 4.049938] bus_for_each_drv+0x64/0xc8
[ 4.049939] __device_attach+0xd8/0x138
[ 4.049940] device_initial_probe+0x10/0x18
[ 4.049941] bus_probe_device+0x90/0x98
[ 4.049942] device_add+0x4ec/0x690
[ 4.049943] fsl_mc_device_add+0x1f4/0x520
[ 4.049944] fsl_mc_obj_device_add+0xc8/0x100
[ 4.049945] dprc_scan_objects+0x224/0x370
[ 4.049946] dprc_probe+0x138/0x3c0
[ 4.049947] fsl_mc_driver_probe+0x20/0x68
[ 4.049948] really_probe+0x108/0x360
[ 4.049949] driver_probe_device+0x58/0x100
[ 4.049950] __device_attach_driver+0x90/0xb0
[ 4.049951] bus_for_each_drv+0x64/0xc8
[ 4.049952] __device_attach+0xd8/0x138
[ 4.049953] device_initial_probe+0x10/0x18
[ 4.049954] bus_probe_device+0x90/0x98
[ 4.049955] deferred_probe_work_func+0x6c/0xa8
[ 4.049956] process_one_work+0x1e0/0x358
[ 4.049957] worker_thread+0x208/0x488
[ 4.049957] kthread+0x118/0x120
[ 4.049959] ret_from_fork+0x10/0x18
[ 4.049982] SMP: stopping secondary CPUs
[ 4.049983] Kernel Offset: 0x21170de00000 from 0xffff800010000000
[ 4.049985] PHYS_OFFSET: 0xfffff60040000000
[ 4.049986] CPU features: 0x0002,20806008
[ 4.049987] Memory Limit: none

 

0 Kudos
1 Solution
11,606 Views
muaxi8
Contributor V

Change the environment variable mcmemsize, the default value is 0x70000000, I will change it to 0x20000000, I don't know if it is because the DDR is too small, our DDR is 4GB

root@TinyLinux:~# ls-listmac
dprc.1/dpmac.18 (end point: dpni.3)
dprc.1/dpmac.17 (end point: dpni.2)
dprc.1/dpmac.16
dprc.1/dpmac.14 (end point: dpni.1)
dprc.1/dpmac.13 (end point: dpni.0)
dprc.1/dpmac.12

View solution in original post

0 Kudos
13 Replies
11,733 Views
yipingwang
NXP TechSupport
NXP TechSupport

Please check whether this problem remains with DPL file flexbuild_lsdk2012/packages/firmware/mc_utils/config/lx2160a/RDB/dpl-2dpni.dtb.

0 Kudos
11,730 Views
muaxi8
Contributor V

the problem is still remains:

[ 3.331092] fsl_mc_allocator dpmcp.5: Adding to iommu group 2
[ 3.337986] fsl_mc_allocator dpmcp.4: Adding to iommu group 2
[ 3.344857] fsl_mc_allocator dpmcp.3: Adding to iommu group 2
[ 3.351732] fsl_mc_allocator dpmcp.2: Adding to iommu group 2
[ 3.358616] fsl_mc_allocator dpmcp.1: Adding to iommu group 2
[ 3.364441] fsl_mc_allocator dpcon.1: Adding to iommu group 2
[ 3.370260] fsl_mc_allocator dpcon.0: Adding to iommu group 2
[ 3.376085] fsl_dpaa2_eth dpni.1: Adding to iommu group 2
[ 3.428339] fsl_dpaa2_eth dpni.0: Adding to iommu group 2
[ 3.483515] fsl_mc_dpio dpio.7: Adding to iommu group 2
[ 3.493459] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 3.493461] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493463] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493464] Workqueue: events deferred_probe_work_func
[ 3.493466] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 3.493467] pc : lpi_write_config+0xbc/0xd8
[ 3.493468] lr : lpi_update_config+0x30/0x80
[ 3.493469] sp : ffff8000101734d0
[ 3.493470] x29: ffff8000101734d0 x28: ffffa13314e703d8
[ 3.493473] x27: 0000000000000000 x26: 0000000000000000
[ 3.493475] x25: ffff000021a83cd4 x24: ffff000021a83d88
[ 3.493478] x23: 0000000000000000 x22: 0000000000000000
[ 3.493480] x21: ffff0000216b5200 x20: ffff000021054800
[ 3.493482] x19: ffffa13314e388c8 x18: ffffffffffffffff
[ 3.493485] x17: 00000000c12a7d13 x16: 00000000f7e81e37
[ 3.493487] x15: ffffa13314e388c8 x14: 000000081c091000
[ 3.493489] x13: ffff80001008d000 x12: ffffa13314839000
[ 3.493492] x11: 0000000000000020 x10: ffff5ecd52ee2000
[ 3.493494] x9 : 15f102bd01018000 x8 : 0000000000000000
[ 3.493496] x7 : 00000000fffff040 x6 : ffffa13314e3c0b8
[ 3.493498] x5 : 0000000000000000 x4 : ffff800010090000
[ 3.493501] x3 : 0000000000000000 x2 : 0000000000000003
[ 3.493503] x1 : 0000000000000000 x0 : ffff800010090000
[ 3.493506] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 3.493507] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #54
[ 3.493508] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3.493509] Workqueue: events deferred_probe_work_func
[ 3.493511] Call trace:
[ 3.493512] dump_backtrace+0x0/0x150
[ 3.493513] show_stack+0x14/0x20
[ 3.493514] dump_stack+0xbc/0x100
[ 3.493515] panic+0x16c/0x37c
[ 3.493516] __stack_chk_fail+0x0/0x18
[ 3.493517] arm64_serror_panic+0x74/0x88
[ 3.493517] do_serror+0x70/0x138
[ 3.493518] el1_error+0x84/0xf8
[ 3.493519] lpi_write_config+0xbc/0xd8
[ 3.493520] lpi_update_config+0x30/0x80
[ 3.493521] its_unmask_irq+0x34/0x40
[ 3.493522] irq_chip_unmask_parent+0x1c/0x28
[ 3.493523] unmask_irq.part.40+0x24/0x48
[ 3.493524] irq_enable+0x6c/0x88
[ 3.493525] __irq_startup+0x7c/0x88
[ 3.493526] irq_startup+0x5c/0x110
[ 3.493527] __setup_irq+0x770/0x7f8
[ 3.493528] request_threaded_irq+0xe0/0x198
[ 3.493529] devm_request_threaded_irq+0x78/0xf8
[ 3.493530] dpaa2_dpio_probe+0x280/0x508
[ 3.493531] fsl_mc_driver_probe+0x20/0x68
[ 3.493532] really_probe+0x108/0x360
[ 3.493532] driver_probe_device+0x58/0x100
[ 3.493533] __device_attach_driver+0x90/0xb0
[ 3.493534] bus_for_each_drv+0x64/0xc8
[ 3.493535] __device_attach+0xd8/0x138
[ 3.493536] device_initial_probe+0x10/0x18
[ 3.493537] bus_probe_device+0x90/0x98
[ 3.493538] device_add+0x4ec/0x690
[ 3.493539] fsl_mc_device_add+0x1f4/0x520
[ 3.493540] fsl_mc_obj_device_add+0xc8/0x100
[ 3.493541] dprc_scan_objects+0x224/0x370
[ 3.493542] dprc_probe+0x138/0x3c0
[ 3.493543] fsl_mc_driver_probe+0x20/0x68
[ 3.493544] really_probe+0x108/0x360
[ 3.493545] driver_probe_device+0x58/0x100
[ 3.493546] __device_attach_driver+0x90/0xb0
[ 3.493547] bus_for_each_drv+0x64/0xc8
[ 3.493548] __device_attach+0xd8/0x138
[ 3.493549] device_initial_probe+0x10/0x18
[ 3.493550] bus_probe_device+0x90/0x98
[ 3.493551] deferred_probe_work_func+0x6c/0xa8
[ 3.493552] process_one_work+0x1e0/0x358
[ 3.493553] worker_thread+0x208/0x488
[ 3.493554] kthread+0x118/0x120
[ 3.493554] ret_from_fork+0x10/0x18
[ 3.493582] SMP: stopping secondary CPUs
[ 3.493583] Kernel Offset: 0x213303000000 from 0xffff800010000000
[ 3.493584] PHYS_OFFSET: 0xffffcefac0000000
[ 3.493585] CPU features: 0x0002,20806008
[ 3.493586] Memory Limit: none

 

0 Kudos
11,714 Views
yipingwang
NXP TechSupport
NXP TechSupport

From your logs it looks like you are using LSDK 1012, but your MC version is 10.29.100.
When you use default MC version which is - 10.24.0, do you still get this issue ?

0 Kudos
11,711 Views
muaxi8
Contributor V

I suspect it was an MC problem, so I upgraded to 10.29,but problem still remain

0 Kudos
11,708 Views
yipingwang
NXP TechSupport
NXP TechSupport

Can you send the full logs, so far you have shared the logs from u-boot shell

0 Kudos
11,705 Views
muaxi8
Contributor V

all log is attatched

0 Kudos
11,682 Views
yipingwang
NXP TechSupport
NXP TechSupport

Looking at your SError dump and objdump of vmlinux it seems that the issue happens while accessing -

if (gic_rdists->flags & RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING)

Now gic_rdists is allocated within gic subsystem and 'flags' member also doesn't seem to be affected by DPAA2 MAC.

Though i am still checking with the DPAA2 driver/MC team but my initial doubt is on the way GIC node is written in the dts file

So Can you share your dts/dtsi files where you have made changes related to both GIC and ethernet

Also i can see following log -
mdio@0x8B97000 node not found in dts
fsl-mc: Deploying data path layout ... SUCCESS
mdio@0x8B97000 node not found in dts

So are you not using EMI2 ?

0 Kudos
11,679 Views
muaxi8
Contributor V

hi , yiping Wang:

I can guarantee that GIC in the kernel device tree has not been changed. Attached is the device tree file i'm using。

this is a Asynchronous SError, The place where the error occurred is not necessarily the function that caused the error。

the print "mdio@0x8B97000 node not found in dts" is the part of uboot log.

mdio2 we used it,but the IN112525 is not used, so that  prints,  but that prints have no effect 

 

muaxi8_0-1631519498642.png

 

0 Kudos
11,660 Views
yipingwang
NXP TechSupport
NXP TechSupport

I would want to see if this is a problem only in the dpaa2-mac driver.
So could you please remove all the objects from the DPL file under dprc.1.
I suppose you will be using following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts

0 Kudos
11,657 Views
muaxi8
Contributor V

it looks like a interupt problem.

remove all the objects from the DPL file under dprc.1,  the log is attached.

yes,i used the following dpl file - packages/firmware/mc_utils/config/lx2160a/RDB/dpl-eth.19.dts.

panic log:

[ 2.923007] SError Interrupt on CPU2, code 0xbf000000 -- SError
[ 2.923009] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923010] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923011] Workqueue: events deferred_probe_work_func
[ 2.923013] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 2.923014] pc : lpi_write_config+0xbc/0xd8
[ 2.923015] lr : lpi_update_config+0x30/0x80
[ 2.923015] sp : ffff800010173950
[ 2.923016] x29: ffff800010173950 x28: 0000000000000000
[ 2.923018] x27: 0000000000000000 x26: 0000000000000000
[ 2.923020] x25: ffff0000216bfad4 x24: ffff0000216bfb88
[ 2.923022] x23: 0000000000000000 x22: 0000000000000000
[ 2.923024] x21: ffff0000213e4200 x20: ffff0000213e0780
[ 2.923026] x19: ffffc9f2693788c8 x18: ffffffffffffffff
[ 2.923027] x17: 0000000024934c66 x16: 0000000025ffb868
[ 2.923029] x15: ffffc9f2693788c8 x14: ffff8000101739e8
[ 2.923031] x13: ffff8000101739df x12: 0000000000000000
[ 2.923033] x11: 0000000000000000 x10: ffff00006794bd00
[ 2.923034] x9 : 0000000000000000 x8 : 0000000000000000
[ 2.923036] x7 : ffff000065009200 x6 : ffffc9f26937c0a8
[ 2.923038] x5 : 0000000000000000 x4 : ffff800010090000
[ 2.923040] x3 : 0000000000000000 x2 : 0000000000000003
[ 2.923041] x1 : 0000000000000000 x0 : ffff800010090000
[ 2.923044] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2.923045] CPU: 2 PID: 21 Comm: kworker/2:0 Not tainted 5.4.47-dirty #82
[ 2.923046] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 2.923047] Workqueue: events deferred_probe_work_func
[ 2.923048] Call trace:
[ 2.923049] dump_backtrace+0x0/0x150
[ 2.923049] show_stack+0x14/0x20
[ 2.923050] dump_stack+0xbc/0x100
[ 2.923051] panic+0x16c/0x37c
[ 2.923052] __stack_chk_fail+0x0/0x18
[ 2.923052] arm64_serror_panic+0x74/0x88
[ 2.923053] do_serror+0x70/0x138
[ 2.923054] el1_error+0x84/0xf8
[ 2.923054] lpi_write_config+0xbc/0xd8
[ 2.923055] lpi_update_config+0x30/0x80
[ 2.923056] its_unmask_irq+0x34/0x40
[ 2.923057] irq_chip_unmask_parent+0x1c/0x28
[ 2.923058] unmask_irq.part.40+0x24/0x48
[ 2.923058] irq_enable+0x6c/0x88
[ 2.923059] __irq_startup+0x7c/0x88
[ 2.923060] irq_startup+0x5c/0x110
[ 2.923061] __setup_irq+0x770/0x7f8
[ 2.923061] request_threaded_irq+0xe0/0x198
[ 2.923062] devm_request_threaded_irq+0x78/0xf8
[ 2.923063] dprc_probe+0x1a0/0x3c0
[ 2.923064] fsl_mc_driver_probe+0x20/0x68
[ 2.923064] really_probe+0x108/0x360
[ 2.923065] driver_probe_device+0x58/0x100
[ 2.923066] __device_attach_driver+0x90/0xb0
[ 2.923067] bus_for_each_drv+0x64/0xc8
[ 2.923068] __device_attach+0xd8/0x138
[ 2.923068] device_initial_probe+0x10/0x18
[ 2.923069] bus_probe_device+0x90/0x98
[ 2.923070] deferred_probe_work_func+0x6c/0xa8
[ 2.923071] process_one_work+0x1e0/0x358
[ 2.923072] worker_thread+0x208/0x488
[ 2.923072] kthread+0x118/0x120
[ 2.923073] ret_from_fork+0x10/0x18
[ 2.923097] SMP: stopping secondary CPUs
[ 2.923098] Kernel Offset: 0x49f258000000 from 0xffff800010000000
[ 2.923099] PHYS_OFFSET: 0xffff81e880000000
[ 2.923100] CPU features: 0x0002,20806008
[ 2.923101] Memory Limit: none

0 Kudos
11,642 Views
yipingwang
NXP TechSupport
NXP TechSupport

The board is based on the LX2080ACE SoC which tells me that there are 8 A72 cores.

But the DTS file seems to be of LX2160A which has 16 cores described there.

Would you please try correcting the CPU entries in your dtsi ?

As there is a good chance that incorrect CPU entries can hit GIC operations.

0 Kudos
11,607 Views
muaxi8
Contributor V

Change the environment variable mcmemsize, the default value is 0x70000000, I will change it to 0x20000000, I don't know if it is because the DDR is too small, our DDR is 4GB

root@TinyLinux:~# ls-listmac
dprc.1/dpmac.18 (end point: dpni.3)
dprc.1/dpmac.17 (end point: dpni.2)
dprc.1/dpmac.16
dprc.1/dpmac.14 (end point: dpni.1)
dprc.1/dpmac.13 (end point: dpni.0)
dprc.1/dpmac.12

0 Kudos
11,623 Views
muaxi8
Contributor V

We also have a custom  board for 2160A, same problem,Attached is the log of 2160A。

0 Kudos