Configure LX2160 dpaa2 configuration and causes kernel dpio internal error

[Jeffrey_Lai]

Hi, 

We use dynamic_dpl.sh script to configure dpaa2 configuration and it causes irq/dpio internal error on CPU15.

The system is built by LSDK20.04 (with kernel 5.4 and 4.19) and network configuration as the following.

root@localhost:~# ls-listni
dprc.1/dpni.1 (interface: eth0, end point: dpmac.3)
dprc.1/dpni.0 (interface: eth1, end point: dpmac.6)
root@localhost:~# ls-listmac
dprc.1/dpmac.17
dprc.1/dpmac.6 (end point: dpni.0)  <--- XFI
dprc.1/dpmac.5
dprc.1/dpmac.4 
dprc.1/dpmac.3 (end point: dpni.1)  <--- XGMII
dprc.1/dpmac.2

Both dpni interface are link and use ssh to login system via eth0(dpni.1) and execute the following command.

echo dpni.0 > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind
restool dpni destroy dpni.0

/usr/local/dpdk/dpaa2/dynamic_dpl.sh dpni dpni

After executed command and the kernel shows internal error about irq/dpio

Jun 5 10:22:21 localhost <info> kernel: [ 77.728739] fsl_dpaa2_eth dpni.0 eth1: Link Event: state down
Jun 5 10:22:21 localhost <info> kernel: [ 78.434085] fsl_dpaa2_eth dpni.0: Dropping the link to dpbp.0
Jun 5 10:22:21 localhost <info> kernel: [ 78.434095] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.15
Jun 5 10:22:21 localhost <info> kernel: [ 78.435104] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.13
Jun 5 10:22:21 localhost <info> kernel: [ 78.435109] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.14
Jun 5 10:22:21 localhost <info> kernel: [ 78.436180] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.12
Jun 5 10:22:21 localhost <info> kernel: [ 78.436185] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.13
Jun 5 10:22:21 localhost <info> kernel: [ 78.437217] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.11
Jun 5 10:22:21 localhost <info> kernel: [ 78.437226] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.12
Jun 5 10:22:21 localhost <info> kernel: [ 78.438235] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.10
Jun 5 10:22:21 localhost <info> kernel: [ 78.438240] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.11
Jun 5 10:22:21 localhost <info> kernel: [ 78.439292] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.9
Jun 5 10:22:21 localhost <info> kernel: [ 78.439297] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.10
Jun 5 10:22:21 localhost <info> kernel: [ 78.440304] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.8
Jun 5 10:22:21 localhost <info> kernel: [ 78.440308] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.9
Jun 5 10:22:21 localhost <info> kernel: [ 78.441344] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.7
Jun 5 10:22:21 localhost <info> kernel: [ 78.441351] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.8
Jun 5 10:22:21 localhost <info> kernel: [ 78.442405] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.6
Jun 5 10:22:21 localhost <info> kernel: [ 78.442410] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.7
Jun 5 10:22:21 localhost <info> kernel: [ 78.443417] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.5
Jun 5 10:22:21 localhost <info> kernel: [ 78.443421] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.6
Jun 5 10:22:21 localhost <info> kernel: [ 78.444428] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.4
Jun 5 10:22:21 localhost <info> kernel: [ 78.444432] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.5
Jun 5 10:22:21 localhost <info> kernel: [ 78.445459] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.3
Jun 5 10:22:21 localhost <info> kernel: [ 78.445466] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.4
Jun 5 10:22:21 localhost <info> kernel: [ 78.446501] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.2
Jun 5 10:22:21 localhost <info> kernel: [ 78.446506] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.3
Jun 5 10:22:21 localhost <info> kernel: [ 78.447513] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.1
Jun 5 10:22:21 localhost <info> kernel: [ 78.447517] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.2
Jun 5 10:22:21 localhost <info> kernel: [ 78.448524] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.0
Jun 5 10:22:21 localhost <info> kernel: [ 78.448528] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.1
Jun 5 10:22:21 localhost <info> kernel: [ 78.449570] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.31
Jun 5 10:22:21 localhost <info> kernel: [ 78.449577] fsl_dpaa2_eth dpni.0: Dropping the link to dpio.0
Jun 5 10:22:22 localhost <info> kernel: [ 78.450598] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.30
Jun 5 10:22:22 localhost <info> kernel: [ 78.489609] fsl_dpaa2_eth dpni.0: Dropping the link to dpmcp.8
Jun 5 10:22:22 localhost <info> kernel: [ 78.564590] iommu: Removing device dpni.0 from group 11
Jun 5 10:22:22 localhost <warning> kernel: [ 78.742824] ttyAMA ttyAMA1: 2 input overrun(s)
Jun 5 10:22:22 localhost <alert> kernel: [ 79.308355] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000048
Jun 5 10:22:22 localhost <alert> kernel: [ 79.317130] Mem abort info:
Jun 5 10:22:22 localhost <alert> kernel: [ 79.319912] ESR = 0x96000004
Jun 5 10:22:22 localhost <alert> kernel: [ 79.322954] Exception class = DABT (current EL), IL = 32 bits
Jun 5 10:22:22 localhost <alert> kernel: [ 79.328860] SET = 0, FnV = 0
Jun 5 10:22:22 localhost <alert> kernel: [ 79.331901] EA = 0, S1PTW = 0
Jun 5 10:22:22 localhost <alert> kernel: [ 79.335029] Data abort info:
Jun 5 10:22:22 localhost <alert> kernel: [ 79.337897] ISV = 0, ISS = 0x00000004
Jun 5 10:22:22 localhost <alert> kernel: [ 79.341720] CM = 0, WnR = 0
Jun 5 10:22:22 localhost <alert> kernel: [ 79.344676] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000062724626
Jun 5 10:22:22 localhost <alert> kernel: [ 79.351277] [0000000000000048] pgd=0000000000000000
Jun 5 10:22:22 localhost <emerg> kernel: [ 79.356144] Internal error: Oops: 96000004 [#1] PREEMPT SMP
Jun 5 10:22:22 localhost <emerg> kernel: [ 79.356152] Process irq/139-dpio.0 (pid: 2300, stack limit = 0x0000000024b480eb)
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356155] CPU: 15 PID: 2300 Comm: irq/139-dpio.0 Kdump: loaded Tainted: G O 4.19.90-rt35 #1
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356156] Hardware name: NXP Layerscape LX2160ARDB (DT)
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356157] pstate: 00000005 (nzcv daif -PAN -UAO)
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356165] pc : dpaa2_eth_poll+0xe0/0x318
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356166] lr : dpaa2_eth_poll+0x100/0x318
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356167] sp : ffff00001222bba0
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356168] x29: ffff00001222bba0 x28: ffff8022ec170800
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356170] x27: 0000000000000000 x26: ffff8022ed1b4980
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356171] x25: 0000000000000000 x24: 0000000000000040
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356173] x23: 0000000000000000 x22: 0000000000000000
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356174] x21: ffff8022ed1b4980 x20: 0000000000000000
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356176] x19: 0000000000000001 x18: 0000000000000000
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356177] x17: 0000000000000001 x16: 0000000000000000
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356179] x15: 0000000000000400 x14: 0000000000000400
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356180] x13: 0000000000000400 x12: 0000000000000000
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356182] x11: 0000000000000002 x10: 0000000000000a60
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356183] x9 : ffff00001222bd40 x8 : 0000000004ffffff
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356185] x7 : ffff000012210780 x6 : 000000000000000f
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356186] x5 : 0000000000000001 x4 : 0000000000000080
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356187] x3 : 0000000000000000 x2 : ffff8022ec3c2820
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356189] x1 : ffff8022ec170800 x0 : ffff8022ec3c2800
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356191] Call trace:
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356193] dpaa2_eth_poll+0xe0/0x318
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356197] net_rx_action+0x1a4/0x358
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356201] do_current_softirqs+0x19c/0x240
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356203] __local_bh_enable+0x54/0x70
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356205] irq_forced_thread_fn+0x74/0xa0
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356206] irq_thread+0x114/0x1c0
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356208] kthread+0x128/0x130
Jun 5 10:22:22 localhost <warning> kernel: [ 79.356211] ret_from_fork+0x10/0x18
Jun 5 10:22:22 localhost <emerg> kernel: [ 79.356213] Code: 11000673 52800014 aa1c03e1 aa1b03e3 (f9402766)
Jun 5 10:22:22 localhost <warning> kernel: [ 79.375231] ---[ end trace 0000000000000002 ]---
Jun 5 10:22:22 localhost <err> kernel: [ 79.375240] genirq: exiting task "irq/139-dpio.0" (2300) is an active IRQ thread (irq 139)
Jun 5 10:22:32 localhost <warning> kernel: [ 89.053210] vfio-fsl-mc dprc.2: DMA mask not set

After trace kernel source but don't know the root cause.

Did  you get any similar issue and how to fix it ?

Thank you,

Jeffrey 

[yipingwang]

I didn't reproduce the issue with LSDK-20.04 on LX2160ARDB board.

1. Could you please share the full log?
2. please diconnect dpmac.6 which connect with dpni.0 from dprc.1 before unbind dpni.0, and try it again.
restool dprc disconnect dprc.1 --endpoint=dpmac.6
echo dpni.0 > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind
restool dpni destroy dpni.0

[Jeffrey_Lai]

Hi yipingwang,

If I use your command, the system halted and also shows the same error message.

The attachment is kernel log that I run restool command as following.

echo dpni.0 > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind

restool dpni destroy dpni.0

/usr/local/dpdk/dpaa2/dynamic_dpl.sh dpni dpni

[yipingwang]

I still didn't reproduce the issue.
could you please provide mc version and mc log?

Enable MC log in dpc.file as below.

mc_general {
/* Configure the DDR log */
log {
mode = "LOG_MODE_ON";
level = "LOG_LEVEL_DEBUG";
};

[Jeffrey_Lai]

Hi yipingwang,

After traced dynamic_dpl.sh, there is DPNI_OPTIONS and I fount out DPNI_OPT_SINGLE_SENDER option causes the issue.

If remove DPNI_OPT_SINGLE_SENDER in DPNI_OPTIONS and run dynamic_dpl.sh dpni dpni, the system does not show any error message.