VFIO_FLS_MC: reset device has failed

[Chidananda22]

LX2160 : I am getting below error while running my application.

After reboot works fine . Next time same error again. This problem seen if i configure as below 

#export DPIO_COUNT=20
set -x
#export DPIO_COUNT=10
#
ls-listni #to check if 40G is enabled or not
#Remove 40G port if it is enabled (40G=dpmac.2)
echo dpni.1 > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind
restool dpni destroy dpni.1

echo 0 > /proc/sys/kernel/randomize_va_space
#export DPIO_COUNT=20
export DPMCP_COUNT=3
export FS_ENTRIES=12
#/usr/local/dpdk/dpaa2/dynamic_dpl.sh dpmac.3
/usr/local/dpdk/dpaa2/dynamic_dpl.sh dpni dpni -b 00:00:00:00:05:00
# dpni.1 00:00:00:00:05:01
# dpni.2 00:00:00:00:05:02
ls-addni --no-link
#eth0 dpni.3
ls-addni --no-link
#eth2 dpni.4
#/////OUTPUT/////////
# Unbinding dprc.2 from VFIO
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/unbind

#dpn.2 is the output of dynamicdpl.sh
#dpni.1 dpmac.4 -Dynamic-
#
#restool dprc disconnect dprc.2 --endpoint=dpni.1

restool dpdmux create --num-ifs=3 --method DPDMUX_METHOD_MAC --max-dmat-entries=3 --manip=DPDMUX_MANIP_NONE
restool dprc connect dprc.1 --endpoint1=dpmac.3 --endpoint2=dpdmux.0.0
restool dprc connect dprc.1 --endpoint1=dpni.1 --endpoint2=dpdmux.0.1
restool dprc connect dprc.1 --endpoint1=dpni.3 --endpoint2=dpdmux.0.2

restool dpdmux create --num-ifs=3 --method DPDMUX_METHOD_MAC --max-dmat-entries=3 --manip=DPDMUX_MANIP_NONE
restool dprc connect dprc.1 --endpoint1=dpmac.4 --endpoint2=dpdmux.1.0
restool dprc connect dprc.1 --endpoint1=dpni.2 --endpoint2=dpdmux.1.1
restool dprc connect dprc.1 --endpoint1=dpni.4 --endpoint2=dpdmux.1.2

#echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

# Bind the DPRC back to VFIO
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
# Export the DPRC
export DPRC=dprc.2

[ 3103.545665] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.545690] ------------[ cut here ]------------
[ 3103.545701] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.545703] Modules linked in: libdes mali_dp
[ 3103.545715] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Not tainted 5.10.35 #5
[ 3103.545717] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.545721] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.545724] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.545727] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.545729] sp : ffff800012ef3c90
[ 3103.545731] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.545735] x27: 0000000000000000 x26: 0000000000000000
[ 3103.545740] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.545745] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.545749] x21: ffff0e7d1edc7800 x20: ffff0e7d3d39f040
[ 3103.545754] x19: ffff0e7d1d71db80 x18: 0000000000000001
[ 3103.545758] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.545763] x15: ffff0e7d3d39f040 x14: 00000000000007dd
[ 3103.545767] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.545771] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.545776] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.545781] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.545785] x5 : 0000000000000000 x4 : ffff0e83fc27d860
[ 3103.545789] x3 : ffff0e83fc284770 x2 : 0000000000000000
[ 3103.545793] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.545798] Call trace:
[ 3103.545802] vfio_fsl_mc_release+0xdc/0x190
[ 3103.545806] vfio_device_fops_release+0x24/0x48
[ 3103.545811] __fput+0x78/0x230
[ 3103.545814] ____fput+0x10/0x20
[ 3103.545818] task_work_run+0x80/0x140
[ 3103.545821] do_exit+0x324/0xa08
[ 3103.545824] do_group_exit+0x44/0xa0
[ 3103.545826] __wake_up_parent+0x0/0x30
[ 3103.545831] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.545834] do_el0_svc+0x24/0x90
[ 3103.545839] el0_svc+0x14/0x20
[ 3103.545841] el0_sync_handler+0xb0/0xb8
[ 3103.545845] el0_sync+0x178/0x180
[ 3103.545847] ---[ end trace d15991a0fbaef713 ]---
[ 3103.546381] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.546399] ------------[ cut here ]------------
[ 3103.546406] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.546411] Modules linked in: libdes mali_dp
[ 3103.546430] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.546435] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.546441] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.546447] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.546453] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.546458] sp : ffff800012ef3c90
[ 3103.546462] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.546466] x27: 0000000000000000 x26: 0000000000000000
[ 3103.546471] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.546475] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.546479] x21: ffff0e7d02d77000 x20: ffff0e7d3d39f040
[ 3103.546484] x19: ffff0e7d1dc13480 x18: ffffb28536ab3470
[ 3103.546488] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.546492] x15: ffff0e7d3d39f040 x14: 0000000000000805
[ 3103.546497] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.546501] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.546506] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.546510] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.546514] x5 : 0000000000000000 x4 : ffff0e83fc27d860
[ 3103.546518] x3 : ffff0e83fc284770 x2 : 0000000000000000
[ 3103.546522] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.546527] Call trace:
[ 3103.546529] vfio_fsl_mc_release+0xdc/0x190
[ 3103.546532] vfio_device_fops_release+0x24/0x48
[ 3103.546536] __fput+0x78/0x230
[ 3103.546539] ____fput+0x10/0x20
[ 3103.546541] task_work_run+0x80/0x140
[ 3103.546544] do_exit+0x324/0xa08
[ 3103.546546] do_group_exit+0x44/0xa0
[ 3103.546548] __wake_up_parent+0x0/0x30
[ 3103.546552] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.546555] do_el0_svc+0x24/0x90
[ 3103.546557] el0_svc+0x14/0x20
[ 3103.546559] el0_sync_handler+0xb0/0xb8
[ 3103.546562] el0_sync+0x178/0x180
[ 3103.546564] ---[ end trace d15991a0fbaef714 ]---
[ 3103.547072] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.547086] ------------[ cut here ]------------
[ 3103.547090] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.547091] Modules linked in: libdes mali_dp
[ 3103.547098] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.547100] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.547103] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.547105] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.547107] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.547109] sp : ffff800012ef3c90
[ 3103.547111] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.547116] x27: 0000000000000000 x26: 0000000000000000
[ 3103.547120] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.547125] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.547129] x21: ffff0e7d02d32000 x20: ffff0e7d3d39f040
[ 3103.547133] x19: ffff0e7d1dc13d80 x18: ffffb28536ab3470
[ 3103.547138] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.547142] x15: ffff0e7d3d39f040 x14: 000000000000082d
[ 3103.547146] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.547151] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.547155] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.547159] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.547164] x5 : ffff0e83fc27d860 x4 : 0000000000000000
[ 3103.547168] x3 : 0000000000000027 x2 : 0000000000000000
[ 3103.547172] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.547176] Call trace:
[ 3103.547179] vfio_fsl_mc_release+0xdc/0x190
[ 3103.547182] vfio_device_fops_release+0x24/0x48
[ 3103.547185] __fput+0x78/0x230
[ 3103.547188] ____fput+0x10/0x20
[ 3103.547190] task_work_run+0x80/0x140
[ 3103.547193] do_exit+0x324/0xa08
[ 3103.547195] do_group_exit+0x44/0xa0
[ 3103.547198] __wake_up_parent+0x0/0x30
[ 3103.547201] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.547204] do_el0_svc+0x24/0x90
[ 3103.547207] el0_svc+0x14/0x20
[ 3103.547209] el0_sync_handler+0xb0/0xb8
[ 3103.547211] el0_sync+0x178/0x180
[ 3103.547213] ---[ end trace d15991a0fbaef715 ]---
[ 3103.547721] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.547735] ------------[ cut here ]------------
[ 3103.547739] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.547740] Modules linked in: libdes mali_dp
[ 3103.547748] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.547750] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.547752] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.547755] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.547757] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.547759] sp : ffff800012ef3c90
[ 3103.547761] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.547765] x27: 0000000000000000 x26: 0000000000000000
[ 3103.547770] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.547777] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.547781] x21: ffff0e7d1e989000 x20: ffff0e7d3d39f040
[ 3103.547785] x19: ffff0e7d1dcc6680 x18: ffffb28536ab3470
[ 3103.547790] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.547794] x15: ffff0e7d3d39f040 x14: 0000000000000855
[ 3103.547798] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.547802] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.547807] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.547812] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.547819] x5 : ffff0e83fc27d860 x4 : 0000000000000000
[ 3103.547823] x3 : 0000000000000027 x2 : 0000000000000000
[ 3103.547827] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.547831] Call trace:
[ 3103.547834] vfio_fsl_mc_release+0xdc/0x190
[ 3103.547837] vfio_device_fops_release+0x24/0x48
[ 3103.547840] __fput+0x78/0x230
[ 3103.547843] ____fput+0x10/0x20
[ 3103.547845] task_work_run+0x80/0x140
[ 3103.547848] do_exit+0x324/0xa08
[ 3103.547850] do_group_exit+0x44/0xa0
[ 3103.547853] __wake_up_parent+0x0/0x30
[ 3103.547858] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.547861] do_el0_svc+0x24/0x90
[ 3103.547864] el0_svc+0x14/0x20
[ 3103.547866] el0_sync_handler+0xb0/0xb8
[ 3103.547868] el0_sync+0x178/0x180
[ 3103.547870] ---[ end trace d15991a0fbaef716 ]---
[ 3103.548378] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.548392] ------------[ cut here ]------------
[ 3103.548395] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.548397] Modules linked in: libdes mali_dp
[ 3103.548404] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.548406] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.548408] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.548411] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.548413] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.548415] sp : ffff800012ef3c90
[ 3103.548417] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.548421] x27: 0000000000000000 x26: 0000000000000000
[ 3103.548426] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.548430] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.548437] x21: ffff0e7d1ea4e000 x20: ffff0e7d3d39f040
[ 3103.548442] x19: ffff0e7d1dcc6f80 x18: ffffb28536ab3470
[ 3103.548446] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.548450] x15: ffff0e7d3d39f040 x14: 000000000000087d
[ 3103.548455] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.548459] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.548464] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.548468] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.548472] x5 : ffff0e83fc27d860 x4 : 0000000000000000
[ 3103.548479] x3 : 0000000000000027 x2 : 0000000000000000
[ 3103.548483] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.548488] Call trace:
[ 3103.548490] vfio_fsl_mc_release+0xdc/0x190
[ 3103.548493] vfio_device_fops_release+0x24/0x48
[ 3103.548497] __fput+0x78/0x230
[ 3103.548500] ____fput+0x10/0x20
[ 3103.548502] task_work_run+0x80/0x140
[ 3103.548505] do_exit+0x324/0xa08
[ 3103.548507] do_group_exit+0x44/0xa0
[ 3103.548509] __wake_up_parent+0x0/0x30
[ 3103.548513] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.548516] do_el0_svc+0x24/0x90
[ 3103.548519] el0_svc+0x14/0x20
[ 3103.548523] el0_sync_handler+0xb0/0xb8
[ 3103.548526] el0_sync+0x178/0x180
[ 3103.548528] ---[ end trace d15991a0fbaef717 ]---
[ 3103.549035] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.549049] ------------[ cut here ]------------
[ 3103.549052] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.549054] Modules linked in: libdes mali_dp
[ 3103.549060] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.549062] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.549065] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.549067] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.549069] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.549071] sp : ffff800012ef3c90
[ 3103.549073] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.549077] x27: 0000000000000000 x26: 0000000000000000
[ 3103.549082] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.549086] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.549090] x21: ffff0e7d1dd80000 x20: ffff0e7d3d39f040
[ 3103.549094] x19: ffff0e7d1dc09880 x18: ffffb28536ab3470
[ 3103.549098] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.549103] x15: ffff0e7d3d39f040 x14: 00000000000008a5
[ 3103.549107] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.549111] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.549116] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.549120] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.549124] x5 : ffff0e83fc27d860 x4 : 0000000000000000
[ 3103.549129] x3 : 0000000000000027 x2 : 0000000000000000
[ 3103.549133] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.549137] Call trace:
[ 3103.549139] vfio_fsl_mc_release+0xdc/0x190
[ 3103.549142] vfio_device_fops_release+0x24/0x48
[ 3103.549145] __fput+0x78/0x230
[ 3103.549148] ____fput+0x10/0x20
[ 3103.549151] task_work_run+0x80/0x140
[ 3103.549153] do_exit+0x324/0xa08
[ 3103.549155] do_group_exit+0x44/0xa0
[ 3103.549158] __wake_up_parent+0x0/0x30
[ 3103.549161] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.549164] do_el0_svc+0x24/0x90
[ 3103.549166] el0_svc+0x14/0x20
[ 3103.549168] el0_sync_handler+0xb0/0xb8
[ 3103.549171] el0_sync+0x178/0x180
[ 3103.549172] ---[ end trace d15991a0fbaef718 ]---
[ 3103.549679] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.549693] ------------[ cut here ]------------
[ 3103.549699] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.549701] Modules linked in: libdes mali_dp
[ 3103.549708] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.549710] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.549712] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.549715] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.549717] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.549718] sp : ffff800012ef3c90
[ 3103.549720] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.549725] x27: 0000000000000000 x26: 0000000000000000
[ 3103.549729] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.549733] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.549738] x21: ffff0e7d1dd9dc00 x20: ffff0e7d3d39f040
[ 3103.549744] x19: ffff0e7d1d6adc80 x18: ffffb28536ab3470
[ 3103.549749] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.549753] x15: ffff0e7d3d39f040 x14: 00000000000008cd
[ 3103.549757] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.549762] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.549766] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.549771] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.549775] x5 : ffff0e83fc27d860 x4 : 0000000000000000
[ 3103.549782] x3 : 0000000000000027 x2 : 0000000000000000
[ 3103.549786] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.549790] Call trace:
[ 3103.549792] vfio_fsl_mc_release+0xdc/0x190
[ 3103.549795] vfio_device_fops_release+0x24/0x48
[ 3103.549798] __fput+0x78/0x230
[ 3103.549801] ____fput+0x10/0x20
[ 3103.549804] task_work_run+0x80/0x140
[ 3103.549806] do_exit+0x324/0xa08
[ 3103.549809] do_group_exit+0x44/0xa0
[ 3103.549811] __wake_up_parent+0x0/0x30
[ 3103.549814] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.549818] do_el0_svc+0x24/0x90
[ 3103.549822] el0_svc+0x14/0x20
[ 3103.549824] el0_sync_handler+0xb0/0xb8
[ 3103.549827] el0_sync+0x178/0x180
[ 3103.549829] ---[ end trace d15991a0fbaef719 ]---
[ 3103.550346] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)
[ 3103.550360] ------------[ cut here ]------------
[ 3103.550368] WARNING: CPU: 2 PID: 3644 at drivers/vfio/fsl-mc/vfio_fsl_mc.c:212 vfio_fsl_mc_release+0xdc/0x190
[ 3103.550373] Modules linked in: libdes mali_dp
[ 3103.550393] CPU: 2 PID: 3644 Comm: gnb_du_layer2 Tainted: G W 5.10.35 #5
[ 3103.550398] Hardware name: NXP Layerscape LX2160ARDB (DT)
[ 3103.550404] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 3103.550410] pc : vfio_fsl_mc_release+0xdc/0x190
[ 3103.550416] lr : vfio_fsl_mc_release+0xdc/0x190
[ 3103.550420] sp : ffff800012ef3c90
[ 3103.550426] x29: ffff800012ef3c90 x28: ffff0e7d3d39f040
[ 3103.550432] x27: 0000000000000000 x26: 0000000000000000
[ 3103.550437] x25: 0000000000000000 x24: ffff0e7d3d39f440
[ 3103.550441] x23: ffff0e7d030cc800 x22: 00000000fffffff3
[ 3103.550456] x21: ffff0e7d1ee17800 x20: ffff0e7d3d39f040
[ 3103.550469] x19: ffff0e7d1dc81580 x18: ffffb28536ab3470
[ 3103.550483] x17: ffff0e7d00001088 x16: ffff0e7d000010a8
[ 3103.550496] x15: ffff0e7d3d39f040 x14: 00000000000008f5
[ 3103.550510] x13: ffff0e7d3d39f4a0 x12: 00000000ffffffea
[ 3103.550523] x11: ffffb28536b23468 x10: ffffb28536b0b428
[ 3103.550536] x9 : ffffb28536b0b480 x8 : 0000000000017fe8
[ 3103.550549] x7 : c0000000ffffefff x6 : 0000000000000001
[ 3103.550564] x5 : 0000000000000000 x4 : ffff0e83fc27d860
[ 3103.550577] x3 : ffff0e83fc284770 x2 : 0000000000000000
[ 3103.550587] x1 : 0000000000000000 x0 : 0000000000000000
[ 3103.550591] Call trace:
[ 3103.550594] vfio_fsl_mc_release+0xdc/0x190
[ 3103.550597] vfio_device_fops_release+0x24/0x48
[ 3103.550600] __fput+0x78/0x230
[ 3103.550603] ____fput+0x10/0x20
[ 3103.550605] task_work_run+0x80/0x140
[ 3103.550608] do_exit+0x324/0xa08
[ 3103.550610] do_group_exit+0x44/0xa0
[ 3103.550612] __wake_up_parent+0x0/0x30
[ 3103.550616] el0_svc_common.constprop.0+0x78/0x1a0
[ 3103.550618] do_el0_svc+0x24/0x90
[ 3103.550621] el0_svc+0x14/0x20
[ 3103.550623] el0_sync_handler+0xb0/0xb8
[ 3103.550626] el0_sync+0x178/0x180
[ 3103.550628] ---[ end trace d15991a0fbaef71a ]---
[ 3103.551136] vfio-fsl-mc dprc.2: VFIO_FLS_MC: reset device has failed (-13)

[Bio_TICFSL]

Hello,

The core problem is a missing restool dprc assign step in your setup script. Your script:

1. Creates dpdmux.0 and dpdmux.1 under dprc.1 (the root container).
2. Connects dpni.1/dpni.2 (which reside inside dprc.2, the VFIO container) to those DPDMUX interfaces.
3. Binds dprc.2 to the VFIO driver.

When the application exits, the VFIO driver attempts to reset all objects inside dprc.2. However, dpni.1/dpni.2 are still externally connected to dpdmux.0/dpdmux.1 objects that live in dprc.1. The MC firmware cannot cleanly reset the container because of these cross-container connections it doesn't have authority over → returns -13 (permission denied).

Why it works after reboot: After a cold boot, the DPAA2 objects are in a clean initial state. The first run succeeds. But on exit, the DPDMUX connections are not properly torn down, leaving the MC in a partial state. On the next run (without reboot), the container reset fails again.

Fix: Add restool dprc assign for DPDMUX Objects

The DPDMUX objects must be assigned into dprc.2 before binding to VFIO. This tells the MC that these objects are owned by the VFIO container and can be reset when the container is released.

Insert the following lines after creating/connecting the DPDMUX objects and before rebinding dprc.2 to VFIO:

# Assign dpdmux objects into dprc.2 so VFIO can reset them cleanly
restool dprc assign dprc.1 --object=dpdmux.0 --child=dprc.2 --plugged=1
restool dprc assign dprc.1 --object=dpdmux.1 --child=dprc.2 --plugged=1

# Now bind dprc.2 back to VFIO
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
export DPRC=dprc.2

Your corrected script section should look like:

# Unbind dprc.2 from VFIO
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/unbind

# Create and connect DPDMUX for port dpmac.3
restool dpdmux create --num-ifs=3 --method DPDMUX_METHOD_MAC --max-dmat-entries=3 --manip=DPDMUX_MANIP_NONE
restool dprc connect dprc.1 --endpoint1=dpmac.3 --endpoint2=dpdmux.0.0
restool dprc connect dprc.1 --endpoint1=dpni.1 --endpoint2=dpdmux.0.1
restool dprc connect dprc.1 --endpoint1=dpni.3 --endpoint2=dpdmux.0.2

# Create and connect DPDMUX for port dpmac.4
restool dpdmux create --num-ifs=3 --method DPDMUX_METHOD_MAC --max-dmat-entries=3 --manip=DPDMUX_MANIP_NONE
restool dprc connect dprc.1 --endpoint1=dpmac.4 --endpoint2=dpdmux.1.0
restool dprc connect dprc.1 --endpoint1=dpni.2 --endpoint2=dpdmux.1.1
restool dprc connect dprc.1 --endpoint1=dpni.4 --endpoint2=dpdmux.1.2

# *** KEY FIX: Assign dpdmux objects to dprc.2 so VFIO can reset them ***
restool dprc assign dprc.1 --object=dpdmux.0 --child=dprc.2 --plugged=1
restool dprc assign dprc.1 --object=dpdmux.1 --child=dprc.2 --plugged=1

# Bind dprc.2 back to VFIO
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
export DPRC=dprc.2

Additional Recommendations

1. Update MC firmware to the latest version for your BSP 

2. Proper teardown between runs (without reboot): Before re-running your setup script, destroy the previously created DPDMUX objects to ensure a clean state:

   restool dpdmux destroy dpdmux.0
   restool dpdmux destroy dpdmux.1
   

3. Check MC console log at /dev/dpaa2_mc_console for additional firmware-level error details that may accompany the -13 error.

Regards

[Chidananda22]

Thanks @Bio_TICFSL  I will execute and confirm