i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enabled Hi NXP Team, We are developing a high-throughput, low-latency networking product using the i.MX95 19x19 LPDDR5 EVK with the official NXP Linux BSP LF 6.6.52_2.2.2. We need to run DPDK with user-space PMDs using vfio-pci on NETC functions (for example, 0002:00:00.0 or SR-IOV VFs). However, binding the device to vfio-pci fails when the SMMU is enabled. In our 64-byte frame benchmark: Linux kernel driver (fsl_enetc4😞 ~170,000 pps Native PMD on a VF with SMMU disabled: ~456,000 pps The performance difference makes a user-space DPDK driver necessary for our application. We cannot disable the SMMU or use enable_unsafe_noiommu_mode=1, as our production system requires hardware-enforced DMA isolation. With the SMMU enabled, unbinding the NETC function from fsl_enetc4 and binding it to vfio-pci fails with -EINVAL: root@imx95-19x19-lpddr5-evk:~# lspci -k -s 0002:00:00.0
0002:00:00.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04)
Subsystem: Philips Semiconductors Device e101
Kernel driver in use: fsl_enetc4
root@imx95-19x19-lpddr5-evk:~# echo 0002:00:00.0 > /sys/bus/pci/drivers/fsl_enetc4/unbind
root@imx95-19x19-lpddr5-evk:~# echo "1131 e101" > /sys/bus/pci/drivers/vfio-pci/new_id
[ 143.562552] vfio-pci: probe of 0002:00:00.0 failed with error -22 From tracing the Linux VFIO and Arm SMMU code, the failure appears to be related to the SMMU coherency capability. vfio requires IOMMU_CAP_CACHE_COHERENCY, while the Arm SMMU driver enables ARM_SMMU_FEAT_COHERENCY based on IDR0.COHACC. On the i.MX95 EVK, IDR0.COHACC reads 0, so the SMMU is reported as not supporting coherent translation-table walks. This results in device_iommu_capable(..., IOMMU_CAP_CACHE_COHERENCY) returning false and vfio-pci failing with -EINVAL. We understand that adding dma-coherent; to the SMMU device-tree node can force the Linux SMMU driver to enable ARM_SMMU_FEAT_COHERENCY. However, this also causes io-pgtable-arm to configure pgtbl_cfg.coherent_walk = true, which disables the CPU cache maintenance normally performed for page-table updates. Therefore, we do not want to enable dma-coherent; unless the i.MX95 hardware is confirmed to support coherent SMMU translation-table walks. If the interconnect does not snoop CPU caches for SMMU table walks, this could potentially result in stale page-table entries and memory corruption during dynamic IOMMU mappings. Could NXP please clarify the following: Does the i.MX95 silicon/interconnect support hardware-coherent SMMU translation-table walks, including the required cache snooping of CPU caches? If coherent table walks are supported, Is there any required configuration in the interconnect, System Manager, ATF/firmware, or other platform initialization that must be enabled before the SMMU reports coherent walks? If coherent table walks are supported, what is the NXP-recommended device-tree/firmware configuration for enabling them safely, and how can this be validated on the EVK? If coherent table walks are not supported, what is NXP's recommended production-safe method for using DPDK with vfio-pci while keeping the SMMU/DMA isolation enabled? Any guidance on the supported configuration for DPDK + vfio-pci with SMMU enabled on the i.MX95 would be greatly appreciated. Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable If the customer wants to use SMMU for other use cases, but acceptable to bypass for the DPDK (does not want to use iommu.passthrough=1 in bootargs or disable SMMU node in dts). Then can try to bind the uio-pci-generic framework and disable VSI-PSI messaging (export ENETC4_VSI_MSG_DISABLE=1). But disabling VSI-PSI messages means user won’t be able to use some ENETC features like promisc, VLAN MAC filtering, link information. Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable I just got feedback from the DPDK team ask the following.
This is the known limitation. The i.MX95-evk is non-IO cache coherent platform and VFIO_PCI with SMMU enabled driver is dependent on “IOMMU_CAP_CACHE_COHERENCY” which we cannot advertise. All the details are already logged in README: dpdk/nxp/README_imx95_enetc_vf_vfio at 25.11-qoriq · NXP/dpdk · GitHub
Even you can check in the dpdk data-path, the driver is either using non-cacheable memory or using SW cache maintenance instructions: dpdk/drivers/net/enetc/enetc_rxtx.c at 25.11-qoriq · NXP/dpdk · GitHub Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable Discussing with DPDK team. Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable Hi yipingwang, Thanks for the prompt reply and insight.We will get back to you if further doubts arise.
View full article