2410515_en-US

[#1]

i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enabled

Hi NXP Team,

We are developing a high-throughput, low-latency networking product using the i.MX95 19x19 LPDDR5 EVK with the official NXP Linux BSP LF 6.6.52_2.2.2.

We need to run DPDK with user-space PMDs using vfio-pci on NETC functions (for example, 0002:00:00.0 or SR-IOV VFs). However, binding the device to vfio-pci fails when the SMMU is enabled.

1. In our 64-byte frame benchmark:

   • Linux kernel driver (fsl_enetc4 ~170,000 pps
   • Native PMD on a VF with SMMU disabled: ~456,000 pps

   The performance difference makes a user-space DPDK driver necessary for our application.

2. We cannot disable the SMMU or use enable_unsafe_noiommu_mode=1, as our production system requires hardware-enforced DMA isolation.
3. With the SMMU enabled, unbinding the NETC function from fsl_enetc4 and binding it to vfio-pci fails with -EINVAL:

root@imx95-19x19-lpddr5-evk:~# lspci -k -s 0002:00:00.0
0002:00:00.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04)
        Subsystem: Philips Semiconductors Device e101
        Kernel driver in use: fsl_enetc4

root@imx95-19x19-lpddr5-evk:~# echo 0002:00:00.0 > /sys/bus/pci/drivers/fsl_enetc4/unbind
root@imx95-19x19-lpddr5-evk:~# echo "1131 e101" > /sys/bus/pci/drivers/vfio-pci/new_id
[  143.562552] vfio-pci: probe of 0002:00:00.0 failed with error -22

1. From tracing the Linux VFIO and Arm SMMU code, the failure appears to be related to the SMMU coherency capability. vfio requires IOMMU_CAP_CACHE_COHERENCY, while the Arm SMMU driver enables ARM_SMMU_FEAT_COHERENCY based on IDR0.COHACC.
2. On the i.MX95 EVK, IDR0.COHACC reads 0, so the SMMU is reported as not supporting coherent translation-table walks. This results in device_iommu_capable(..., IOMMU_CAP_CACHE_COHERENCY) returning false and vfio-pci failing with -EINVAL.
3. We understand that adding dma-coherent; to the SMMU device-tree node can force the Linux SMMU driver to enable ARM_SMMU_FEAT_COHERENCY. However, this also causes io-pgtable-arm to configure pgtbl_cfg.coherent_walk = true, which disables the CPU cache maintenance normally performed for page-table updates.
4. Therefore, we do not want to enable dma-coherent; unless the i.MX95 hardware is confirmed to support coherent SMMU translation-table walks. If the interconnect does not snoop CPU caches for SMMU table walks, this could potentially result in stale page-table entries and memory corruption during dynamic IOMMU mappings.

Could NXP please clarify the following:

1. Does the i.MX95 silicon/interconnect support hardware-coherent SMMU translation-table walks, including the required cache snooping of CPU caches?
2. If coherent table walks are supported, Is there any required configuration in the interconnect, System Manager, ATF/firmware, or other platform initialization that must be enabled before the SMMU reports coherent walks?
3. If coherent table walks are supported, what is the NXP-recommended device-tree/firmware configuration for enabling them safely, and how can this be validated on the EVK?
4. If coherent table walks are not supported, what is NXP's recommended production-safe method for using DPDK with vfio-pci while keeping the SMMU/DMA isolation enabled?

Any guidance on the supported configuration for DPDK + vfio-pci with SMMU enabled on the i.MX95 would be greatly appreciated.

Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable

If the customer wants to use SMMU for other use cases, but acceptable to bypass for the DPDK (does not want to use iommu.passthrough=1 in bootargs or disable SMMU node in dts).
Then can try to bind the uio-pci-generic framework and disable VSI-PSI messaging (export ENETC4_VSI_MSG_DISABLE=1). But disabling VSI-PSI messages means user won’t be able to use some ENETC features like promisc, VLAN MAC filtering, link information.

Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable

I just got feedback from the DPDK team ask the following.

This is the known limitation. The i.MX95-evk is non-IO cache coherent platform and VFIO_PCI  with SMMU enabled driver is dependent on “IOMMU_CAP_CACHE_COHERENCY” which we cannot advertise.
All the details are already logged in README: dpdk/nxp/README_imx95_enetc_vf_vfio at 25.11-qoriq · NXP/dpdk · GitHub

Even you can check in the dpdk data-path, the driver is either using non-cacheable memory or using SW cache maintenance instructions: dpdk/drivers/net/enetc/enetc_rxtx.c at 25.11-qoriq · NXP/dpdk · GitHub

Re: i.MX95 19x19 EVK: SMMU coherent table walks (IDR0.COHACC) and vfio-pci for DPDK with SMMU enable

Discussing with DPDK team.