PCI device assignment on iMX8 QM board

mtosatti — Wed, 27 Sep 2023 21:12:30 GMT

We are doing some testing on iMX8 QM board.
We are trying to assign a network PCI card to a KVM guest.
Unfortunately, the device tree for the board does not specify an IOMMU for the PCI bus 1 (where this network card resides).
so we tried adding iommu and iommu-maps properties to the device tree:

diff --git a/arch/arm64/boot/dts/freescale/imx8qm-ss-hsio.dtsi b/arch/arm64/boot/dts/freescale/imx8qm-ss-hsio.dtsi
index 09fde1445c45..d1c1d8a89509 100644
--- a/arch/arm64/boot/dts/freescale/imx8qm-ss-hsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8qm-ss-hsio.dtsi
@@ -133,6 +133,23 @@ pciea: pcie@0x5f000000 {
hsio-cfg = <PCIEAX1PCIEBX1SATA>;
local-addr = <0x40000000>;
status = "disabled";
+ iommus = <&smmu 0x15 0x7f80>;
+ iommu-map = <0x0 &smmu 0x15 0x1000>;

which then puts all PCI devices in bus 1 under a single IOMMU group:

./3
./3/devices
./3/devices/0001:00:00.0
./3/devices/0001:02:02.0
./3/devices/0001:05:00.0
./3/devices/0001:02:01.0
./3/devices/0001:03:00.0
./3/devices/5f000000.pcie
./3/devices/0001:05:00.1
./3/devices/0001:02:03.0
./3/devices/0001:01:00.0
./3/type
./3/reserved_regions (edited)

However, this check on drivers/vfio/vfio_iommu_type1.c:

/* Determine bus_type in order to allocate a domain */
ret = iommu_group_for_each_dev(iommu_group, &bus, vfio_bus_type);
if (ret) {
goto out_free;
}

static int vfio_bus_type(struct device *dev, void *data)
{
struct bus_type **bus = data;

if (*bus && *bus != dev->bus)
return -EINVAL;

*bus = dev->bus;

return 0;
}

Fails since for ./3/devices/5f000000.pcie:

qemu-kvm-1218 [000] ..... 73.450622: vfio_bus_type: devname: 0001:00:00.0

And:

For 5f000000.pcie:

subsystem -> ../../../../bus/platform

whereas for the other devices in the IOMMU group

subsystem -> ../../../../../../../bus/pci

By skipping this check:

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 852e6c5643e5..4ce00dc42e7f 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -1699,8 +1699,12 @@ static int vfio_bus_type(struct device *dev, void *data)
{
struct bus_type **bus = data;

- if (*bus && *bus != dev->bus)
- return -EINVAL;
+ if (*bus && *bus != dev->bus) {
+ if (strncmp(dev_name(dev), "0001:00:00.0", 12) != 0) {
+ trace_printk("devname: %s\n", dev_name(dev));
+ return -EINVAL;
+ }
+ }

*bus = dev->bus;

We are able to assign the PCI device to the guest, but:

[ 84.621002] igb 0000:00:03.0 enp0s3: igb: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[ 88.071936] igb 0000:00:03.0: Detected Tx Unit Hang
[ 88.071936] Tx Queue <0>
[ 88.071936] TDH <1>
[ 88.071936] TDT <1>
[ 88.071936] next_to_use <1>
[ 88.071936] next_to_clean <0>
[ 88.071936] buffer_info[next_to_clean]
[ 88.071936] time_stamp <ffffac2a>
[ 88.071936] next_to_watch <00000000c2df116c>
[ 88.071936] jiffies <ffffad38>
[ 88.071936] desc.status <168000>
[ 90.070738] igb 0000:00:03.0: Detected Tx Unit Hang
[ 90.070738] Tx Queue <0>
[ 90.070738] TDH <1>
[ 90.070738] TDT <1>
[ 90.070738] next_to_use <1>
[ 90.070738] next_to_clean <0>
[ 90.070738] buffer_info[next_to_clean]
[ 90.070738] time_stamp <ffffac2a>
[ 90.070738] next_to_watch <00000000c2df116c>
[ 90.070738] jiffies <ffffae00>
[ 90.070738] desc.status <168000>

Any ideas on how to fix this?

We are using origin/toradex_5.15-2.1.x-imx branch of git://git.toradex.com/linux-toradex.git

i.MX ProcessorsのトピックPCI device assignment on iMX8 QM board

PCI device assignment on iMX8 QM board