Imx8qm qemu ethernet passthrough almost working

[jsah]

I have managed to get ethernet fec device passthrough almost working on imx8qm qemu. The device can be seen in my guest vm. The ethernet link and connection speed are recognised correctly, the ethernet indicator leds work as expected and can ethernet properties etc be adjusted from guest shell.

In short what was done is the lsio_mu2 was activated in host devicetree and passed through to guest along with fec1, enet0_lpcg. To get this working I needed to add IMX_SC_R_MU_2A and IMX_SC_R_ENET_0 to domu init_on_rsrcs and rsrcs similar to what was done in: https://community.nxp.com/t5/i-MX-Processors/imx8qm-enable-lsio-mu2/td-p/1469782

The lsio_mu2 and scu seem to work fine on guest but the problem with fec1 is that it does not get any interrupts through to guest. The /proc/interrupts show the passed through interrupt ids correctly in both host and guest but no interrupt messages come to either guest or host. Only the lsio_mu2 seems to get its interrupts on guest so the interrupt passthrough seems to work in general. Also intentionally doing some misconfiguration on guest scu/lsio_mu2 will prevent guest from finding scu so I'm assuming scu and mailbox are setup correctly on guest.

I've tried to disable fec devices from uboot and host by deleting some device tree properties and setting compatible to "fsl,dummy" so that the device status is "okay" but the device would not actually be in use in host. This seems to work for host but fec devices seems to be active during uboot no matter what i do.

It may very well be that the lack of fec interrupts is a symptom of something that is missing or configured incorrectly but I'm running out of ideas what to do or check next. I would appreciate any hints on this.

[jsah]

Adding device tree dts files for imx8qm host and guest for my ethernet passthrough setup.

In the current versions only fec1 network device is passed through to guest and fec2 is disabled.

Note that this setup does not currently work completely there is some problem with getting fec1 interrupts through to guest.

Vfio can be setup with sourcing following script

#!/bin/bash

export PLATFORM_DEVICE_NAME=5b040000.ethernet
export PLATFORM_CLOCKC_NAME=5b230000.clock-controller
export PLATFORM_MAILBOX_NAME1=5d1d0000.mailbox

export PLATFORM_PATH=/sys/bus/platform/devices
export PLATFORM_DEVICE_PATH=${PLATFORM_PATH}/${PLATFORM_DEVICE_NAME}
export PLATFORM_CLOCKC_PATH=${PLATFORM_PATH}/${PLATFORM_CLOCKC_NAME}
export PLATFORM_MAILBOX_PATH1=${PLATFORM_PATH}/${PLATFORM_MAILBOX_NAME1}

echo vfio-platform > ${PLATFORM_MAILBOX_PATH1}/driver_override
echo ${PLATFORM_MAILBOX_NAME1} > /sys/bus/platform/drivers/vfio-platform/bind

echo vfio-platform > ${PLATFORM_CLOCKC_PATH}/driver_override
echo ${PLATFORM_CLOCKC_NAME} > /sys/bus/platform/drivers/vfio-platform/bind

echo vfio-platform > ${PLATFORM_DEVICE_PATH}/driver_override
echo ${PLATFORM_DEVICE_NAME} > /sys/bus/platform/drivers/vfio-platform/bind

To start qemu with passthrough. The script takes one argument which is the common basename of kernel "<name>.Image", guest "<name>.ext2" -filesystem and guest devicetree "<name>.dtb" -file.

#!/bin/bash

qemu-system-aarch64 \
-M virt,gic-version=host -enable-kvm \
-cpu host \
-m 512M \
-kernel "$1.Image" \
-drive file="$1.ext2",if=virtio,format=raw -dtb "$1.dtb" \
-nographic \
-append "loglevel=7 rootwait root=/dev/vda console=ttyAMA0 earlycon earlyprintk" \
-device vfio-platform,host=${PLATFORM_DEVICE_NAME} \
-device vfio-platform,host=${PLATFORM_CLOCKC_NAME} \
-device vfio-platform,host=${PLATFORM_MAILBOX_NAME1} \
-monitor unix:qemu-monitor-socket,server,nowait

[Bio_TICFSL]

good job!

[jsah]

Thanks, I think my dts are close to being correctly done but the problem still remains. The ethernet device interrupts are not coming to guest and while running the guest the interrupts are not coming to host either so my guess is that either something is blocking interrupts or they are not being fired at all.

I tested with XEN on the example xen image (L_VIRT_4.14.98_2.0.0_4.11_0.10_ga_images_MX8QMMEK) and to me it seemed that there was no ethernet passed through to guest. It would not supprise me if it has never been done even in xen (as it is not the most common usecase).

My current best guess is that ethernet is forcefully started during boot and hooked to lsio_mu1. This might prevent me from setting up ethernet on lsio_mu2 later on. Ethernet device does not seem to support device reset (which i had to disable so allow vfio) ("static bool reset_required = false" in vfio_platform.c)

So far i have been unable in trying to completely disable ethernet in uboot. I've tried disabling all fec and ethernet related devices in uboot config but ethernet still turns on (and even works) in uboot.