Hi @HenrikZ,
I've just cracked a similar issue, here are my findings if they help..
1) Guarantee your uboot is outputting debug to the correct peripheral. The latest Yocto 6.12.20 defaults to HDMI and has disabled the usual uart output. I ended up using the imx7s-warp as a known uart serial output. Whatever the case, verify uboot on something else.
2) Read you fuses in uboot:
=> fuse read 1 0 4 (imx7s - imx7d)
Reading bank 1:
Word 0x00000000: 20000200 10220000 00000000 10002820
The fuse banks are described in the RM for your specific chip.
3) If you have no OS or uboot in eMMC, your device will revert back to serial mode, check you have the correct images (fuse and IO dependant).

4) Re the above, check your eMMC etc.
uboot=> mmc info
uboot=> mmc parts
My issue was the imx couldn't find the uboot image in flash so it reverted back to serial download mode. With the new unprogrammed eMMC (or whatever you use) make the necessary partitions, make one partition bootable (flags 0x0c) for uboot, the others for the kernel, dtb and FS. For that I used a variety of Linux tools such as dd, disks, gparted, mkfs etc together with some uboot commands. Most of the Linux desktop commands I robbed from a old MFGTool xml script. Once the image was on board it worked.
4) If using an EVK the boot dip switches can override booting and will put you back into serial download mode - which is useful for reflashing. Make sure they are correct.
Hope some of this may help.