Hi,
Once you have a build running on your custom board you can try the following.
First locate your correct device tree file(s), sometimes with Yocto and certain builds it may not be the files you think.. Then as a quick test comment out some obviously unused items and rebuild. Check you boot log to make sure the devices have disappeared or are no longer probed. You can trim down the device tree a fair bit if needed. Once you have worked out what can go, create some device tree patch file(s) and add these to your sources/meta-yours/recipes-kernel etc to make the changes permanent. This is fine if you want to work with modified EVK builds. You can remove unwanted drivers as well using bitbake linux-imx -c menuconfig thus having a similar effect as removing them from your device tree.
Just be aware if you tweak the actual device tree file(s) up in /arch/arm64/boot/dts/freescale somewhere then do a kernel clean and build you will loose your changes. Do your changes after a kernel clean if you need to do one, creating patch files is a better practice so perhaps worth looking into sooner rather than later.
Hope it helps, it's a big learning curve.