So I've got all my hardware tested and everything works on my custom board, but I really need to pare things down to bare minima because I want my device to boot as fast as possible, draw the least amount of power necessary, and give me the ability to do what I want. I'm building u-boot, kernel, and device tree from the source code using the Make command in a linux host environment. I'm not sure at all what filesystem or rootfs (not sure what to call it), for example linaro debian (which takes several minutes to get to the desktop environment on my 5" LCD). Most of my research keeps pointing me to Yocto, but I really want to avoid that because it's just too complicated and takes way too long to do anything at all with. Building from source seems the fastest and most efficient way to go.
That said, what I want to end up with is a system that can display a UI on a 5" paralllel LCD, talk to a couple of i2c devices like the tsc2007 touchscreen controller and a battery fuelgauge, and run a UI on startup that I build in C# in xamarin and run with mono. How do I reach my goals?? All of the documentation I've been looking at keeps pointing me to Yocto, but surely there's a better/faster/easier way.