As per my understanding, your issue is Yocto build speed. If my understanding is right, you can fine-tune your Yocto build speed substantially with the following factors:
The above modifications should significantly improve your Yocto build speed.
Moreover, in addition to these, please check whether, CPU cores are getting a bottleneck for you, as you are attempting to execute many builds simultaneously. If that is the case, then try to restrict the CPU cores used by each build just by adding the below two lines to your <your_yocto_build_dir>/conf/local.conf:
BB_NUMBER_THREADS = "2"
PARALLEL_MAKE = "-j 2"
By the way, adjust the number 2 with the number that fits your system. Remember, lower number slows down the Yocto build. Higher number increases that specific Yocto build, but can starve others, i.e other Yocto builds also.
After all these changes, move to the Yocto base directory, and execute source setup-environment ./<your_yocto_build_dir>.
Thanks a lot,
Dibyajyoti