In my opinion, you don´t need to care about "re-building" gcc each time you have a new clean build.
The option given to you to use an external toolchain with yocto, gives you the power to actually choose which *other* toolchain you want, for example, the linaro one.
So, if you really want a *different* toolchain, I think you should work to use an external toolchain.
But, in other hand, if you only complain that build time take too much, there are other things you may change on your configuration in order to try to reduce this time.
Yocto always create a folder with the "pre-built" binaries from each package, it´s called sstate-cache. You can share this folder with your other build folders, or even with your co-workers. (https://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#var-SSTATE_DIR)
In my personal opinion, you should not use the meta-toolchain as "external" toolchain, you should use yocto toolchain "as-is", if you´re using an stable branch it will only be changed in case of a bugfix, and I think you want bugfixes.
Please, apologize my delay, I´ve been busy with meta-fsl-arm release these past days.... And please, let me know your thoughts.