I am building for an i.mx53 QSB. I am not using buildroot to build the kernel or u-boot. Though I probably could.
I am using it to build the toolchain and the rootfs - though I typically do a fair amount of post build customization of buildroot rootfs's as I do not like their initialization scheme and revert to a more traditional one with start/stop scripts in /etc/init.d and links in /etc/rcS.d for those that are enabled.
All the typical buildroot (and pretty much every build environment) caveats apply.
Don't run buildroot as root, don't build it from /usr/src, frequently some choices of libraries gcc versions, cpu's .... are incompatible. As an example I beleive some versions of gcc 4.6 will not build some libraries.
Overall i like buildroot better than LTIB, but nothing is perfect.