You can but building times will take much longer (approximately 2 times longer for the core-image-minimal) compared to a build done on a native machine. In case you can not do the build on a native machine, make sure your virtual has enough hard-disk room (at least 50GB). For example, these are the build folders sizes after baking core-image-minimal:
build$ du -h --max-depth=1
1.3G ./sstate-cache
3.2M ./cache
12K ./.hob
32K ./conf
22G ./tmp
23G .
The tmp folder is by far the largest (containing building statistics, source code, deployed images, etc.)
build/tmp$ tree -L 2 -d
.
├── buildstats
│ ├── cogl-imx6qsabresd
│ ├── fsl-image-gui-imx6qsabresd
│ ├── fsl-image-gui-sdk-imx6qsabresd
│ ├── mesa-dri-imx6qsabresd
│ ├── mesa-imx6qsabresd
│ └── pseudo-native-imx6qsabresd
├── cache
│ └── default-eglibc
├── deploy
│ ├── images
│ ├── licenses
│ └── rpm
├── log
│ ├── cleanlogs
│ └── cooker
├── pkgdata
│ ├── all-poky-linux
│ ├── all-poky-linux-gnueabi
│ ├── armv7a-vfp-neon-poky-linux-gnueabi
│ ├── imx6qsabresd-poky-linux
│ └── imx6qsabresd-poky-linux-gnueabi
├── sstate-control
├── stamps
│ ├── all-poky-linux
│ ├── all-poky-linux-gnueabi
│ ├── armv7a-vfp-neon-poky-linux-gnueabi
│ ├── imx6qsabresd-poky-linux
│ ├── imx6qsabresd-poky-linux-gnueabi
│ ├── work-shared
│ └── x86_64-linux
├── sysroots
│ ├── imx6qsabresd
│ ├── imx6qsabresd-tcbootstrap
│ └── x86_64-linux
├── work
│ ├── all-poky-linux
│ ├── all-poky-linux-gnueabi
│ ├── armv7a-vfp-neon-poky-linux-gnueabi
│ ├── imx6qsabresd-poky-linux
│ ├── imx6qsabresd-poky-linux-gnueabi
│ └── x86_64-linux
└── work-shared
└── gcc-4.7.2-r13