Yes, we have seen the same thing. You are not missing anything. You may want to use shared files for download and sstate, which can save you about 2G.
This comes from the Freescale Yocto User Guide
A Yocto build can take considerable build resources both in time and disk usage especially when building in multiple build directories. There are methods to optimize this use a shared sstate cache and
downloads directory. These can be set at any location in the local.conf
DL_DIR="/opt/freescale/yocto/imx/download"
SSTATE_DIR="/opt/freescale/yocto/imx/sstate-cache"
These directories should have appropriate permissions. Note the shared sstate helps when multiple build directories are set – each uses a shared cache minimizing build time. A shared download directory
minimizes fetch time. Without these settings, Yocto defaults to the build directory for the sstate cache and downloads.
Every package downloaded in the DL_DIR directory is marked with a <package name>.done. To avoid fetching, touch the .done file for the package name.