There are two format system image(ext4): raw and sparse. The raw image is larger, you can mount it to ext4 directly(mount -t ext4 system.img system). The sparse image is supported lp5.0, it is a bit smaller. The below part will take IMX7D SDB board as example. You can change the setting according your platform hardware.
1 generate sparse image
In Lollipop 5.0 and 5.1, both raw image and sparse image will be generated by default.
sparse system image is located in: out/target/product/sabresd_7d/system_sparse.img
raw system image is located in:out/target/product/sabresd_7d/system.img
Below are the steps used in android build system to generate system.img
build out sparse image(-s means the building system image is sparse)
make_ext4fs -s -T -1 -S out/target/product/sabresd_7d/root/file_contexts -l 374476800 -a system out/target/product/sabresd_7d/obj/PACKAGING/systemimage_intermediates/system.img out/target/product/sabresd_7d/system