Hello Chintan Raval,
As Marco Madrigal mentioned the easiest way to achieve this is creating your image with only the required packages. Core-image-minimal is a great place to start! As this inherits from the core-image.bbclass and allows to add some standard IMAGE_FEATURES.
You can find more information on the Yocto Reference Manual:
https://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-classes-core-image
Alternatively, you can disable the GPU support on the kernel defconfig.
I hope this helps!
Regards,