I can't talk for KDS behaviour, but if the heap allocator finds that there is insufficient space left in the heap area, then you should get a NULL back from malloc(): http://www.cplusplus.com/reference/cstdlib/malloc/
Your code should then take the appropriate action after checking for the NULL being returned.
For details of how to modify the amount of space reserved for the heap, please read the MCUXpresso IDE v10.1.0 User Guide, chapter 14, " Memory Configuration and Linker Scripts" - in particular section 14.9, "More advanced heap/stack placement".
Regards,
MCUXpresso IDE Support