My understanding is that if there are no characters in the serial buffer then the underlying _io_serial_int_read returns zero (no characters read); this causes _io_fgetc to return IO_EOF; which causes _io_fgetline to return IO_EOF; which causes _io_fgets to return NULL. Nothing blocks.
If there are some characters in the serial buffer, but not a line-end character, then _io_fgetline gets a few characters from _io_fgetc before getting IO_EOF when the buffer runs out. At this point _io_fgetline returns IO_EOF, having populated the caller's buffer with the characters retrieved so far. _io_fgets then returns in the normal way. Again, nothing blocks.
In other words, the serial receive buffer becoming empty has much the same effect as receiving a line-end.
Edit: typo.