You're using that counter as a counting semaphore. This is something that all code using interrupts has to get right. Real basic stuff. The following is a "general introduction" to the problem:
Semaphore (programming) - Wikipedia, the free encyclopedia
The following section says that disabling interrupts is a standard practice (I'll get back to this):
Real-time operating system - Wikipedia, the free encyclopedia
The different threads (mainline and interrupt in this case) need exclusive access to that counter/semaphore.
That can either be provided by the hardware if it allows "atomic access" (in this case, atomic decrement). The ColdFire chip's SUB and SUBQ will both work with a memory location as the destination, so can support this (unlike chips such as the Power ones, which can only load and store memory).
As you've found, this will work as long as the compiler selects one of these instructions, but there's no guarantee of that. The compiler is free to keep that counter in a register for a while if the optimiser wants to. You may be able to wave a "volatile" at the compiler, but that doesn't really guarantee anything. "counter--" may turn into a load/subtract/store, just like it did when you declared it as 16 bits.
The only safe ways to write this code (so it works and keeps working) are:
1 - Surround the mainline decrement with interrupt disable and enables. That is the standard/normal approach.
2 - Perform the subtraction with inline assembly to force the atomic form of the instruction.
3 - Perform the subtraction in a subroutine written in assembly that uses the right instruction.
Another option is to read up on the "TAS" instruction, which is specifically meant for implementing semaphores, but that's overkill in this case, and would probably cause other problems. The 68000 (which the ColdFire derives from) had CAS and CAS2, which were better suited for this, but the ColdFire didn't get these instructions. Anyway, in your case the SUB does what you need.
> Disabling interrupts is not an option.
Why? Is your serial port code running in "user mode" instead of "kernel mode" or something? You're running on a MCF51QE, so it isn't like you're running Linux or anything. The "Disable/Decrement/Enable" sequence is extremely quick. Why can't you use this, or why don't you think you can use this?
Tom