When you stop the external clock (or even before), can you examine SYNSR and see where the MCU thinks the clock is coming from? Is EXTOSC always set, and OCOSC/CRYOSC clear? There is loss of clock circuitry (SYNCR->LOCEN), to automatically switch clocks or reset the MCU, but the data sheet seems to imply is disabled when running off an external clock. See Reference Manual section 6.8.4.8.