Hello Martin,
Perhaps the RANGE bit, within ICSC2, should be set for high frequency operation, even though you are using an external oscillator module. The output from the module will need to be a square wave with full logic swing at 5 volts, particularly at the frequency you are using.
For a much lower oscillator frequency of, say 5 MHz, a lower swing may be tolerated by configuring as an external crystal, complete with bias resistor between the two oscillator pins at the MCU, and then AC coupling the oscillator module output into EXTAL.
When you fail to enter FBELP mode, does operation continue in the default FEI mode? Another factor might be that, if the BDM is active when you attempt to switch for FBELP mode, this may prevent the switching from occuring. I am not sure about this.
Using a suitable crystal, achieving 50ppm stability over 0-70 degree C may be achievable, but would be difficult with the Pierce oscillator configuration used, due to temperature variation of the load capacitance. Even if the external capacitors are NPO type, the pin capacitance will appear in parallel with each capacitor, and will be a significant component when a low capacitance crystal is used. You would also need to provide frequency trimming of the reference.
However, an oscillator module of 5MHz maximum frequency might be utilised. When using FEE mode, the bus frequency would be subject to short term jitter, a characteristic of the FLL circuit, but this will not affect the long term frequency accuracy. The jitter can be avoided within the timing modules, by choosing a clock source other than the bus frequency.
The RTC module may alternatively be driven by ICSERCLK, directly from your external oscillator. The TPM and MTIM modules may alternatively be driven by one half the ICSFFCLK frequency. For a 4MHz oscillator this would be 15.626 kHz, and for a 5MHz oscillator, 19.53125 kHz.
Regards,
Mac