Sounds like a wrong FCDIV setting. Even if you calculate divider properly, FCDIV is write once register. See in the docs:
- "Bit 7 of this register is a read-only flag. Bits 6:0 may be read at any time but can be written only one time."
This means that FCDIV has to be written all bits simultaneously. 2nd write is ignored. For example
FCDIV.PRDIV8 = 1;
FCDIV.DIV = 7; // ignored
will lock FCDIV.PRDIV8 == 1, FCDIV.DIV == 0 (power on default). And
FCDIV.DIV = 5;
FCDIV.PRDIV8 = 1; // ignored
will lock DIV==5, PRDIV8 = 0