I am not a hardware designer, but picked up one or the other thing during the years ...
First, I guess you are talking about a custom board, i.e. your own design, don't you ?
Second, such instabilities can come from the clock path or the power supply path. Insufficient power supply or capacitive buffering can cause such problems, especially during transients (like clock switching).
And third, I would check the PLL setting code, so as to not exceed any parameter limits, even temporarily (multiplier, divider).
I would try lower core clock values, and see if they work more stable. At least as a start.