After several hours of struggle and confusing manual, here is the problem on LPC824:
According to LPC82x User Manual, pag. 56, the PLL block have some high frequency oscillator (called CCO) that can work (can be locked) in FCCO = 156...320 MHz range.
After CCO, exists a post divider that can be set to 4 divider values (1,2,4 and 8) automatically further divided by 2 for 50 % duty-cycle, resulting in real P = 2,4,8 or 16. This is for obtaining FCLKOUT (MAIN CLOCK) that is limited to 100MHz.
To close the PLL loop, exists also a M divider (called feedback in user manual) that can be set such as real divider values = 1...32
Now when injecting 12 MHz from IRC into PLL and set M = 5, MAIN CLOCK will be 60 MHz with no effect of changing P value. That is, any value set to P the clock will be 60 MHz (IRC * M). Since FCCO can be locked only in 156...320 MHz range, we deduce that lock is produced at 240 MHz and real P is 4 (P = 2, 2*P = 4).
We enabled CLOCKOUT to monitor MAIN SYSTEM CLOCK and this indeed outputs 60 MHz, 72 MHz or whatever M is set to multiply the 12 MHz IRC.
Any value programmed in P bits (pag. 35, SYSPLLCTRL register) have no effect.
Taking example from table 57 (pag. 58) if keep M=5 and set P = 4 (real P = 8) the CCO should (still) lock on 240 MHz but MAIN CLOCK should be 30 MHz (well under 100 MHz limit). Instead, example keep the MAIN CLOCK = 60 MHz and modify the SYSTEM CLOCK from SYSAHBCLKDIV register.
Appears that P divider is just not working, and set at fixed value P = 2 (real P for PLL formula = 4) regarding PSEL bits setting in SYSPLLCTRL register.
Can someone confirm this?