Let's assume that state of cores 1-7 is unknown. They can run any code or they in holdoff.
How to reset cores 1-7 from core 0 to initial state so they start execution from bootpage after holdoff?
What is recommended reset sequence?
Since you don't know anything about the state of cores 1-7, it's possible that they could be actively interfering with core 0 or otherwise causing trouble. In general this isn't a situation that we support recovering from.
Also note that if core 1-7 have modified memory that core 0 is interested in, that memory may still be in the cache of core 1-7 and the modifications will be lost when you reset the cores.
But if you want to try, or can put some bounds on the state of cores 1-7:
1. Make sure no core is in a low power state (using the RCPM registers, and make sure no core re-enters low power state after core 0 pulls them out).
2. Reset cores 1-7 using MPIC_PIR and the sequence documented in section 22.3.11 "
Processor initialization register (MPIC_PIR)" of the P4080 reference manual.
3. Set up the boot page mapping to point to the code you want the cores to run.
4. Release the cores from holdoff using CONFIG_BRR.
Note that at this point the timebase will not be synchronized across cores. You can synchronize them by disabling the timebase for all cores using RCPM_TBENR, setting the timebase to the same value on all cores, and then enabling all the timebases at once using RCPM_TBENR.
In my case cores 1-7 in early boot stage and I am sure that they not interfering with core 0.
So my reset sequence is:
1. Force cores 1-7 to holdoff by setting zeros in CONFIG_BRR.
2. Update RCPM registers to prevent low power state.
3. Reset cores 1-7 using MPIC_PIR.
4. Set boot page translation.
5. Disable timebase for all cores using RCPM_TBENR.
5. Release cores from holdoff using CONFIG_BRR.
7. Wait for cores 1-7 early init flags(assume that early init code reset timebase on each core).
8. Reset timebase on core 0.
9. Enable timebase for all cores at once using RCPM_TBENR.
Is it correct?
I don't think it's legal to put cores back into boot holdoff once you've released them, short of a core reset. Could you elaborate on what higher-level problem you're trying to solve?
I am trying to boot FreeBSD on P4080. U-boot with mp_holdoff=0 leave cores 1-7 in spinlocks as described in ePAPR and provide "cpu-release-addr" in FDT to kick them. But FreeBSD loader does not fixup "cpu-release-addr" in their FDT and kernel don't know how to kick cores. In this case I want to forcibly reset cores unaware of u-boot mp_holdoff.
I suggest either modifying FreeBSD to play nice with ePAPR, or adding an option to U-Boot to leave the cores in holdoff from the beginning.
...and as you point out, such a U-Boot option already exists (mp_holdoff). Why not just set it?
I use mp_holdoff=1. With this option u-boot leave cores 1-7 in holdoff but sometimes in inconsistent state and I get for exampe this:
SMP: CPU 2 didn't wake up (trace code 0).
SMP: CPU 3 didn't wake up (trace code 0).
SMP: CPU 4 didn't wake up (trace code 0).
SMP: CPU 5 didn't wake up (trace code 0).
SMP: CPU 6 didn't wake up (trace code 0).
SMP: CPU 7 didn't wake up (trace code 0).
sSMMPP:: 8A PC PCUPsU f#o1u nlda;u n8c hCePdU
usable; 2 CPUs woken
WARNING: WITNESS option enabled, expect reduced performance.
uRgoeont0 .m1o:u n<tF rweaeistcianlge >f oart: uussbbuuss00
It seems like cores start execution of unpredictable code.
I must be sure that the cores will start execute code from my boot page after holdoff.
If you set mp_holdoff U-Boot shouldn't be messing with the secondary cores. Then its up to the OS to release them safely, by setting up an appropriate boot page mapping. I don't see how resetting the cores is going to improve things. They're alredy in the reset state.
There's probably a bug in what FreeBSD is doing to release the cores. Or can you see with a debugger that the secondary cores are not in reset?
Retrieving data ...