I got an iMX6Q Lite from Boundary Devices, and I am using their Ubuntu based released, which is working fine. Their Linux seems to be based on NXP/Freescale's Linux tree, which completely run in secure world.
I'm trying to port the Linux into the normal world, but I can't seem to make SMP work. If I disable SMP, I'm fine.
If I enable SMP but only core0 to be started, I'm fine. But as soon as I try to start a second core, I randomly crash during init. My best run was having a prompt, but can't login due to an infinite error:
nitrogen login: root
login: PAM Fa�lure, aborting: Criinit: ttymxc1 main process (916) terminated with status 99
init: ttymxc1 main process ended, respawning
Are there know Linux BSP for iMX6 boards that are running in the normal world ?
PS: here is a summary of my changes into Linux to run it in normal mode:
- remove access to the diagnostic register, they are done prior to Linux by the secure world
- remove access to the L2 cache PL310, they are replaced by SMC commands to the monitor which performs them
- remove access to the SCU/SCR registers to enable the second core. Secondary cores are prepared by the secure world to be woken up using an IRQ, and Linux code is replaced by a call to arch_send_wakeup_ipi_mask(cpumask_of(cpu));
Solved! Go to Solution.
Ok, it is finally done.
The main thing I was missing was properly install a secure monitor on each core, because my patch about Linux accessing the SNVS RTC can be executed by any core, so there were smc performed on each core. Also, my cache maintenance routines for secondary cores was a bit buggy. This has to be fail proof.
- remove all secure access from Linux. In my case, it was access to the SCU, SRC, PL310 & SNVS. We decided to implem them using call to the secure monitor at runtime.
- start all core and "prepare" them to be runnable by Linux: correctly install a secure monitor, set the ACTLR/NSACR/GIC secure bits correctly and make it way (e.g. using WFI) for Linux to provide a jump address, and jump to this address using an ERET from Monitor Mode, so you can switch from secure code (wait loop in the monitor / secure kernel) to normal code for Linux.
- patch the way Linux starts the cores (if you prevent access to the SRC via CSU, you can implement them using an SMC call. For the time being, Linux still writes the address in the SRC but "wake up" the core using a SGI)
I'm still working on it but that's not my top priority. I had a big issue activating L2 cache & SCU but that's done, and now I have random "freeze" at boot (with a jtag, I can see that the core0 is not stopped, the kernel is printing lots of stuff on the uart, but nothing can be seen on my console). My guess is that Linux from the Normal Side uses the SRC component to wake up the other cores, still in TZ mode, and kaboom.
I have some code to remove this behavior, but I didn't have time to test it yet.
The minimal things you need to do from the TZ side for each core is to correctly initialize the GIC, the SCTLR & ACTLR registers (some bits are write secure only in the latter) and some other secure only registers like errata & stuff.
I'll let you know when I can resume this work.
I changed to an i.MX6Q Nitorgen6 Max board and applied your linux patches(Thank you for sharing them!!) to a 4.1 kernel.
Now it almost starts fine (without SMP), but it gets stuck on one of the drivers (after "backlight_lcd supply power not found, using dummy regulator", waiting for an interrupt I think).
Could you share your secure world configuration?
(I set the CSL=0x00ff00ff and CSU_SA= 0x55555554). Is there something else (GIC? ) you are setting ?
We are slowly going on with SMP, I now see a couple of things that were missing / badly done (like a real monitor on each core, or cache invalidation at startup).
For the CSU, we set all CSL to "secure / non secure RW" just like you did, and we set all the Masters to "Normal world only" in the CSU_SA (without any lock).
If you don't have a monitor/secure world running along side your linux, you need to correctly configure the gic at boot time so it can be accessed by the normal world. iirc the minimal things to do are:
- set the group of SGI / PPI (first 32 irqs) to 1 (non secure)
- set the priority register in the GICC
- enable group 0 / 1 through IRQ in the GICC control register
Thank you very much Vincent!
I was missing some gic config u-boot has when set to run with "CONFIG_ARMV7_NONSEC" and now linux4.1 boots in the normal world.
Good luck with making the SMP work and thanks again!
vsiles can you at least tell me (not code) what do you initialize in your bootloader (cache, peripheral, etc)? because ive seen that u-boot does some of those but i couldn't make u-boot work because of compilation errors, i really need to get this working for this school project and i'm getting to the deadline.
thanks in advance.
We are still using u-boot as a boot loader. We use the "SPL/u-boot" split, so that SPL is running in secure mode, and loading u-boot + our custom OS. The execution goes to our custom OS, which in turns will schedule "u-boot" in the non secure world, at some point.
The only modifications we have in the "non secure" uboot is to remove some of the "secure only" operation (like accessing some secure devices, or part of the GIC) so it doesn't fail to execute. Such operations are either performed already by the SPL, or added to our boot sequence in the secure OS.
If u-boot is not compiling, you should first solve this problem: we have a sabrelite board from boundary devices, and their uboot compiles just fine.
Dear Gary, thank you for the links.
We already have moved the L2 cache code to the monitor and patched linux to perform the configuration using smc, but we might have missed a spot. I'll check on Monday.
But I'm not sure this is the same issue: one time I successfully booted in SMP, but the SD card was read-only, so Linux whole FS was read-only. When the SD card is not read-only, booting failed.
Any thoughts on this behavior could help :smileywink:
I have a https://boundarydevices.com/product/sabre-lite-imx6-sbc/ which is an i.MX6Q based board. We are Secure World already contains our own micro-kernel, and we want to have Linux in the Normal world to provide a "nice" interface to the whole system while having the sensitive software/data in the secure world. The issue is that BD's Linux is directly accessing some secure resources (like SRC module to control the boot of secondary cores, or direct accesses to the SCU/L2 caches). At the moment, we provide proxy (through SMC calls) so that Linux can configure the L2 cache correctly, but I did not yet succeed to fix the SMP support.
I'll have a look at the mainline Linux to see if they directly access SRC or not. Thank you for the information.