"Normal world" Linux for iMX6Q Lite

[vsiles]

I got an iMX6Q Lite from Boundary Devices, and I am using their Ubuntu based released, which is working fine. Their Linux seems to be based on NXP/Freescale's Linux tree, which completely run in secure world.

I'm trying to port the Linux into the normal world, but I can't seem to make SMP work. If I disable SMP, I'm fine.

If I enable SMP but only core0 to be started, I'm fine. But as soon as I try to start a second core, I randomly crash during init. My best run was having a prompt, but can't login due to an infinite error:

nitrogen login: root
login: PAM Fa�lure, aborting: Criinit: ttymxc1 main process (916) terminated with status 99
init: ttymxc1 main process ended, respawning

Are there know Linux BSP for iMX6 boards that are running in the normal world ?

Best,

Vincent

PS: here is a summary of my changes into Linux to run it in normal mode:

- remove access to the diagnostic register, they are done prior to Linux by the secure world

- remove access to the L2 cache PL310, they are replaced by SMC commands to the monitor which performs them

- remove access to the SCU/SCR registers to enable the second core. Secondary cores are prepared by the secure world to be woken up using an IRQ, and Linux code is replaced by a call to  arch_send_wakeup_ipi_mask(cpumask_of(cpu));

[vsiles]

Hi !

I'm still working on it but that's not my top priority. I had a big issue activating L2 cache & SCU but that's done, and now I have random "freeze" at boot (with a jtag, I can see that the core0 is not stopped, the kernel is printing lots of stuff on the uart, but nothing can be seen on my console). My guess is that Linux from the Normal Side uses the SRC component to wake up the other cores, still in TZ mode, and kaboom.

I have some code to remove this behavior, but I didn't have time to test it yet.

The minimal things you need to do from the TZ side for each core is to correctly initialize the GIC, the SCTLR & ACTLR registers (some bits are write secure only in the latter) and some other secure only registers like errata & stuff.

I'll let you know when I can resume this work.

[dariussuciu]

Hey Vincent,

I changed to an i.MX6Q Nitorgen6 Max board and applied your linux patches(Thank you for sharing them!!) to a 4.1 kernel.

Now it almost starts fine (without SMP), but it gets stuck on one of the drivers (after "backlight_lcd supply power not found, using dummy regulator", waiting for an interrupt I think).

Could you share your secure world configuration?

(I set the CSL=0x00ff00ff and CSU_SA= 0x55555554). Is there something else (GIC? ) you are setting ?

[vsiles]

Hi!

We are slowly going on with SMP, I now see a couple of things that were missing / badly done (like a real monitor on each core, or cache invalidation at startup).

For the CSU, we set all CSL to "secure / non secure RW" just like you did, and we set all the Masters to "Normal world only" in the CSU_SA (without any lock).

If you don't have a monitor/secure world running along side your linux, you need to correctly configure the gic at boot time so it can be accessed by the normal world. iirc the minimal things to do are:

- set the group of SGI / PPI (first 32 irqs) to 1 (non secure)

- set the priority register in the GICC

- enable group 0 / 1 through IRQ in the GICC control register

[dariussuciu]

Thank you very much Vincent!

I was missing some gic config u-boot has when set to run with "CONFIG_ARMV7_NONSEC" and now linux4.1 boots in the normal world.

Good luck with making the SMP work and thanks again!

[ailtonlopes]

Hey vsiles‌ & dariussuciu‌ do you guys have the monitor implementation because im getting some problems implementing it.

Kind Regards, 

Ailton Lopes

[vsiles]

Sorry, but that's code owned by my company, I can't help you on the monitor part

[ailtonlopes]

vsiles can you at least tell me (not code) what do you initialize in your bootloader (cache, peripheral, etc)? because ive seen that u-boot does some of those but i couldn't make u-boot work because of compilation errors, i really need to get this working for this school project and i'm getting to the deadline. 

thanks in advance. 
Kind regards

[vsiles]

We are still using u-boot as a boot loader. We use the "SPL/u-boot" split, so that SPL is running in secure mode, and loading u-boot + our custom OS. The execution goes to our custom OS, which in turns will schedule "u-boot" in the non secure world, at some point.

The only modifications we have in the "non secure" uboot is to remove some of the "secure only" operation (like accessing some secure devices, or part of the GIC) so it doesn't fail to execute. Such operations are either performed already by the SPL, or added to our boot sequence in the secure OS.

If u-boot is not compiling, you should first solve this problem: we have a sabrelite board from boundary devices, and their uboot compiles just fine.

[vsiles]

Dear Gary, thank you for the links.

We already have moved the L2 cache code to the monitor and patched linux to perform the configuration using smc, but we might have missed a spot. I'll check on Monday.

But I'm not sure this is the same issue: one time I successfully booted in SMP, but the SD card was read-only, so Linux whole FS was read-only. When the SD card is not read-only, booting failed.

Any thoughts on this behavior could help :smileywink:

Best,

V.

[vsiles]

Dear Gary,

I have a https://boundarydevices.com/product/sabre-lite-imx6-sbc/  which is an i.MX6Q based board. We are Secure World already contains our own micro-kernel, and we want to have Linux in the Normal world to provide a "nice" interface to the whole system while having the sensitive software/data in the secure world. The issue is that BD's Linux is directly accessing some secure resources (like SRC module to control the boot of secondary cores, or direct accesses to the SCU/L2 caches). At the moment, we provide proxy (through SMC calls) so that Linux can configure the L2 cache correctly, but I did not yet succeed to fix the SMP support.

I'll have a look at the mainline Linux to see if they directly access SRC or not. Thank you for the information.

Best,

V.