topic Re: "Normal world" Linux for iMX6Q Lite in i.MX Processors

"Normal world" Linux for iMX6Q Lite

vsiles — Fri, 13 May 2016 14:07:22 GMT

I got an iMX6Q Lite from Boundary Devices, and I am using their Ubuntu based released, which is working fine. Their Linux seems to be based on NXP/Freescale's Linux tree, which completely run in secure world.

I'm trying to port the Linux into the normal world, but I can't seem to make SMP work. If I disable SMP, I'm fine.

If I enable SMP but only core0 to be started, I'm fine. But as soon as I try to start a second core, I randomly crash during init. My best run was having a prompt, but can't login due to an infinite error:

nitrogen login: root

init: ttymxc1 main process ended, respawning

Are there know Linux BSP for iMX6 boards that are running in the normal world ?

Best,

Vincent

PS: here is a summary of my changes into Linux to run it in normal mode:

- remove access to the diagnostic register, they are done prior to Linux by the secure world

- remove access to the L2 cache PL310, they are replaced by SMC commands to the monitor which performs them

- remove access to the SCU/SCR registers to enable the second core. Secondary cores are prepared by the secure world to be woken up using an IRQ, and Linux code is replaced by a call to arch_send_wakeup_ipi_mask(cpumask_of(cpu));

Re: "Normal world" Linux for iMX6Q Lite

gary_bisson — Sat, 14 May 2016 16:47:51 GMT

Hi Vincent,

Can you please clarify which board you are using? We have a Nitrogen6_Lite which is a i.MX6 Solo board, not sure what is the i.MX6Q Lite.

Why do you need to run on non-secure world? I think it is supported by the mainline kernel, have you tried it? Our Nitrogen6x board is supported in mainline kernel but know that some features are missing like the use of the GPU.

Regards,

Gary

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Sun, 15 May 2016 16:36:25 GMT

Dear Gary,

I have a https://boundarydevices.com/product/sabre-lite-imx6-sbc/ which is an i.MX6Q based board. We are Secure World already contains our own micro-kernel, and we want to have Linux in the Normal world to provide a "nice" interface to the whole system while having the sensitive software/data in the secure world. The issue is that BD's Linux is directly accessing some secure resources (like SRC module to control the boot of secondary cores, or direct accesses to the SCU/L2 caches). At the moment, we provide proxy (through SMC calls) so that Linux can configure the L2 cache correctly, but I did not yet succeed to fix the SMP support.

I'll have a look at the mainline Linux to see if they directly access SRC or not. Thank you for the information.

Best,

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Tue, 17 May 2016 07:05:06 GMT

Dear Gary,

I just checked mainline Linux and the issue is the same: it is made to work in the normal (which is fine since Linux is made to work alone, with no monitor/secure world blob, it should run by itself). For example in arch/arm/mach-imx/platsmp.c, you can see that the imx boards are using the "SRC" module to completely reset the secondary cores (which is secure only in a trustzone case), whereas the ls1021a boards are just issuing an IPI to the secondary cores to wake them up, which is fine.

I'm looking for a Linux kernel where Linux only uses "normal" world resources. At the moment, I'm trying to fix Linux's SMP management to be able to boot all 4 cores of my board, but for some reason, with SMP enabled and more then 1 core running, I'm getting very early random crash of the board, and I don't understand them.

Also I'm still trying to make my bus blaster/openocd combo work on this board so I can try to debug via jtag, but so far, no luck.

Any help is welcome :smileywink:

Best,

Vincent

Re: "Normal world" Linux for iMX6Q Lite

gary_bisson — Fri, 20 May 2016 18:07:32 GMT

Hi Vincent,

Sorry for the delay, I was attending the NXP FTF this week. I've seen that you've posted a new thread with a more explicit title, I think this is good. Also, I'd send that same post to the kernel mailing list to know if Shawn or Fabio or any other person have tried that before.

As for the JTAG, here is a post of someone using OpenOCD with Segger J-Link JTAG on SabreLite:

Using JTAG for imx6 board bringup

Regards,

Gary

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Sat, 21 May 2016 18:02:49 GMT

Thank you for that !

About the Jtag, I check the schematics of my adapter and there are quite some differences with BD's one, so we order an adapter from BD. I'll try as soon as we receive it !

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Fri, 24 Jun 2016 10:00:34 GMT

Dear gary_bisson,

did you have any time/luck investigating this issue ?

Re: "Normal world" Linux for iMX6Q Lite

gary_bisson — Fri, 24 Jun 2016 11:02:45 GMT

Hi Vincent,

No I unfortunately have no time to investigate this issue although I'm really interested in the outcome.

I know that some people have been able to use Linux in non-secure world in order to use TrustZone.

Linux in non-secure world, Missing prefetch abo... | NXP Community

imx6 Trustzone: How to boot smp kernel in normal world?

The first link might be of interest. Let me know how it goes.

Regards,

Gary

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Fri, 24 Jun 2016 21:44:38 GMT

Dear Gary, thank you for the links.

We already have moved the L2 cache code to the monitor and patched linux to perform the configuration using smc, but we might have missed a spot. I'll check on Monday.

But I'm not sure this is the same issue: one time I successfully booted in SMP, but the SD card was read-only, so Linux whole FS was read-only. When the SD card is not read-only, booting failed.

Any thoughts on this behavior could help :smileywink:

Best,

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Mon, 27 Jun 2016 14:11:58 GMT

Dear Gary,

I double-checked this L2 cache thing and we patched it (with a call to the monitor).

I attach to this message the patches I currently have on BD's linux vendor tree, along with the .config file:

repo git://github.com/boundarydevices/linux-imx6.git

branch boundary-imx_3.14.28_1.0.0_ga

revision a21d55a52d708d393b32bec813f8f13a0633d65c

With patches 0001 to 0005 (included), we can run our secure OS alongside Linux quite fine. The only issue is that Linux can't start the other cores. Patches 0006 is my current status at enabling SMP. I can't attach code of the secure os but, in a few words:

- I'm using the SRC module to wake up the secondary cores (in secure mode)

- I configure them to be usable in the normal world (same diag_register as cpu0, initialize the gic per cpu regs, nsacr configuration...)

- I make them loop until linux wake them up with an SGI. At this point, a switch to normal world happens and the core then jumps to Linux entry point.

CSU is fully opened (all csl registers are 00ff00ff and Type1 Masters are marked as Non Secure).

Am I missing something ?

For the records, our secure os works flawlessly alongside a QorIQ ls1021a in SMP mode with the same kind of boot mecanism.

Best,

Vincent

Re: "Normal world" Linux for iMX6Q Lite

gary_bisson — Mon, 27 Jun 2016 16:07:04 GMT

Hi Vincent,

Thanks for sharing your patches! Unfortunately I won't have time to look into it in a while.

However I want to emphasize that I doubt this problem is Sabre-Lite specific but instead I'm pretty sure it can be reproduced on any i.MX6 board (in case a person wants to try).

You might get more traction posting your patches on your mx6-generic post:

i.MX6Q SMP in normal world | NXP Community

Also, renaming your post to be "non-secure" instead of "normal", I believe this is the proper ARM term to describe what you want to achieve.

As for your problem, I wouldn't be surprised if L2 was involved in the issue since it is shared across the cores and that you have random behaviors (which is always the case with cache issues).

Regards,

Gary

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Mon, 27 Jun 2016 16:52:12 GMT

Okay, I'll look more into it. I'm not that used to cache configuration yet. I wonder something: my monitor is only on cpu0 and all SMC are directory scheduled to be performed by cpu0. Does the L2 cache needs access by *any* CPU or can I only access it using cpu0+smc, on behalf of another cpu ?

Re: "Normal world" Linux for iMX6Q Lite

darius-andreisu — Wed, 05 Oct 2016 21:02:43 GMT

Hey Vincent, have you had any luck on making SMP work in the normal world?

I am also working on making a linux 4.6 to work in the normal world, but on a IMX53 board.

I get strange hangs at boot-up, and am thinking of switching to a IMX6 board, and try using https://boundarydevices.com/imx-linux-kernel-4-1-15/

I see you have worked for a while on trying to make linux 3.10 work and have made quite some progress. Have you also tested the 4.1 version?

Thanks,

Darius

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Thu, 06 Oct 2016 07:06:19 GMT

Hi !

I'm still working on it but that's not my top priority. I had a big issue activating L2 cache & SCU but that's done, and now I have random "freeze" at boot (with a jtag, I can see that the core0 is not stopped, the kernel is printing lots of stuff on the uart, but nothing can be seen on my console). My guess is that Linux from the Normal Side uses the SRC component to wake up the other cores, still in TZ mode, and kaboom.

I have some code to remove this behavior, but I didn't have time to test it yet.

The minimal things you need to do from the TZ side for each core is to correctly initialize the GIC, the SCTLR & ACTLR registers (some bits are write secure only in the latter) and some other secure only registers like errata & stuff.

I'll let you know when I can resume this work.

Re: "Normal world" Linux for iMX6Q Lite

dariussuciu — Fri, 04 Nov 2016 20:22:00 GMT

Hey Vincent,

I changed to an i.MX6Q Nitorgen6 Max board and applied your linux patches(Thank you for sharing them!!) to a 4.1 kernel.

Now it almost starts fine (without SMP), but it gets stuck on one of the drivers (after "backlight_lcd supply power not found, using dummy regulator", waiting for an interrupt I think).

Could you share your secure world configuration?

(I set the CSL=0x00ff00ff and CSU_SA= 0x55555554). Is there something else (GIC? ) you are setting ?

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Mon, 07 Nov 2016 08:13:02 GMT

Hi!

We are slowly going on with SMP, I now see a couple of things that were missing / badly done (like a real monitor on each core, or cache invalidation at startup).

For the CSU, we set all CSL to "secure / non secure RW" just like you did, and we set all the Masters to "Normal world only" in the CSU_SA (without any lock).

If you don't have a monitor/secure world running along side your linux, you need to correctly configure the gic at boot time so it can be accessed by the normal world. iirc the minimal things to do are:

- set the group of SGI / PPI (first 32 irqs) to 1 (non secure)

- set the priority register in the GICC

- enable group 0 / 1 through IRQ in the GICC control register

Re: "Normal world" Linux for iMX6Q Lite

dariussuciu — Tue, 08 Nov 2016 23:43:22 GMT

Thank you very much Vincent!

I was missing some gic config u-boot has when set to run with "CONFIG_ARMV7_NONSEC" and now linux4.1 boots in the normal world.

Good luck with making the SMP work and thanks again!

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Wed, 09 Nov 2016 08:03:17 GMT

Ok, it is finally done.

The main thing I was missing was properly install a secure monitor on each core, because my patch about Linux accessing the SNVS RTC can be executed by any core, so there were smc performed on each core. Also, my cache maintenance routines for secondary cores was a bit buggy. This has to be fail proof.

To summarize:

- remove all secure access from Linux. In my case, it was access to the SCU, SRC, PL310 & SNVS. We decided to implem them using call to the secure monitor at runtime.

- start all core and "prepare" them to be runnable by Linux: correctly install a secure monitor, set the ACTLR/NSACR/GIC secure bits correctly and make it way (e.g. using WFI) for Linux to provide a jump address, and jump to this address using an ERET from Monitor Mode, so you can switch from secure code (wait loop in the monitor / secure kernel) to normal code for Linux.

- patch the way Linux starts the cores (if you prevent access to the SRC via CSU, you can implement them using an SMC call. For the time being, Linux still writes the address in the SRC but "wake up" the core using a SGI)

Re: "Normal world" Linux for iMX6Q Lite

fabio_estevam — Wed, 09 Nov 2016 13:01:21 GMT

Hi Vincent,

If you could share your patches, that would be great! Thanks

Re: "Normal world" Linux for iMX6Q Lite

vsiles — Wed, 09 Nov 2016 13:25:02 GMT

Here are my patch on linux (not completely cleaned up, I'm sorry) along with the current .config & git information. The board is a boundary device sabre_lite.

I am not allowed to share the code of the secure os / monitor but I can answer some questions if my previous answer is not clear enough.

Best,

Vincent