"Normal world" Linux for iMX6Q Lite

[vsiles]

I got an iMX6Q Lite from Boundary Devices, and I am using their Ubuntu based released, which is working fine. Their Linux seems to be based on NXP/Freescale's Linux tree, which completely run in secure world.

I'm trying to port the Linux into the normal world, but I can't seem to make SMP work. If I disable SMP, I'm fine.

If I enable SMP but only core0 to be started, I'm fine. But as soon as I try to start a second core, I randomly crash during init. My best run was having a prompt, but can't login due to an infinite error:

nitrogen login: root
login: PAM Fa�lure, aborting: Criinit: ttymxc1 main process (916) terminated with status 99
init: ttymxc1 main process ended, respawning

Are there know Linux BSP for iMX6 boards that are running in the normal world ?

Best,

Vincent

PS: here is a summary of my changes into Linux to run it in normal mode:

- remove access to the diagnostic register, they are done prior to Linux by the secure world

- remove access to the L2 cache PL310, they are replaced by SMC commands to the monitor which performs them

- remove access to the SCU/SCR registers to enable the second core. Secondary cores are prepared by the secure world to be woken up using an IRQ, and Linux code is replaced by a call to  arch_send_wakeup_ipi_mask(cpumask_of(cpu));

[vsiles · Accepted solution]

Ok, it is finally done.

The main thing I was missing was properly install a secure monitor on each core, because my patch about Linux accessing the SNVS RTC can be executed by any core, so there were smc performed on each core. Also, my cache maintenance routines for secondary cores was a bit buggy. This has to be fail proof.

To summarize:

- remove all secure access from Linux. In my case, it was access to the SCU, SRC, PL310 & SNVS. We decided to implem them using call to the secure monitor at runtime.

- start all core and "prepare" them to be runnable by Linux: correctly install a secure monitor, set the ACTLR/NSACR/GIC secure bits correctly and make it way (e.g. using WFI) for Linux to provide a jump address, and jump to this address using an ERET from Monitor Mode, so you can switch from secure code (wait loop in the monitor / secure kernel) to normal code for Linux.

- patch the way Linux starts the cores (if you prevent access to the SRC via CSU, you can implement them using an SMC call. For the time being, Linux still writes the address in the SRC but "wake up" the core using a SGI)

[ailtonlopes]

Hey vsiles‌ can you give me some help with the SMP,  i got linux working fine with the core 0, but when i jump to the jump address set by linux on the socondary cores the kernel crashes. I think that it may be something that im missing but im not sure, can you give me some hints since you've already solve that problem and also what values are you giving to SCLR and NSACR.
Me i did the same as in the core 0 but it still crashing, im also curious if i should set the SMP bit on SCLR or linux will do that, i set the bit to allow ns access to SCLR. 

Kind regards,
Ailton Lopes

[vsiles]

I won't be able to tell much since most of the code happens in the secure monitor / secure kernel and I can't write about it.

In my scenario, we only have a NS/S sharing on core 0. All other cores are 100% under NS control, so the main idea is that you need to set a minimal configuration to allow the normal world to work correctly. The main things to initialize are the GIC (you need to allow NS access to interrupts / update the groups), NSACR (quite obvious, it depends on what you want to allow the NS work to access) and ACTLR (for the SMP bit). But sure to launch linux with cache flushed, MMU off and access to the GIC.

[fabio_estevam]

Hi Vincent,

If you could share your patches, that would be great! Thanks

[vsiles]

Here are my patch on linux (not completely cleaned up, I'm sorry) along with the current .config & git information. The board is a boundary device sabre_lite.

I am not allowed to share the code of the secure os / monitor but I can answer some questions if my previous answer is not clear enough.

Best,

Vincent

[mach]

Hello Vincent,

I have applied your patches, and set the GICD_SGIR=14 | 1<<(cpu + 16) to generate SGI (GICD_CTRL=0x3) in secure world(optee_os). However, it didn't generate a sgi.

Could you share more details about booting second cpu from secure world?

Thanks.

[vsiles]

Hi !

Did you used the SRC component (have a look at the ref manual) to start the secondary cores ? The main scheme here is:

• from secure world, start the secondary cores using SRC
• configure them to be runnable by Linux in the Normal World (NSACR, Gic, ...)
• make them wait on a wfi based loop
• from the normal world: set the jump address of the secondary cores and send an sgi to break the wait loop

Only sending the SGI from Linux won't start the core, you have to do some work on the secure world (optee in your case) first. I can't share the code of this part I'm afraid, but you should have enough information in this thread to have the whole idea. If you still fail, I'll detail it a bit further

[mach]

Thank you for your patience, Vincent.

Now, other three cores are on a wfi based loop by SRC in secure world. But when I send 14 SGI from linux to break the wait loop, cpu1/2/3 can't enter irq_handler (by secure or nonsecure vector base address), and remain on a wfi based loop . I have config NSACR/GIC/... and set the GICD_SGIR=14 | 1<<(cpu + 16). 

[vsiles]

I don't advise to have the "wfi" loop with irq/fiq enabled. You should make sure they are both disable, so that when the irq happens, you know exactly which code wil be executed. My wfi loop looks like

1) store a ref / dummy value is r0

2) wfi

3) read the place where the jump address should have been written by linux (I still use the SRC register for this atm)

4) if this address is != r0, then goto 1)

5) else you have a jump address. Go to monitor mode and configure everything to perform a return to SVC Non Secure

[mach]

Hello Vincent,

Follow your steps, I implemented the function of wfi based loop in secure world (not in monitor). There are two questions:

The first is why we should make cpu1/2/3 stay at wfi state, rather than jumping to non-secure world directly by judging SRC_GPR? Even though I failed by using the following code, I guess that the cause is lack of some configures from secure world to non-secure world. So the second question is about the configures.

The reason why I don't want to use wfi is that I send SGI failed, but I observed that every core has a secure or nonsecure vector base address by DS-5.

My kernel's version is 4.1.15 and the board is imx6q-sdb.

I'm sorry to ask you questions again, but it's important for me and I tried many times, Thanks.

FUNC cpu_jump_addr , :

UNWIND( .fnstart)

//judge which cpu by SRC_SCR, so that we can read jump_addr

   ldr r0, =0x020D8000

   ldr r1, [r0, #SRC_SCR]

   and r2,r1,#0x00800000

   cmp r2,#0

   beq cpu1

   and r2,r1,#0x01000000

   cmp r2,#0

   beq cpu2

   mov r3,#0x38

   b cpu

cpu1:

   mov r3,#0x28

   b cpu

cpu2:

   mov r3,#0x30

cpu:

   add r3,r3,r0

   ldr r2,=cpu_jump_addr

loop:

   //wfi

   ldr r1, [r3]

   cmp r1,r2

   beq loop

   mrs r2, cpsr

   bic r2, r2, #0xff

   orr r2, r2, #0x93 //93 when nmi is enabled, or 0xd3

   msr cpsr,r2

   mrc p15,0,r2,c1,c1,0

   orr r2, r2, #0x25

   mcr p15,0,r2,c1,c1,0

   mov pc,r1

UNWIND( .fnend)

END_FUNC cpu_jump_addr

[vsiles]

Thank you for that !

About the Jtag, I check the schematics of my adapter and there are quite some differences with BD's one, so we order an adapter from BD. I'll try as soon as we receive it !

[gary_bisson]

Hi Vincent,

Can you please clarify which board you are using? We have a Nitrogen6_Lite which is a i.MX6 Solo board, not sure what is the i.MX6Q Lite.

Why do you need to run on non-secure world? I think it is supported by the mainline kernel, have you tried it? Our Nitrogen6x board is supported in mainline kernel but know that some features are missing like the use of the GPU.

Regards,

Gary

[vsiles]

Dear Gary,

I just checked mainline Linux and the issue is the same: it is made to work in the normal (which is fine since Linux is made to work alone, with no monitor/secure world blob, it should run by itself). For example in arch/arm/mach-imx/platsmp.c, you can see that the imx boards are using the "SRC" module to completely reset the secondary cores (which is secure only in a trustzone case), whereas the ls1021a boards are just issuing an IPI to the secondary cores to wake them up, which is fine.

I'm looking for a Linux kernel where Linux only uses "normal" world resources. At the moment, I'm trying to fix Linux's SMP management to be able to boot all 4 cores of my board, but for some reason, with SMP enabled and more then 1 core running, I'm getting very early random crash of the board, and I don't understand them.

Also I'm still trying to make my bus blaster/openocd combo work on this board so I can try to debug via jtag, but so far, no luck.

Any help is welcome :smileywink:

Best,

Vincent

[gary_bisson]

Hi Vincent,

Sorry for the delay, I was attending the NXP FTF this week. I've seen that you've posted a new thread with a more explicit title, I think this is good. Also, I'd send that same post to the kernel mailing list to know if Shawn or Fabio or any other person have tried that before.

As for the JTAG, here is a post of someone using OpenOCD with Segger J-Link JTAG on SabreLite:

Using JTAG for imx6 board bringup

Regards,

Gary

[vsiles]

Dear gary_bisson​,

did you have any time/luck investigating this issue ?

[gary_bisson]

Hi Vincent,

No I unfortunately have no time to investigate this issue although I'm really interested in the outcome.

I know that some people have been able to use Linux in non-secure world in order to use TrustZone.

Linux in non-secure world, Missing prefetch abo... | NXP Community

imx6 Trustzone: How to boot smp kernel in normal world?

The first link might be of interest. Let me know how it goes.

Regards,

Gary

[vsiles]

Dear Gary,

I double-checked this L2 cache thing and we patched it (with a call to the monitor).

I attach to this message the patches I currently have on BD's linux vendor tree, along with the .config file:

repo git://github.com/boundarydevices/linux-imx6.git

branch boundary-imx_3.14.28_1.0.0_ga

revision a21d55a52d708d393b32bec813f8f13a0633d65c

With patches 0001 to 0005 (included), we can run our secure OS alongside Linux quite fine. The only issue is that Linux can't start the other cores. Patches 0006 is my current status at enabling SMP. I can't attach code of the secure os but, in a few words:

- I'm using the SRC module to wake up the secondary cores (in secure mode)

- I configure them to be usable in the normal world (same diag_register as cpu0, initialize the gic per cpu regs, nsacr configuration...)

- I make them loop until linux wake them up with an SGI. At this point, a switch to normal world happens and the core then jumps to Linux entry point.

CSU is fully opened (all csl registers are 00ff00ff and Type1 Masters are marked as Non Secure).

Am I missing something ?

For the records, our secure os works flawlessly alongside a QorIQ ls1021a in SMP mode with the same kind of boot mecanism.

Best,

Vincent

[gary_bisson]

Hi Vincent,

Thanks for sharing your patches! Unfortunately I won't have time to look into it in a while.

However I want to emphasize that I doubt this problem is Sabre-Lite specific but instead I'm pretty sure it can be reproduced on any i.MX6 board (in case a person wants to try).

You might get more traction posting your patches on your mx6-generic post:

i.MX6Q SMP in normal world | NXP Community

Also, renaming your post to be "non-secure" instead of "normal", I believe this is the proper ARM term to describe what you want to achieve.

As for your problem, I wouldn't be surprised if L2 was involved in the issue since it is shared across the cores and that you have random behaviors (which is always the case with cache issues).

Regards,

Gary

[vsiles]

Okay, I'll look more into it. I'm not that used to cache configuration yet. I wonder something: my monitor is only on cpu0 and all SMC are directory scheduled to be performed by cpu0. Does the L2 cache needs access by *any* CPU or can I only access it using cpu0+smc, on behalf of another cpu ?

[darius-andreisu]

Hey Vincent, have you had any luck on making SMP work in the normal world?

I am also working on making a linux 4.6  to work in the normal world, but on a IMX53 board.

I get strange hangs at boot-up, and am thinking of switching to a IMX6 board, and try using https://boundarydevices.com/imx-linux-kernel-4-1-15/ 

I see you have worked for a while on trying to make linux 3.10 work and have made quite some progress. Have you also tested the 4.1 version?

Thanks,

Darius