What can break branch instructions in Normal World when booting Linux?

[darius-andreisu]

I am trying to run Linux 4.6.7  on a IMX53-QSB board using u-boot 2016 as the bootloader. 

The it boots perfectly fine in Secure Mode, but after configuring u-boot to switch to Normal Mode before launching linux, it causes only certain branch instructions to not work (No instruction is executed at the branch target. The branch target points to the correct address), very early in the Linux start sequence. For example the branch to the start_kernel function hangs.

The configuration made before starting linux is the following:

CSU CSL_0-31: 0x00FF00FF

TZIC_INTSEC_0-3: 0xFFFFFFFF

TZIC_PRIOMASK: 0x1F1F1F1F

TZIC_PRIORITY_0-31: 0x1F1F1F1F

TZIC_INTCTRL: 0x80010001

TZIC_SYNCCTRL: 0x00000000

IIM_SCS1: 0x7D

M4IF_WMIS0: 0x80000000 (so no memory should be unrechable)

I previously used the configuration for Linux 2.6.35 and with u-boot-2009 and it worked fine. However,  we now require a Linux with a version >= 4.0 to run.

Do you know of some security register, that I missed setting, some Linux patches that need to be applied, or what else could be the cause of this problem?

Thank you for your help! 

[TomE]

How do you switch modes with this chip? The 70 Megabyte and 5100 page manual only mentions "secure mode" 10 times, and gives no details on what it is or how to change it.

Where do you get the "i.MX53 Security Reference Manual"? It doesn't show up on their site. The i.MX6 ones are available, but that's not the same thing.

Anyway, Google found this, which looks similar to your problem:

http://stackoverflow.com/questions/31589143/boot-linux-in-normal-world

Not surprisingly, it points back to a post on this site from 2013:

https://community.nxp.com/thread/308152

Tom

[darius-andreisu]

Thanks for the reply Tom,

I switch to the Normal World by setting the NS bit to 1 in the Secure Configuration Register.

The reference is available for download, but you need approval from NXP before getting it: Search Results: MCIMX53 Multimedia Applications Processor Security Reference Manual 

I have read that post also, but the branch target memory address  in my case points to the correct instruction in my case. I am able to move the value at that address and print it. Comparing it to the dissembled linux image, the instruction at the target address is the one I expect.

After some more testing, I thought there might be a caching issue, because adding some more assembly instructions (that print test messages) makes some branch instruction to work. However, the issue persists even after disabling the L2 cache, ICache and Dcache.

Thanks,

Darius

[TomE]

Thanks for the search pattern for finding that manual. It needed the exact wording ("MCIMX53" and not "i.MX53") to get a match.

Yes, the form demands "NXP Salesperson/FAE Name:". I don't have one of these. We're using the i.MX53 on a small board supplied by another company, so we're one removed from NXP.

NXP happily supplied me with the "Security Reference Manual for i.MX 6Dual, 6Quad, 6Solo, and 6DualLite Families of Applications Processors" without requiring that. So why is the i.MX53 more "secret" than the i.MX6?

> I thought there might be a caching issue

From your description it does sound like it. Have you tried fully flushing the caches (all of them) before switching modes rather than just disabling them? And somehow waiting for the flush to complete, as the CPU may be able to continue to execute ahead for many microseconds while the cache flush is proceeding.

Could the "Branch Target" or "Global History" parts of the "Program Flow Prediction" hardware be causing your problems? You can disable it.

Maybe you're actually getting an interrupt, and they're not set up to work properly in the "Normal World".

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0344i/Cacgacec.html

> I switch to the Normal World by setting the NS bit to 1 in the Secure Configuration Register.

What is the advantage in an embedded product to switching to "Normal Mode"? Who or what are you trying to defend against? Can you say or is this answered by "if I told you I'd have to ..."?

"Secure Configuration Register" and "NS" don't appear in the 5100 page "i.MX53 Multimedia Applications Processor Reference Manual Rev 2.1". I'm guessing this information must only be in the Security Reference Manual..

I'm wrong again. The "Security Configuration Register" is in CP15 in the ARM Core. That's in the "Technical Reference Manual". It says to read the "Architecture Reference Manual" to find out how it works. I'm reading that manual, and can't find anything that simply details how this works or how to use it.

> Secure Mode, but after configuring u-boot to switch to Normal Mode

Is that what NXP/Freescale call "TrustZone (Secure) Privilege (Supervisor) Mode" and "Non-TrustZone (Regular) Privilege (Supervisor) Mode"? No, I see you're using the ARM Core security stuff. There might be some overlap with TrustZone.

I see you're using the same words as in this article. Have you read it as it details what they had to do to a Linux Kernel to get it booting in the "non-secure world":

https://genode.org/documentation/articles/trustzone

It says they have to boot securely to a "Hypervisor" and have that switch and load the operating system. Might you need to do that?

Interestingly TrustZone doesn't allow them to trap and emulate accesses as the CPU doesn't seem to provide "Precise" exceptions in this case. But the i.MX53 implementation does allow this. That's good, but doesn't allow a "generic" solution to run on multiple ARM cores.They couldn't partition memory (with the "General" platform) either, as that isn't covered by the "Security" system. But they could do this on the i.MX53. Likewise the Interrupts can't be divided up. And the IPU/GPU are married and can't be assigned properly.

You need Kernel 4 or greater.

The last supported kernel (last supported software) for this chip is 2.6.35.

As you have found, the "Community" dropped support for this chip a year or more ago.

Why are you using this chip? There are more powerful members of the i.MX6 family, and they're cheaper too.

Tom

[darius-andreisu]

Thank you for the hints Tom, 

I tried disabling all caches from both u-boot and the kernel, and letting it run for a while, but the issue does not seem to be related to the cache in the end.

Also disabled the Branch Prediction, Program Flow prediction and other kernel features, but the issue still persists.

Interrupts are masked before starting the kernel by setting the interrupt priority register, so the kernel should not get any interrupts before it unmasks them.

>What is the advantage in an embedded product to switching to "Normal Mode"? Who or what are you trying to defend against? Can you say or is this answered by "if I told you I'd have to ..."?

The Normal Mode is supposed to hold an untrusted OS, linux in our case, which does not have full access to the entire memory and some peripherals. This allows us to use some software in the Secure Mode running on a very small  secure OS, where it would be protected from usual user and kernel access. 

>It says they have to boot securely to a "Hypervisor" and have that switch and load the operating system. Might you need to do that?

That is not necessary, we have also successfully run a  linux 2.6.35 in Normal Mode, but the problems arise after trying to make a 4.6 version work, even though it seemed to work in secure mode just fine on the i.MX53 board. Yes, the community seems to have dropped support, and there is another thread "Normal world" Linux for iMX6Q Lite , where Vincent Siles tried to make linux 3.14 work on an I.MX6 board. 

>Why are you using this chip? There are more powerful members of the i.MX6 family, and they're cheaper too.

This is a chip that we already had and on which we made linux 2.6.35 boot in Normal Mode, so we already had a starting point.  We are buying  and switching to a I.MX6 board and we will try to work on that board instead, since there is a official 4.1 linux support and most of the community is now working on this boards. The differences between 2.6 and 4.1 are too big, so trying to port the changes from one to another would be a very difficult task.

>NXP happily supplied me with the "Security Reference Manual for i.MX 6Dual, 6Quad, 6Solo, and 6DualLite Families of Applications Processors" without requiring that. So why is the i.MX53 more "secret" than the i.MX6?

I don't know why it would be more secret. We got it after signing and NDA, but now since we switch to a I.MX6 board, we need the Security Reference Manual for i.MX 6Dual, 6Quad. The process seems the same as for I.MX53, I am still "waiting for approval" to get the pdf, for 5 days now. Is it possible to share the one you got, since I don't know how long the approval process will take, or share the procedure that you used to get it, without signing an NDA? 

Thank you again for you help,

Darius

[TomE]

It looks like the experts in this matter are all on that other thread. This one (referenced from your link) looks closer to what might be happening to you:

https://app-community.nxp.com/thread/379983

> The process seems the same as for I.MX53, I am still "waiting for approval" to get the pdf,

> for 5 days now. Is it possible to share the one you got,

I don't think I can, especially since you've asked me in public! :-)

It isn't the same process. I just filled this page in:

http://www.nxp.com/webapp/sps/download/mod_download.jsp?colCode=IMX6DQ6SDLSRM&appType=moderatedWitho...

It just requested that my Profile be filled in. I got an answer within a few days. No requirement to involve a sales rep.

The manual doesn't seem to be all that different between the i.MX5 and i.MX6. It seems to detail TrustZone and not the ARM security stuff. You're not using TrustZone, so I think you need the ARM documentation rather than anything from NXP.

Tom

[Bio_TICFSL]

Hi Sucio,

Latest official MX53QSB BSP from nxp is based on 2.6.35 kernel version, for latest kernel version support you should try the Yocto Community BSP:

 meta-fsl-arm - Layer containing Freescale ARM hardware support metadata 

Regards

[darius-andreisu]

Thanks for the pointing to using Yocto, but unfortunately, looking at their commits, they seem to have dropped support for this board in 2015.

"2015-08-06: Drop Freescale official Linux kernel for i.MX23, i.MX28 and i.MX5 SoC families"

Thanks,

Darius