building u-boot image with trusty OS for imx8qx

[rob_mclean]

I've been trying to build a u-boot image including the trusty OS for an i.MX8QX processor with limited success.  My end goal is to run Android 9.0 on my custom target hardware with support for the Trusty TEE.  I've been taking direction/inspiration from the Android Users guide.  When I've needed to match hardware in those instructions, I've been using the Trusty version of the imx8xqxp_mek.

I can build a working u-boot image (without spl) that does not include the Trusty OS, so I'm confident that I have the proper ahab container, bl31 and scfw_tcm images for my hardware.  I assume that because my u-boot image without Trusty works, I've got u-boot (without spl) configured properly.

After I changed my u-boot configuration to include SPL, and I built the Trusty OS lk.bin, I am able to create a flash.bin when I include the tee.bin and u-boot-spl.bin by using this make command:

make SOC=imx8qx flash_spl

The file size seems to be big enough to have included all the extra binaries, but when I attempt to load that flash.bin onto my target, there is no output on the console.  I only know that the processor has attempted to do something because the current draw on my power supply jumps from 190mA to 250mA just after I attempt to load flash.bin.

I'm wondering if this is the expected behavior if there is a root of trust failure because the images aren't properly signed, or is some part of the SOC not enabled (like the CAAM) that should be because I haven't enabled it in the device tree?

[Yuri]

@rob_mclean 
Hello,
 

   use "i.MX_Linux_Users_Guide.pdf" and Chapter 5 (Configuring OP-TEE) of
"i.MX_Porting_Guide.pdf" in NXP Linux documentation for more details
about OP-TEE support.

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

Regards,
Yuri.

[rob_mclean]

Hi Yuri,

The i.MX_Porting_Guide mentioned above is located here: https://www.nxp.com/docs/en/user-guide/IMX_PORTING_GUIDE.pdf

After my first read through chapter 5 of the porting guide, I'm not sure if the u-boot configuration to support OP-TEE with Linux is applicable to my Trusty OS/Android application, but maybe I'm still missing something.

I've been taking inspiration for configuring my u-boot from the "imx8qxp_mek_android_trusty_defconfig" that comes with the u-boot distro.  That defconfig doesn't enable any of the TEE or OPTEE configs in u-boot.

I chose that defconfig because it matches my CPU, and it is referenced in the Android Users Guide located here: https://www.nxp.com/docs/en/user-guide/ANDROID_USERS_GUIDE.pdf

I've edited my initial post to include my end goal to help reduce confusion about what I'm trying to accomplish.

[rob_mclean]

I re-read this document in detail: https://www.nxp.com/docs/en/application-note/AN12312.pdf

I was not signing the third container (the one that contains the Trusty OS kernel) in any of my previous attempts.  I'm wondering if the brickish behavior I've been seeing with my device when using that 3 container u-boot image is normal behavior when the third container fails to authenticate even though I haven't programmed the hash into the OTP fuses.

Now I'm wondering if there is a way to bypass authentication of the containers and still get the Trusty OS loaded using the SPL enabled u-boot?

[rob_mclean]

after another reading of that same secure boot with AHAB app note I discovered section 4.

The second bullet point answers my question about bypassing authentication by saying if the lifecycle is in an open mode image authentication failure is detected but ignored.  My device is in the "NXP closed" security lifecycle just like the one in that example.  I think that explains why I see 1 SECO event with the "ahab_status" command using the unsigned version of u-boot that I have working.

As it relates to my goal of building a 3 container u-boot that uses SPL to load the Trusty TEE OS, I found the very end of section 3.1.1 of the Android Security Users Guide that shows an example of the "ahab_status" when a u-boot has 3 containers, and it shows that there are 2 SECO events one associated with the authentication failures for each of the unsigned containers (the first container is signed by NXP).

So I'm taking that to mean what I'm trying to do, has been done, but maybe not with Android version 9.0 r3.

[Yuri]

@rob_mclean 
Hello,

   all three containers should be signed.

Regards,
Yuri.

[rob_mclean]

Hi Yuri,

I assume the OTP fuses with the SRK hash need to be blown as in order for this to all work.  However, I'm a bit hesitant to blow the fuses if I don't have some reassurance that I can build a u-boot container that works.

I'm using this u-boot with the Android 9.0 release, and I didn't do anything to sign the second container in the non-SPL version (2 containers instead of 3) of u-boot that does work.  I don't see anything in the rest of the Android u-boot makefiles that is signing it, and I don't see the code signing tool in my Android build tree.

I'm guessing that authentication isn't working because I haven't blown the SRK hash fuses and set the security life cycle to OEM closed.  Which would  explain this output I'm seeing to the "ahab_status" call which I can make from my working version of u-boot:

=> ahab_status

Lifecycle: 0x0020, NXP closed

SECO Event[0] = 0x0087EE00
CMD = AHAB_AUTH_CONTAINER_REQ (0x87)
IND = AHAB_NO_AUTHENTICATION_IND (0xEE)

sc_seco_get_event: idx: 1, res:3
=> <INTERRUPT>

I assume that message pertains to the second container, because everything I've read about the first container says that the pre-built binary available for download is signed by NXP.

If all that is true, I think my problem must be that either I'm not building SPL correctly, or I'm not getting SPL to load everything it is supposed to load.

[rob_mclean]

I still haven't figured out if my problem is that my SPL is not getting located correctly by the linker, and/or if there are other problems getting SPL to start u-boot.  Also, this design uses UART2 as the console, which is working with u-boot, but I'm not sure if I've got it properly configured for SPL.

At this point I'm trying to separate out the Trusty OS, from building a boot loader that includes a container with SPL and a container with u-boot.  I feel like getting that working is the first step in getting the Trusty OS loaded by the bootloader.

At this point my device is bricked when I use the bootloader that includes SPL.  So I think what I need is help configuring u-boot with SPL to provide the correct binaries to the imx-mkimage utility.  Otherwise maybe with a more detailed description of the boot process I could figure out the configuration on my own.  I feel like the documentation I've seen doesn't describe where the images get located in memory (so that I can understand what to do with the linker script), and how the images get loaded into memory (so I can make sure that the addresses in the linker script match the code that puts the binaries at the right location).

[rob_mclean · Accepted solution]

Hi @Yuri ,

I made some small progress today in getting u-boot working from SPL.

I discovered the "u-boot,dm-spl" flag in the "arch/arm/dts/fsl-imx8qxp-mek-u-boot.dtsi" file.  It is described in some detail in the "doc/driver-model/README.txt" file in my u-boot source tree.

It seems that flag has something to do with keeping the driver code from getting relocated during during the boot process.  The above documentation didn't say much about when that relocation happens, or why.

When I blindly added it to all the nodes in my device tree that I think I might be using in SPL, SPL printed out the banner, and a bunch of other stuff giving me hints as to what is broken and what I should be looking at next.  The serial protocol used 1.2M baud instead of 115,200 baud that I have u-boot configured for.

This post is not the answer to all my questions about getting Trusty TEE OS working, but it does answer why SPL was bricking my device, and that was the first most basic problem.  Hopefully from here on out the documentation referenced here combined with basic troubleshooting of u-boot will get me the rest of the way.  If not I can always ask another question.

[rob_mclean]

I was finally able to get Trusty running (to a point) on my hardware.

I think I heard that u-boot provides native support for dispatching OPTEE, but I'm not sure if there is native support for dispatching the Trusty OS from u-boot.  I know that the Arm Trusted Firmware (ATF) does provide a Secure Payload Dispatcher (SPD) for trusty, so I configured the ATF to use that SPD.  In addition to that ATF change, I had to make sure all the u-boot device tree nodes were properly annotated with "u-boot,dm-spl".

In case this helps anyone: There were several obvious device tree nodes that needed the u-boot,dm-spl annotation, the one that took the longest to find was this one:

&{/imx8qx-pm} {
        u-boot,dm-spl;
        connectivity_power_domain{
                conn_sdhc0{
                        u-boot,dm-spl;
                };
        };
};

Without that one the eMMC was unusable because it appeared there there was a clock related to the eMMC that SPL wasn't able to enable.

To configure the ATF (bl31.bin) to use the trusty SPD meant I needed to add "SPL=trusty" to the make parameters for bl31 in the AndroidUboot.mk file found in the ${ANDROID_BUILD_TOP}/device/<vendor>/<hw>/<product> directory.  After those changes, SPL was able to properly load the third container with the Trusty OS, u-boot proper and ATF.  Then ATF properly kicked off the Trusty OS, and then jumped to u-boot proper. I was able to confirm this because u-boot reported that it was able to use IPC to communicate with the Trusty OS.

However, among the first things that Trusty and u-boot attempted to do was to initialize the RPMB keys in the eMMC by blowing those fuses.  Apparently I had accidentally blown those fuses at some point while I was experimenting.  So at the moment my hardware is unable to boot using Trusty TEE OS because the "destroyed" RPMB keys in the eMMC which cause u-boot to hang.  I have new hardware on the way which should allow me to complete testing.

So a word to the wise... Don't experiment with the RPMB fuses if you plan to use Trusty TEE, or else you might need to replace your eMMC.

[rob_mclean]

The problem with my console baud rate was a somewhat related problem in that the problem was that the device tree was modified so that the "clock-names" and "clk" properties were removed from the device tree nodes by the use of CONFIG_OF_SPL_REMOVE_PROPS.  I removed those from the list and the driver was able to find the clock speed for the console UART, and it was able to initialize the console to the correct baud rate.

[Yuri]

@rob_mclean 
Hello,

As for documentation - use i.MX Android™ Security User's Guide for Rev. P9.

 https://www.nxp.com/docs/en/supporting-information/android_P9.0.0_2.3.2_docs.zip 

I am afraid, we do not have more i.MX8/X documentation and examples.

Regards,
Yuri.