Trusty OS forces u-boot into recovery mode

[rob_mclean]

I've been successful at building/booting u-boot with Trusty for a while.  Recently I made a change to u-boot and that resulted in u-boot booting into recovery mode.

Doing the obvious, I suspected something was wrong with the change I made to u-boot, but when I backed out that change, the problem of booting into recovery mode persists.  Also, if I just change or delete  the "boot_recovery" u-boot variable and save that change to the u-boot environment, when I power cycle the device, the "boot_recovery" variable returns set to "yes".

There was something else odd with the u-boot environment.  It has some variables which have the Linux style continuation character in them, and are prefixed with the Linux command line double dashes.  Here are those suspicious u-boot environment variables:

recovery_command=recovery\
--prompt_and_wipe_data\
--reason=init_user0_failed\

I was hoping that by finding those strings in the code, I could determine what the problem was, but I was not able to find those strings in the code for U-Boot, Linux or Android.  However, I did find them in the Trusty image.  Specifically, they trace to the file: "<TRUSTY_ROOT>/trusty/prebuilts/aosp/android/out/target/product/trusty/system.img".  Since that is a prebuilt binary file, I can't tell what causes those variables to get placed in the u-boot environment.  Turns out that file comes from an AOSP repository, so I'm not sure I'll ever see the code for it.

I do have quite a bit of the debugging information from the Trusty LK turned on at boot.  Here's what I see at startup:

U-Boot SPL dub-2019.04-r2.1-g7a82280-dirty (May 24 2021 - 12:28:20 +0000)
Spaulding Normal Boot: 115200
Trying to boot from MMC1
Load image from MMC/SD 0x6fc00

welcome to lk/MP

boot args 0x2000000 0xfe000000 0x2000 0x0
generic_arm64_map_regs: failed -5 name=gic
initializing trusty (Built: 08:25:28 May 19 2021)
Initializing Trusted OS SMC handler
avb: 285: Initializing AVB App
creating port (com.android.trusty.avb)
int rebuild_hset_all(struct uctx *):250: Rebuilding all handles set
hwcrypto: 222: Initializing
hwrng_caam: 44: Init HWRNG service provider
hwrng_srv: 256: Start HWRNG service
creating port (com.android.trusty.hwrng)
int rebuild_hset_all(struct uctx *):250: Rebuilding all handles set
hwcrypto_caam: 85: Init HWCRYPTO service provider
hwcrypto_srv: 360: Start HWCRYPTO service
creating port (com.android.trusty.hwcrypto)
hwkey_caam: 213: Init HWKEY service provider
hwkey_srv: 351: Start HWKEY service
creating port (com.android.trusty.hwkey)
hwcrypto: 237: enter main event loop
creating port (com.android.trusty.keymaster.secure)
int rebuild_hset_all(struct uctx *):250: Rebuilding all handles set
creating port (com.android.trusty.keymaster)
trusty_gatekeeper: 291: Initializing
creating port (com.android.trusty.gatekeeper)
int rebuild_hset_all(struct uctx *):250: Rebuilding all handles set
creating port (com.android.trusty.storage.proxy)
int rebuild_hset_all(struct uctx *):250: Rebuilding all handles set


U-Boot dub-2019.04-r2.1-g7a82280-dirty (May 24 2021 - 12:28:20 +0000)

CPU: NXP i.MX8QXP RevB A35 at 1200 MHz at 34C
DRAM: 2 GiB
MCA: HW_VER=1 FW_VER=0.12
MMC: FSL_SDHC: 0, FSL_SDHC: 1
In: serial
Out: serial
Err: serial
Model: Spaulding Medical OneDevice.
ConnectCore 8X SOM variant 0x02: Industrial QuadXPlus, 16GB eMMC, 2GB LPDDR4, -40/+85C, Wireless, Bluetooth
Board version undefined, ID 2105
Boot: MMC0

BuildInfo:
- SCFW 494c97f3, SECO-FW dd989819, IMX-MKIMAGE d7f9440d, ATF 31192ca
- U-Boot dub-2019.04-r2.1-g7a82280-dirty

flash target is MMC:0
Net: Net Initialization Skipped
No ethernet found.
INFO Initializing Trusty device
INFO selected trusty api version: 3 (requested 3)
INFO Initializing Trusty IPC device
INFO Initializing RPMB storage proxy service
_Bool super_block_valid(const struct block_device *, const struct super_block *):154: bad magic, 0x8c6a2ba476cc72f0
int fs_init_from_super(struct fs *, const struct super_block *, _Bool):316: loaded super block version 1
creating port (com.android.trusty.storage.client.tp)
creating port (com.android.trusty.storage.client.tdea)
INFO Initializing Trusty AVB client
INFO Initializing Trusty Keymaster client
INFO Initializing Trusty Hardware Crypto client
Fastboot: Normal

******************************************
* Warning: Booting into recovery mode... *
******************************************

Hit any key to stop autoboot: 0
=>

Does this sound familiar to anyone?

[billchen]

what you called it " Trusty OS" , I guess it's LK , 

[rob_mclean]

I should add that I thought I had seen something similar previously.  I thought I remember being able to fix it by erasing and reprogramming the Android userdata and cache images along with the bootloader, and resetting the u-boot environment to the defaults with "env default -a -f".

I tried doing that several times (in several different ways) with this problem, and the problem is still there.

[rob_mclean · Accepted solution]

I found the problem, and it wasn't directly related to Trusty afterall.

It turned out that our custom partition sizing had only allocated 2M for the bootloader. That was enough for the 1.1M u-boot without Trusty OS, but u-boot with Trusty OS was 2.4M.  Given that scenario, I'm not even sure how the device was working at all.

When I increased the space allocated for the bootloader to 8M, the problem went away.