imx8ulp: rare reset by secure enclave when using SD boot

[martinetd]

Hi,

We're using imx8ulp in "flash_singleboot_m33"mode with:

m33 firmware is based on MCUXpresso SDK 2.16.000 MIMXRT700 (https://github.com/nxp-mcuxpresso/mcux-sdk commit 021190013e4d)

uboot 2023.04 based on lf-6.1.55-2.2.2

upower firmware-upower-1.3.0

ele firmware-ele-imx-0.1.1

To ensure we're booting in a clean state we're checking `CMC_RTD->SRS` in the m33 firmware and force a reset through PMIC (write 0x14 (Cold reset) to reg 9 (SW_RST))

This has been working well, e.g. when a software watchdog hits we notice this, (log a message) and trigger a cold reset to get in clean state.

However, with some SD cards (but not all the time...), the first boot fails with CMC_RTD->SRS = 0x40000030

According to the enums I found it means these bits:

#define CMC_SRS_WARM_MASK                        (0x10U)
/*! WARM - Warm Reset
 *  0b1..Reset generated by Warm Reset source.
 */
#define CMC_SRS_FATAL_MASK                       (0x20U)
/*! FATAL - Fatal Reset
 *  0b1..Reset was generated by a fatal reset source.
 */
#define CMC_SRS_SRR_MASK                         (0x40000000U)
/*! SRR - Secure enclave Reset Request
 *  0b1..Reset generated by SRR.
 */

When this happens, triggering a cold reset will reset, but it'll immediately trigger a new reset by SRR afterwards, making the soc reset endlessly.

For now I've added a check on the SRR bit and just skip triggering a cold reset in this case, which works, but I'd like to know why this reset happens in the first place as it seems dependent on hardware (doesn't happen with all SD cards, stops happening even with the same SD card if I rewrite the same firmware(!), doesn't happen on all of our boards even with a SD card that reproduces the issue reliably on one board...)

So:

- Is it possible to get more information other than that single bit? Does the reset leave some other registers we could dump? Given it's security related I doubt there is a fault handler we could catch a handler before reset like hardfault...

- If you (anyone) have any idea what this could be we'd be interested

Thank you,

Dominique

[AldoG]

Hello,

Is there any reason that you are using imxrt SDK instead of the 8ULP package?
Could you share the details of the SD cards that cause this issue?

Best regards/Saludos
Aldo.

[martinetd]

> Is there any reason that you are using imxrt SDK instead of the 8ULP package?

Sorry, I said imxrt because it was the most recent commit with a version in the git log, but we're using standard 8ulp code.

Our firmware is based on the power mode switch demo ( mcux-sdk/examples/evkmimx8ulp/demo_apps/power_mode_switch/ )

> Could you share the details of the SD cards that cause this issue?

We suspected the SD card as well, but it happened with two completely different models ("transcend 8GB class 4" and "samsung 128 evo plus U3"), which otherwise seem to work perfectly once the board has booted.

Also, rewriting the same firmware, changing board or just waiting a few hours often makes the issue go away so it's a bit difficult to try combinations.

Thinking about the issue naively the most likely candidate would be something like a checksum mismatch (invalid data read) loading some firmware? but we were looking at the SD card clk while this happened and the first reset happens pretty late (the clk had time to go from the default of ~400kHz to dozen of MHz range), so I'm not sure how much the ele is still involved with firmware validation this late?

Thank you

[AldoG]

Hello,

If SD card is working in some scenarios then we can discard a problem with them.

It may be indeed an issue with ELE FW, are you using Single boot with EdgeLock secure enclave FW?

Best regards/Saludos,
Aldo.

[martinetd]

Thank you for the response

> It may be indeed an issue with ELE FW, are you using Single boot with EdgeLock secure enclave FW?

I'm sorry, I'm not familiar with the terminology here.

We're building the boot image with this command:

make -f soc.mak SOC=iMX8ULP REV=A2 flash_singleboot_m33

after placing the following files in the imx-mkimage/iMX8ULP directory:

- mx8ulpa2-ahab-container.img: from firmware-ele-imx-0.1.1

- upower.bin: from firmware-upower-1.3.0

- m33_image.bin: built from a project derived from the 8ulp power_mode_switch demo

- u-boot.bin, u-boot-spl.bin and mkimage_uboot: built from u-boot derived from lf-6.1.55-2.2.0

I now see there's a firmware-ele-imx-2.0.1-0a66c34.bin update in the latest releases, I can't try to reproduce today but I'll retry with this version tomorrow and report back.

Does that answer your question?

Thanks

[AldoG]

Hello,

Yes, indeed this issue might be caused by using different ELE FW version, using different ELE versions are not allowed.

Did you had time to test using latest fw version?

Best regards/Saludos,
Aldo.

[martinetd]

Sorry for the delay, I have no updated to firmware-ele-imx-2.0.1-0a66c34/mx8ulpa2-ahab-container.img (sha256sum 87c8adae96fa9222440f4832d1699afa6406e4056142a48e9dc24eb16930c9ae) and could confirm I still reproduce the issue.

Thank you

[AldoG]

Hello,

One additional question on the SD cards that the issue is seen does only flash.bin gets loaded or there is a rootfs as well?
Are you testing downloading only flash.bin on clean SD cards?

Best regards/Saludos,
Aldo.

[martinetd]

Hello,

There is a full rootfs as well, but timing-wise I'm pretty sure the reset happens before linux is loaded (I've unfortunately not kept the trace but the reset is in the order of 10ms in, while u-boot is still doing its internal processing, maybe even before relocation)
Also, secure boot is not enabled so I believe kernel signing is not done (auth_cntr is not called)

I can try to blank the sd card and write only the boot image if you want to confirm

Thanks
Dominique

[AldoG]

Hello,

Yes, please try it.
Also, make sure to not mix versions of software/firmware for version 6.1.55-2.2.2 should be as following:

Uboot
Version: 2023.04
imx-atf
Version: 2.8
linux-imx
Version: 6.1
ele-imx
Version: 0.1.0
upower
Version: 1.3.1

Best regards/Saludos,
Aldo.

[martinetd]

Hi,

Okay, I've just checked with no rootfs (I just deleted the partition table so no kernel is being loaded) and can still reproduce.

For versions, I'm on the lf-6.1.55-2.2.2 tag for imx-atf (v2.8 alright), u-boot (2023.04 + some patches to support our board), imx-mkimage
upower is 1.3.1; there is no linux involved anymore.
I had upgraded ele-imx to 2.0.1-0a66c34 previously but reverted back to 0.1.0 and still reproduce with that as well.

Thank you