i.MX6ULL and uboot-imx 2020.04 Does not work with hab_failsafe

[JohnKlug]

I wasn't sure if we had a hardware problem or not, so I tried the Variscite Dart board with NAND Flash, and it also fails.

I set the keys, and hab_status shows everything is OK.  I also did

fuse prog 0 6 0x2

To close the device.

This is what I see (which matches our boards behavior):

C:\NXP_UUU>uuu -lsusb
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.43-0-ga9c099a

Connected Known USB Devices
        Path     Chip    Pro     Vid     Pid     BcdVersion
        ==================================================
        1:143    MX6ULL  SDP:    0x15A2 0x0080   0x0001




C:\NXP_UUU>uuu mlinux-mtcap3-test-image-mtcap3-20210729190502-uuu.zip
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.43-0-ga9c099a

Success 0    Failure 1


1:143    1/ 1 [HID(R):LIBUSB_ERROR_TIMEOUT           ] SDP: boot -f u-boot-usb-signed.imx

[JohnKlug]

The device is closed.  BT_FUSE_SEL has selected Boot From Fuses.  One can change BOOT_MODE[0]  from low to high to select serial download mode.  Then the device will progam with UUU  on power up.  It is not possible that my images are not signed (serial download image has different CSF requirements from NAND), because if I substitute an unsigned image, serial download mode does not work.

So the problem must be in U-Boot or in the ROM.

It is desirable that one be able to program the device in the event of a problem image without opening the chasis to get at the BOOT_MODE[0] select.

I am using NAND flash.

[igorpadykov]

from team:

----------------

Does customer mean they can make device to go into SDP mode, but can't boot a bootloader by UUU?

Like what I said before, customer can try to boot a bootloader firstly on an open device, and then sign this bootloader, and then use this bootloader to burn the images to NAND flash with UUU.

Customer can check "HAB v4 closed chip support" in UUU document here[1].

[1]https://github.com/NXPmicro/mfgtools/wiki/UUU-default-support-protocol-list

----------------

Best regards
igor

[JohnKlug]

If the CPU is placed in serial download mode, the NAND can be written.  However this requires an intervention (opening the case) to change the pin setting.  If an image fails to verify you are supposed to be able to select hab_failsafe (A ROM function) from U-Boot:

=> help hab_auth_img_or_fail
hab_auth_img_or_fail - authenticate image via HAB on failure drop to USB BootROM mode

Usage:
hab_auth_img_or_fail addr length ivt_offset
addr - image hex address
length - image hex length
ivt_offset - hex offset of IVT in the image
=> help hab_failsafe
hab_failsafe - run BootROM failsafe routine

Usage:
hab_failsafe

This function is documented in Table 5, page 4 of AN 12263, hab_rvt.failsafe.

I don't see how this can be a signing issue, since the USB will boot in serial download mode.  The problem is a USB timeout from hab_rvt.failsafe, which is called by U-Boot.  I can tell that failsafe is called, since the VID, PID is advertised to Linux.  However, there is no response when uuu contacts the CPU.

[igorpadykov]

from team

----------------------

Maybe customer's board is hang when the second bootloader initialize the DDR again, so please let customer use below commands step by step in UUU. the difference between below commands and SDP:boot is that we don't initialize DDR

-----.

original boot cmd is bundle of three command

Sdp: dcd -f uboot

Sdp: write -f uboot

Sdp: jump -f uboot -ivt

now customer can try to use

Sdp: write -f uboot

Sdp: jump -f uboot -ivt

please also sign the uboot.

----------------------

Best regards
igor

[JohnKlug]

"SDP: write -f u-boot-usb-signed.imx" also fails.

I tried this with both our board and Variscite.  The result is the same.  Also, it does not matter if uuu is started first or hab_failsafe:

jklug@jklugDesk:~$ uuu  -lsusb
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.107-15-gd1c466c

Connected Known USB Devices
	Path	 Chip	 Pro	 Vid	 Pid	 BcdVersion
	==================================================
	3:12	 MX6ULL	 SDP:	 0x15A2	0x0080	 0x0001

jklug@jklugDesk:~$ uuu uuu.d
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.107-15-gd1c466c

Success 0    Failure 1                                                                                                                        
                                                                                                                                               
                                                                                                                                               
3:12     1/ 2 [HID(W):LIBUSB_ERROR_TIMEOUT           ] SDP: write -f u-boot-usb-signed.imx              


jklug@jklugDesk:~$ uuu -lsusb
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.107-15-gd1c466c

Connected Known USB Devices
	Path	 Chip	 Pro	 Vid	 Pid	 BcdVersion
	==================================================
	3:12	 MX6ULL	 SDP:	 0x15A2	0x0080	 0x0001

[igorpadykov]

Hi John

could you please provide full part number.

Best regards
igor

[JohnKlug]

Variscite markings on Chip:

MCIMX6Y2DMV09AB

Our custom board:

MCIMX6Y1CVM05AB

bash-5.0# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 24.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5

Hardware : Freescale i.MX6 Ultralite (Device Tree)
Revision : 0000
Serial : 301a79d2677f0fb5

[igorpadykov]

from team:

----------------

Could you please let customer dump the ROM log and HAB persistent memory by debugger after they try to download image in fail safe mode?

ROM log. Start address 0x00901CF4 , size 1k

HAB log: Start address 0x904000, size 0xb80

.........................

I don't have socket board to try customer's silicon(rev1.2), but I do see the similar issue with i.MX6ULL version(rev 1.0). 

The issue I see is that HAB library is in wrong state after we call the hab_failsafe to go into SDP mode. then it will have HAB event to block the device boot by UUU in closed state.

As I don't have closed device on hand, I use an open device to test and I see the HAB events disappear with below patch by 

----

diff --git a/arch/arm/mach-imx/hab.c b/arch/arm/mach-imx/hab.c

index 664444bfea..c5aa43b333 100644

--- a/arch/arm/mach-imx/hab.c

+++ b/arch/arm/mach-imx/hab.c

@@ -185,8 +185,9 @@ void hab_rvt_failsafe(void)

                return;

        }

#endif

-

+       hab_rvt_entry();

        save_gd();

------

Please let customer also notice that they need to sign the DCD part of the image. like below

Blocks =  0x877ff400 0x00000000 0x00085c00  "u-boot-dtb.imx" , \

          0x00910000 0x0000002c 0x000001e8  "u-boot-dtb.imx"

-----

After boot a signed bootloader by UUU in failsafe mode. There is no HAB events. Please let customer try on their side. Thank you.

---

=> hab_status

Secure boot disabled

HAB Configuration: 0xf0, HAB State: 0x66
No HAB Events Found!

----------------

Best regards
igor

[JohnKlug]

I applied the patch.  I have attached a dump of memory after the new hab_failsafe fails.

Now USB will no longer enumerate after hab_failsafe:

jklug@jklugDesk:~$ dmesg -w
[374062.118933] usb 3-12: new high-speed USB device number 108 using xhci_hcd
[374067.230687] usb 3-12: device descriptor read/64, error -110
[374082.445953] usb 3-12: device descriptor read/64, error -110
[374082.661930] usb 3-12: new high-speed USB device number 109 using xhci_hcd
[374087.773695] usb 3-12: device descriptor read/64, error -110
[374102.988926] usb 3-12: device descriptor read/64, error -110
[374103.092919] usb usb3-port12: attempt power cycle
[374103.716871] usb 3-12: new high-speed USB device number 110 using xhci_hcd
[374108.724643] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[374113.940381] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[374114.144377] usb 3-12: device not accepting address 110, error -62
[374114.256363] usb 3-12: new high-speed USB device number 111 using xhci_hcd
[374119.268128] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[374124.483930] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[374124.687838] usb 3-12: device not accepting address 111, error -62
[374124.687868] usb usb3-port12: unable to enumerate USB device

[igorpadykov]

from team:

-------------

Also please let customer do not connect JTAG during the test, this is because JTAG
connection will trigger SNVS HP violation in closed state.

It seems like that the ROM log address is changed in rev1.2 chip, please let customer
read the ROM log base address on 0x1E0, and dump ROM log from this base address
and size can be 1kB. I suspect the base address for rev1.2 is 0x00901d14, but need customer to test.

I see the customer's log after they applied the patch. Based on the HAB log in that file,
I see the HAB event is disappeared as expected, but customer said that the device doesn't go into SDP mode.

I need to test with closed device in next week. but currently, please let customer dump the
ROM log with the correct address like I said in previous reply.

-------------

Best regards
igor

[igorpadykov]

additional suggestion from team:

-------------------

I tried with closed i.MX6UL rev1.2 chip on socket board today as I don't have i.MX6ULL rev 1.2 chip here. I see the ROM is updated from rev 1.0 which I tested before. Finally, I can make it work by below steps, please let customer try on their side.

1) Apply the patch I shared before

2) Sign the image with below CSF file. <-the same signed image(test_6ul_nodcd.bin) used for boot and failsafe.

---

[Header]
Version = 4.2
Hash Algorithm = sha256
Engine Configuration = 0
Certificate Format = X509
Signature Format = CMS
Engine = SW

[Install SRK]
# Index of the key location in the SRK table to be installed
File = "../crts/SRK_1_2_3_4_table.bin"
Source index = 0

[Install CSFK]
# Key used to authenticate the CSF data
File = "../crts/CSF1_1_sha256_2048_65537_v3_usr_crt.pem"

[Authenticate CSF]

[Unlock]
Engine = CAAM
Features = RNG, MID

[Install Key]
# Key slot index used to authenticate the key to be installed
Verification index = 0
# Target key slot in HAB key store where key will be installed
Target Index = 2
# Key to install
File= "../crts/IMG1_1_sha256_2048_65537_v3_usr_crt.pem"

[Authenticate Data]
# Key slot index used to authenticate the image data
Verification index = 2
# Authenticate Start Address, Offset, Length and file
Blocks = 0x877ff400 0x00000000 0x00086c00 "u-boot-dtb.imx"

-----

 3) burn the SD card and boot up to uboot console.

4) run "hab_status" command, there should be no HAB event.

5) run "hab_failsafe" command, the chip should go into SDP mode

6) in UUU console. type in "uuu.exe SDP: write -f test_6ul_nodcd.bin -ivt 0", there should no issue

7) in UUU console, type in "uuu.exe -v SDP: jump -f test_6ul_nodcd.bin -ivt", we can see the image is boot from failsafe.

Full log is 

---

U-Boot 2020.04-00001-g44f5949dd9-dirty (Aug 23 2021 - 03:06:55 -0700)

CPU: i.MX6UL rev1.2 528 MHz (running at 396 MHz)
CPU: Industrial temperature grade (-40C to 105C) at 40C
Reset cause: POR
Model: Freescale i.MX6 UltraLite 14x14 EVK Board
Board: MX6UL 14x14 EVK
DRAM: 512 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

[*]-Video Link 0 (480 x 272)
[0] lcdif@21c8000, video
In: serial
Out: serial
Err: serial
switch to partitions #0, OK
mmc0 is current device
flash target is MMC:0
Net: 
Error: ethernet@20b4000 address not set.

Error: ethernet@20b4000 address not set.

Error: ethernet@20b4000 address not set.
FEC: can't find phy-handle

Error: ethernet@20b4000 address not set.
Could not get PHY for FEC0: addr 2

Error: ethernet@20b4000 address not set.
FEC: can't find phy-handle

Error: ethernet@20b4000 address not set.
Could not get PHY for FEC0: addr 2
No ethernet found.

Fastboot: Normal
Normal Boot
Hit any key to stop autoboot: 0 
=> hab_status

Secure boot enabled

HAB Configuration: 0xcc, HAB State: 0x99
No HAB Events Found!

=> hab_failsafe

U-Boot 2020.04-00001-g44f5949dd9-dirty (Aug 22 2021 - 20:19:07 -0700)

CPU: i.MX6UL rev1.2 528 MHz (running at 396 MHz)
CPU: Industrial temperature grade (-40C to 105C) at 43C
Reset cause: POR
Model: Freescale i.MX6 UltraLite 14x14 EVK Board
Board: MX6UL 14x14 EVK
DRAM: 512 MiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

[*]-Video Link 0 (480 x 272)
[0] lcdif@21c8000, video
In: serial
Out: serial
Err: serial
Card did not respond to voltage select!
flash target is MMC:1
Card did not respond to voltage select!
MMC card init failed!
Card did not respond to voltage select!
** Block device MMC 1 not supported
Net: 
Error: ethernet@20b4000 address not set.

Error: ethernet@20b4000 address not set.

Error: ethernet@20b4000 address not set.
FEC: can't find phy-handle

Error: ethernet@20b4000 address not set.
Could not get PHY for FEC0: addr 2

Error: ethernet@20b4000 address not set.
FEC: can't find phy-handle

Error: ethernet@20b4000 address not set.
Could not get PHY for FEC0: addr 2
No ethernet found.

Fastboot: Normal
Boot from USB for mfgtools
*** Warning - Use default environment for mfgtools
, using default environment

Run bootcmd_mfg: run mfgtool_args;if iminfo ${initrd_addr}; then if test ${tee} = yes; then bootm ${tee_addr} ${initrd_addr} ${fdt_addr}; else bootz ${loadaddr} ${initrd_addr} ${fdt_addr}; fi; else echo "Run fastboot ..."; fastboot 0; fi;
Hit any key to stop autoboot: 0

## Checking Image at 86800000 ...
Unknown image format!
Run fastboot ...
=> hab_status

Secure boot enabled

HAB Configuration: 0xcc, HAB State: 0x99
No HAB Events Found!

=>

----

-------------------

Best regards
igor

[JohnKlug · Accepted solution]

With some modifications, your last post does work.

DCD's are required for serial download boot.  Using your new CSF file does not change that.  NAND boot does not require DCD's.  Apparently including DCD's also breaks the hab_failsafe boot.  How was one to know this?  I made the assumption that hab_failsafe would be the same as serial download, but it is not.

So the hab_failsafe boot works with the same U-Boot image as the NAND boot image, which does not work with the serial download boot.

The following in your CSF file makes no difference so I did not change ours (this was the only other difference between your CSF file and our two CSF files):

[Unlock]
Engine = CAAM
Features = RNG, MID

Changes required for hab_failsafe boot (to summarize):

1. Do not use DCD's with hab_failsafe. Use NAND boot image, not serial download image which has DCD's.
2. Use hab_rvt_failsafe patch for hab_failsafe in U-Boot.
3. Use the following uuu commands to boot the image during hab_failsafe:

SDP: write -f u-boot-nand-signed-pad.imx -ivt 0
SDP: jump -f u-boot-nand-signed-pad.imx

Here is the serial download command to boot using uuu:

SDP: boot -f u-boot-usb-signed.imx

Note that the nand image has no DCD's, and the serial download (USB) image has DCD's.  Also, the NAND image must be padded, and the usb serial download image does not require padding.  Presumably hab_failsafe does not require padding, but I did not test this.

[john_kaye]

John: I've been following the thread and I've found it very useful, thanks! What size certificates are you using? I'm able to boot my closed imx6ULL board when in SDP and failsafe modes, but the board will not boot from an initial power-on or cold boot state. When it fails to cold boot, it enumerates as if in SDP mode, but I'm unable to use uuu to recover it, I actually have to force it into SDP mode by setting the boot mode pin and doing a POR. I'm curious if you've run into this. Thanks, -John

[JohnKlug]

I am using 4096 byte certificates.

For normal serial download you must have DCD's in the U-Boot CSF file.

For hab_failsafe you must not use DCD's.  I found that my NAND boot image worked from hab_failsafe, though I am planning on using a special image, with most U-Boot features eliminated from the NAND boot image.

[john_kaye]

Thanks. I believe that I've done my due diligence validating that my images are valid prior to closing my device, but there is a difference in behavior between a closed and open device -- my closed device won't cold boot but can boot from a warm POR. I wonder if there are additional fuse settings that must be made. My eMMC-based device is configured to boot from fuses rather than internal boot and BT_MMU_DISABLE is set.

[JohnKlug]

We are not doing SD card boot.  The point of my post is that we are getting a USB timeout on the serial download.

We have a NAND implementation (our implementation has no SD card access):

3) burn the SD card and boot up to uboot console.

This has never worked for me for USB boot:

# Authenticate Start Address, Offset, Length and file
Blocks = 0x877ff400 0x00000000 0x00086c00 "u-boot-dtb.imx"

For USB boot you need both HAB blocks and DCD blocks:

From my post on Friday (which works when doing serial download when not in hab_failsafe mode):

Blocks = 0x877ff400 0x00000000 0x000a0c00 "u-boot.imx", \
             0x00910000 0x0000002c 0x00000200 "u-boot.imx"

 
I will try to get the rest of my CSF file to match your suggestion.

[JohnKlug]

Here is the dump you requested.

JTAG will not connect when the CPU is in hab_failsafe.  I had to boot to U-Boot, connect the JTAG, then call hab_failsafe (which caused a USB timeout error as usual).  Then I was able to get the dump.

The dump is attached.  I have not yet applied your patch.  I will try that next.

[JohnKlug]

We already know about the two line "blocks =" in the u-boot.csf.  The boot from serial download at power up will not work without it.  Unless you are trying to tell me that there is a difference between the U-Boot boot  image used in serial download and hab_failsafe.  We are booting from USB to write the NAND and this part is successful.

For instance our last U-Boot build output shows this:

Image Type:   Freescale IMX Boot Image
Image Ver:    2 (i.MX53/6/7 compatible)
Mode:         DCD
Data Size:    667744 Bytes = 652.09 KiB = 0.64 MiB
Load Address: 877ff420
Entry Point:  87800000
HAB Blocks:   0x877ff400 0x00000000 0x000a0c00
DCD Blocks:   0x0000002c 0x00910000 0x00000200

We put the following lines in our u-boot.csf file:

    Blocks = 0x877ff400 0x00000000 0x000a0c00 "u-boot.imx", \
             0x00910000 0x0000002c 0x00000200 "u-boot.imx"

[JohnKlug]

And it does not help to start uuu before powering up the device.

[igorpadykov]

Hi John

from team:

----------------

After closing the board, customer need to boot a signed bootloader image instead of unsigned image in that zip file. They need to boot a signed flash.bin firstly and then burn the images and rootfs step by step. They can check some reference script in UUU github page.

----------------

Best regards
igor