Kernel Hangs at Boot With Secure Boot Enabled

[brandensherrell]

I feel as though I am flooding this community with questions regarding Secure Boot, so for that I apologize. At any rate, now that I finally have the device booting in a secure state I have run into yet another issue; the kernel is hanging after loading and fails to boot.

All validation succeeds for U-boot, PPA, bootscript, and the kernel and evidence of this can be seen in the log below. My bootscript simply does the following:

cp.b 0x60A00000 0x81000000 0x3500000

esbc_validate 0x63F40000

bootm 0x81000000

If I flash a RCW with SBEN=0 then I can successfully validate all of these components individually and boot the kernel without issue (successful boot log shown after failed boot log below). It seems that when I am booting with Secure Boot enabled that the kernel is hanging. To provide a bit more information, I the core never releases any other core (i.e. only core0 is running) and is stuck at a yield instruction:

Additionally, we are operating in a secure state at this point per the HPSR register.

Here is the U-boot log for reference. I noticed that PPA seems to issue "PPA validation started .. " quite a few times.

=> cpld reset altbank

U-Boot 2016.012.0+ga9b437f (Jun 03 2016 - 13:05:36 -0400)

SoC:  unknown (0x87920410)

Clock Configuration:

       CPU0(A53):1500 MHz  CPU1(A53):1500 MHz  CPU2(A53):1500 MHz

       CPU3(A53):1500 MHz

       Bus:      400  MHz  DDR:      1600 MT/s  FMAN:     500  MHz

Reset Configuration Word (RCW):

       00000000: 0810000f 0c000000 00000000 00000000

       00000010: 14550002 80004012 e0625000 61002000

       00000020: 00000000 00000000 00000000 00038800

       00000030: 00000000 00001100 00000096 00000001

I2C:   ready

Model: LS1043A RDB Board

Board: LS1043ARDB, boot from vBank 4

CPLD:  V1.4

PCBA:  V3.0

SERDES Reference Clocks:

SD1_CLK1 = 156.25MHZ, SD1_CLK2 = 100.00MHZ

DRAM:  Initializing DDR....

Detected UDIMM Fixed DDR on board

2 GiB (DDR4, 32-bit, CL=11, ECC off)

SEC0: RNG instantiated

PPA validation startedPPA validation Successful

PPA Firmware: Version 0.2

Firmware 'Microcode version 0.0.1 for LS1021a r1.0' for 1021 V1.0

QE: uploading microcode 'Microcode for LS1021a r1.0' version 0.0.1

PPA validation startedPPA validation Successful

Using SERDES1 Protocol: 5205 (0x1455)

Flash: 128 MiB

NAND:  512 MiB

MMC:   FSL_SDHC: 0

Using default environment

EEPROM: NXID v1

PCIe1: disabled

PCIe2: Root Complex no link, regs @ 0x3500000

PCIe3: Root Complex no link, regs @ 0x3600000

In:    serial

Out:   serial

Err:   serial

SCSI:  Error: SCSI Controller(s) 1B4B:9170 not found

Net:   Fman1: Uploading microcode version 106.4.17

FM1@DTSEC1, FM1@DTSEC2, FM1@DTSEC3 [PRIME], FM1@DTSEC4, FM1@DTSEC5, FM1@DTSEC6, FM1@TGEC1

Hit any key to stop autoboot:  0

esbc_validate command successful

## Executing script at 60060000

esbc_validate command successful

## Loading kernel from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'kernel@1' kernel subimage

     Description:  ARM64 Linux kernel

     Type:         Kernel Image

     Compression:  gzip compressed

     Data Start:   0x810000d4

     Data Size:    5384092 Bytes = 5.1 MiB

     Architecture: AArch64

     OS:           Linux

     Load Address: 0x80080000

     Entry Point:  0x80080000

   Verifying Hash Integrity ... OK

## Loading ramdisk from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'ramdisk@1' ramdisk subimage

     Description:  Ramdisk

     Type:         RAMDisk Image

     Compression:  uncompressed

     Data Start:   0x8152914c

     Data Size:    26922046 Bytes = 25.7 MiB

     Architecture: AArch64

     OS:           Linux

     Load Address: unavailable

     Entry Point:  unavailable

   Verifying Hash Integrity ... OK

## Loading fdt from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'fdt@1' fdt subimage

     Description:  Flattened Device Tree blob

     Type:         Flat Device Tree

     Compression:  uncompressed

     Data Start:   0x81522924

     Data Size:    26534 Bytes = 25.9 KiB

     Architecture: AArch64

   Verifying Hash Integrity ... OK

   Loading fdt from 0x81522924 to 0x90000000

   Booting using the fdt blob at 0x90000000

   Uncompressing Kernel Image ... OK

   Loading Ramdisk to ce653000, end cffffc3e ... OK

   Loading Device Tree to 00000000ce639000, end 00000000ce6527a5 ... OK

PPA validation startedPPA validation Successful

PPA validation startedPPA validation Successful

Starting kernel ...

Successful booting of this exact same kernel image  (but with SBEN=0) is shown below. It does appear to be a little different.

=> cpld reset altbank

U-Boot 2016.012.0+ga9b437f (Jun 03 2016 - 13:05:36 -0400)

SoC:  unknown (0x87920410)

Clock Configuration:

       CPU0(A53):1500 MHz  CPU1(A53):1500 MHz  CPU2(A53):1500 MHz

       CPU3(A53):1500 MHz

       Bus:      400  MHz  DDR:      1600 MT/s  FMAN:     500  MHz

Reset Configuration Word (RCW):

       00000000: 0810000f 0c000000 00000000 00000000

       00000010: 14550002 80004012 e0425000 61002000

       00000020: 00000000 00000000 00000000 00038800

       00000030: 00000000 00001101 00000096 00000001

I2C:   ready

Model: LS1043A RDB Board

Board: LS1043ARDB, boot from vBank 4

CPLD:  V1.4

PCBA:  V3.0

SERDES Reference Clocks:

SD1_CLK1 = 156.25MHZ, SD1_CLK2 = 100.00MHZ

DRAM:  Initializing DDR....

Detected UDIMM Fixed DDR on board

2 GiB (DDR4, 32-bit, CL=11, ECC off)

SEC0: RNG instantiated

PPA Firmware: Version 0.2

Firmware 'Microcode version 0.0.1 for LS1021a r1.0' for 1021 V1.0

QE: uploading microcode 'Microcode for LS1021a r1.0' version 0.0.1

Using SERDES1 Protocol: 5205 (0x1455)

Flash: 128 MiB

NAND:  512 MiB

MMC:   FSL_SDHC: 0

Using default environment

EEPROM: NXID v1

PCIe1: disabled

PCIe2: Root Complex no link, regs @ 0x3500000

PCIe3: Root Complex no link, regs @ 0x3600000

In:    serial

Out:   serial

Err:   serial

SCSI:  Error: SCSI Controller(s) 1B4B:9170 not found

Net:   Fman1: Uploading microcode version 106.4.17

FM1@DTSEC1, FM1@DTSEC2, FM1@DTSEC3 [PRIME], FM1@DTSEC4, FM1@DTSEC5, FM1@DTSEC6, FM1@TGEC1

Hit any key to stop autoboot:  0

=> source 0x60060000

## Executing script at 60060000

esbc_validate command successful

## Loading kernel from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'kernel@1' kernel subimage

     Description:  ARM64 Linux kernel

     Type:         Kernel Image

     Compression:  gzip compressed

     Data Start:   0x810000d4

     Data Size:    5384092 Bytes = 5.1 MiB

     Architecture: AArch64

     OS:           Linux

     Load Address: 0x80080000

     Entry Point:  0x80080000

   Verifying Hash Integrity ... OK

## Loading ramdisk from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'ramdisk@1' ramdisk subimage

     Description:  Ramdisk

     Type:         RAMDisk Image

     Compression:  uncompressed

     Data Start:   0x8152914c

     Data Size:    26921871 Bytes = 25.7 MiB

     Architecture: AArch64

     OS:           Linux

     Load Address: unavailable

     Entry Point:  unavailable

   Verifying Hash Integrity ... OK

## Loading fdt from FIT Image at 81000000 ...

   Using 'config@1' configuration

   Trying 'fdt@1' fdt subimage

     Description:  Flattened Device Tree blob

     Type:         Flat Device Tree

     Compression:  uncompressed

     Data Start:   0x81522924

     Data Size:    26534 Bytes = 25.9 KiB

     Architecture: AArch64

   Verifying Hash Integrity ... OK

   Loading fdt from 0x81522924 to 0x90000000

   Booting using the fdt blob at 0x90000000

   Uncompressing Kernel Image ... OK

   Using Device Tree in place at 0000000090000000, end 00000000900197a5

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0

[    0.000000] Initializing cgroup subsys cpu

...

[yipingwang · Accepted solution]

Hello Branden Sherrell,

The cause of the problem is that device tree download to the incorrect address at the end of the Linux Kernel log with secure boot enabled, please refer to to the following lines from the log, in the normal secure boot log there should be no these lines.

Uncompressing Kernel Image ... OK

Loading Ramdisk to ce653000, end cffffc3e ... OK

Loading Device Tree to 00000000ce639000, end 00000000ce6527a5 ... OK

Please try the following boot script.

cp.b 0x60A00000 0x81000000 0x3500000

esbc_validate 0x63F40000

setenv bootargs "console=ttyS0,115200 root=/dev/ram0 earlycon=uart8250,0x21c0500";

setenv fdt_high "0xffffffffffffffff";

setenv initrd_high "0xffffffffffffffff";

bootm $img_addr

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[brandensherrell]

It seems that the Secure Boot u-boot does not use the environment variables? If so, what tells Secure Boot to ignore its environment variable set?

If I flash a RCW with SBEN=0 then I can trap into the u-boot shell and print the default environment variable. It also gives 10 seconds to do so, while the Secure Boot u-boot gives 0 seconds (i.e. no chance to trap into a shell).

[yipingwang]

Hello Branden Sherrell,

In secure boot mode, u-boot cannot be stopped to edit the environment variable, u-boot will execute the boot script to boot Linux Kernel directly.

It's is necessary to add the following bootargs parameters in the boot script. Even through these parameters have already been saved in the u-boot environment space, in secure boot mode, these parameter will not be used.

setenv bootargs "console=ttyS0,115200 root=/dev/ram0 earlycon=uart8250,0x21c0500";

setenv fdt_high "0xffffffffffffffff";

setenv initrd_high "0xffffffffffffffff";

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[brandensherrell]

Hi Yiping, that was it! How could you tell from the logs that this was happening?

I read from uboot documentation that setting fdt_high and initrd_high to 0xffffffffffffffff indicates that they should not be copied to RAM. This makes sense considering the fdt and initrd are contained within the FIT image.

However, both of those environment variables were already defined in my uboot environment. The only difference between your script and mine was that the default environment bootargs was:

bootargs=console=ttyS0,115200 root=/dev/ram0 earlycon=uart8250,mmio,0x21c0500 mtdparts=60000000.nor:1m(nr_bank0_rcw),1m(nor_bank0_uboot),1m(nor_bank0_uboot_env),1m(nor_bank0_fman_uconde),40m(nor_bank0_fit),1mnor_bank4_rcw),1m(nor_bank4_uboot),1m(nor_bank4_uboot_env),1m(nor_bank4_fman_ucode),40m(nor_bank4_fit);7800000.flash:1m(nand_uboot),1m(nand_uboot_env),20m(nand_fit);spi0.0:1m(uboot),5m(kernel),1m(dtb),9m(filesystem)

And you issued a

bootm $img_addr

rather than directly using the load address.

bootm 0x81000000

Would you mind providing a bit of info about why this change was necessary and why the default bootargs string did not work properly?