IMXRT1061 cold reboot failure

[MCW]

Hi NXP team,

I use the customer board with IMXRT1061.

And the QSPI flash is MX25L12833FM2I.

One of my test items is to repeatedly cold reboot the board 100 times.

The interval time between power off and on is 5 seconds, ensuring that there is no electric remaining in the board.

However, there may be 8 to 10 boot-up failure.

Therefore,  I started sorting out the information and conducting some experiments, but the root cause has still not been localized.

More details are listed below, following the sequence from beginning to end:

1. Use the same board and bin file, and just enable the HAB secure boot. The result of 100 cold reboots is a pass.
2. Normally, the the frequency of the QSPI clock is started at 30Mhz, and then increased to 100Mhz because the BootRom reads the information from the FCFB section. At the end, it should be up to 120Mhz since I set it by the api called FLEXSPI_SetFlashConfig. However, the issue is that I cannot get the signal of the 120MHz frequency when a cold boot failure occurs. I also checked the power sequence to ensure the power for the IMXRT1061 is 200ms later than the flash, which means the flash should be in a stable state when accessed by FlexSPI.
3. By controlling the debug LEDs, I can determine which stage the code is executed at. I found FLEXSPI_SetFlashConfig is completed, but there seems to be an issue between setting the FlexSPI to executing the XIP code. At this point, the data from IMXRT1060 is still being sent to the QSPI flash cycle during the cycle with 100Mhz. To be honest, I do not know the reason for having this kind of signal after the cold reboot failure.
4. I also changed the FCFB content from 100Mhz to 120 MHz and set it to 8 dummy cycles in order to reduce the frequency switching. The fail rate decline at 1%.
5. My project has three types of boards. One of them has a higher probability of reboot failure.

According to this, is there any method or experiment I can do?

If any statement is unclear, please let me know.

Thanks a lot.

Best regards,

Doris

[Sam_Gao]

Hi @MCW 

' I found FLEXSPI_SetFlashConfig is completed, but there seems to be an issue between setting the FlexSPI to executing the XIP code.'

--> Would you please help check with more details? it's better to confirm this issue is from FlexSPI or NOR Flash, I suppose that the 100MHz/120MHz setting should be done from MCU FlexSPI side.

Btw, how about the falure rate for the three types of boards(Same HW/SW?)?

B.R,

Sam

[MCW]

Hi Sam,

Thanks for your relay.

1. it's better to confirm this issue is from FlexSPI or NOR Flash, I suppose that the 100MHz/120MHz setting should be done from MCU FlexSPI side.  -> Actually, I have no idea how to tell the problem is from which side. Do you have any recommendation I can try? 
2. how about the fail rate for the three types of boards(Same HW/SW?)? -> Basically, the hardware design for all three types of boards is the same, including the power supply and the routing between the MCU and the QSPI flash. And I used the same bin file for them. But so sorry, I didn't record the figures. I just noticed one of them has more pieces which has reboot fail issue. After that, I focused on the experiment with the single board. The fail rate was reduced from 11% to 0.4% when I changed the FCFB flash clock frequency at 120Mhz.

Best regards,

Doris

[Sam_Gao]

@MCW 

'the issue is that I cannot get the signal of the 120MHz frequency when a cold boot failure occurs'. ’ At this point, the data from IMXRT1060 is still being sent to the QSPI flash cycle during the cycle with 100Mhz. To be honest, I do not know the reason for having this kind of signal after the cold reboot failure.‘ 'The fail rate was reduced from 11% to 0.4% when I changed the FCFB flash clock frequency at 120Mhz.'

 -->

I'm worried that MCU doesn't actually work at 120MHz, it's better to confirm it.

BTW, how about the power voltage when issue happened both for MCU and Flash?

It's hard to say where is the root cause, but it seems to be related to the timing.

[MCW]

Hi @Sam_Gao ,

Sorry for the late reply.

I used the logic analyzer to confirm the frequency, which was 120 MHz.

Recently, I changed the optimization level from 'optimize for size' to 'no optimization' during compilation.

All kinds of boards booted up successfully over 1000 times.

However, the size of bin file is much larger than my flash layout.

Is there any method to apply no optimization for certain files?

Best regards,

Doris

[Sam_Gao]

It is better to find which step is wrong if you provide more information to me @MCW 

1. If the flash is not enough, it is better to use 'optimize for size'', would you please share which kind of optimization to me? '-O0', '-Og' or '-O3'.

2. Please give me which SDK version are you using

3. By the way, would you please check the power timing during booting, especially for the power side? What is the capacitance of NVCC_PLL from your side? it is better to recommended to increase it slightly if these testing is at a lower tempeature which will changes the capacitance value.

4. Please follow the blow screenshot which from QSPI Flash MX25L12833FM2I you mentioned to try.

[MCW]

Hi Sam,

1. If the flash is not enough, it is better to use 'optimize for size'', would you please share which kind of optimization to me? '-O0', '-Og' or '-O3'. -> Normally, -Os. And I have tried to use -O0 for "no optimization".

2. Please give me which SDK version are you using -> In fact, we developed our project based on Zephyr 2.7.1.

3. What is the capacitance of NVCC_PLL from your side? -> I have checked it, there are the same, 0.22uF/25V and 4.7uF/10V, respectively.

Thanks for your help.

Best regards,
Doris

[MCW]

Hi  @Sam_Gao ,

I have conducted further experiments and would like to clarify some points.

Apologies for the earlier misinformation.

Firstly, the flash clock frequency did not increase from 100MHz to 120MHz.

Secondly, even with the "compile no optimization" setting, after modifying the code and rebuilding, the cold reboot failure still persists.

As part of my debugging process, I used specific LED indicators after the flash initialization and found that the program is stuck at the point where it attempts to print the first log. After reviewing the UART-related code, I discovered that it is getting stuck in the following while loop, as the TDRE register remains at 0.

Could you please advise what could cause this behavior, where the TDRE register is consistently 0?

Thank you.

Best regards,

Doris

[Sam_Gao]

Hi,

From my side, I searched with the key words 'zephyr, 2.7.1, mimxrt1060' your mentioned, https://github.com/zephyrproject-rtos/zephyr/blob/zephyr-v2.7.1/boards/arm/mimxrt1060_evk/mimxrt1060... 

- For LPUART_GetStatusFlags(config->base) & LPUART_STAT_TDRE_MASK, please see below comments.

https://github.com/zephyrproject-rtos/zephyr/blob/2eb8cfb7f5084d040725dc0f4c61b526fa40b8f0/drivers/s...

/*!
 * @brief Gets LPUART status flags.
 *
 * This function gets all LPUART status flags. The flags are returned as the logical
 * OR value of the enumerators @ref _lpuart_flags. To check for a specific status,
 * compare the return value with enumerators in the @ref _lpuart_flags.
 * For example, to check whether the TX is empty:
 * @code
 *     if (kLPUART_TxDataRegEmptyFlag & LPUART_GetStatusFlags(LPUART1))
 *     {
 *         ...
 *     }
 * @endcode
 *
 * @Param base LPUART peripheral base address.
 * @return LPUART status flags which are ORed by the enumerators in the _lpuart_flags.
 */
uint32_t LPUART_GetStatusFlags(LPUART_Type *base);

And the definition of LPUART_STAT_TRDE as below:

#define LPUART_STAT_TC_MASK                      (0x400000U)
#define LPUART_STAT_TC_SHIFT                     (22U)
/*! TC - Transmission Complete Flag
 *  0b0..Transmitter active (sending data, a preamble, or a break).
 *  0b1..Transmitter idle (transmission activity complete).
 */
#define LPUART_STAT_TC(x)                        (((uint32_t)(((uint32_t)(x)) << LPUART_STAT_TC_SHIFT)) & LPUART_STAT_TC_MASK)

#define LPUART_STAT_TDRE_MASK                    (0x800000U)
#define LPUART_STAT_TDRE_SHIFT                   (23U)
/*! TDRE - Transmit Data Register Empty Flag
 *  0b0..Transmit FIFO level is greater than watermark.
 *  0b1..Transmit FIFO level is equal or less than watermark.
 */
#define LPUART_STAT_TDRE(x)                      (((uint32_t)(((uint32_t)(x)) << LPUART_STAT_TDRE_SHIFT)) & LPUART_STAT_TDRE_MASK)

[MCW]

Hi @Sam_Gao ,

I apologize for not describing the issue clearly earlier. I would like to clarify the situation regarding the behavior of the UART module's TDRE flag after initialization.

After boot-up and initializing the UART module, the TDRE flag fails to be set to 1 successfully 5-6 times out of 100. I am trying to understand why this happens.

Could you please provide any insights or suggestions on what might be causing this issue?

Thank you for your help.

Best regards,

Doris

[Sam_Gao]

Hi,

I am confused again why boot-up problem is related to UART according to your point of view, it seems it boot from XIP(FlexSPI Nor Flash) not serial download.

TC - Transmission Complete Flag, TDRE - Transmit Data Register Empty Flag.

it is needed to check why TDRE is happened from application.

B.R,

Sam

[MCW]

Hi @Sam_Gao ,

Yes, my custom boards are boot from the nor flash. Initially, I suspected that the boot failure issue was due to incorrect flash-related settings. However, after further clarification, I found that during boot failure, the program actually gets stuck in the mcux_lpuart_poll_out function's while loop. The failure always occurs when trying to print the first line of the debug log. This leads me to wonder about the cause of TDRE being 0 after UART initialization, since the reset value of TDRE should be 1.

Additionally, I’ve been experimenting with the SDK sample code and found that disabling the clock gate for UART1-UART8 before setting the UART source clock can prevent the issue. Could you please explain why disabling the clock gate could resolves the issue?

As I mentioned earlier, I also tried introducing a 100 NOP delay before setting the source clocks for all modules, but this solution does not work consistently across all boards.

If any part of the description is unclear, please feel free to let me know. Thank you for your help.

Best regards,

Doris

[Sam_Gao]

@MCW 

' disabling the clock gate for UART1-UART8 before setting the UART source clock can prevent the issue.'

I am able to give comments, I try to understand if you can help give the source code to describe in details. 

About this boot-fail case, I am still confused if the root casue is from print/UART which seems be 100% issue as your mentioned.

[MCW]

Hi @Sam_Gao ,

I wanted to share an update.

I added a delay time (100 NOPs) before setting the source clock for the UART.

It seems to have fixed the issue of the system getting stuck in the while loop of mcux_lpuart_pool_out during boot-up.

Attached is the code for my clock initialization process.

Thank you.

/**

 *

 * @brief Initialize the system clock

 *

 * @return N/A

 *

 */

static ALWAYS_INLINE void clock_init(void)

{

    /* Boot ROM did initialize the XTAL, here we only sets external XTAL

     * OSC freq

     */

    CLOCK_SetXtalFreq(24000000U);

    CLOCK_SetRtcXtalFreq(32768U);

    /* Set PERIPH_CLK2 MUX to OSC */

    CLOCK_SetMux(kCLOCK_PeriphClk2Mux, 0x1);

    /* Set PERIPH_CLK MUX to PERIPH_CLK2 */

    CLOCK_SetMux(kCLOCK_PeriphMux, 0x1);

    /* Setting the VDD_SOC to 1.5V. It is necessary to config AHB to 600Mhz

     */

    DCDC->REG3 = (DCDC->REG3 & (~DCDC_REG3_TRG_MASK)) | DCDC_REG3_TRG(0x12);

    /* Waiting for DCDC_STS_DC_OK bit is asserted */

    while (DCDC_REG0_STS_DC_OK_MASK !=

            (DCDC_REG0_STS_DC_OK_MASK & DCDC->REG0)) {

        ;

    }

#ifdef CONFIG_INIT_ARM_PLL

    CLOCK_InitArmPll(&armPllConfig); /* Configure ARM PLL to 1200M */

#endif

#ifdef CONFIG_INIT_SYS_PLL

    CLOCK_InitSysPll(&sysPllConfig); /* Configure SYS PLL to 528M */

#endif

#ifdef CONFIG_INIT_USB1_PLL

    CLOCK_InitUsb1Pll(&usb1PllConfig); /* Configure USB1 PLL to 480M */

#endif

#ifdef CONFIG_INIT_ENET_PLL

    CLOCK_InitEnetPll(&ethPllConfig);

#endif

#ifdef CONFIG_INIT_VIDEO_PLL

    CLOCK_InitVideoPll(&videoPllConfig);

#endif

#ifdef CONFIG_HAS_ARM_DIV

    CLOCK_SetDiv(kCLOCK_ArmDiv, CONFIG_ARM_DIV); /* Set ARM PODF */

#endif

    CLOCK_SetDiv(kCLOCK_AhbDiv, CONFIG_AHB_DIV); /* Set AHB PODF */

    CLOCK_SetDiv(kCLOCK_IpgDiv, CONFIG_IPG_DIV); /* Set IPG PODF */

    /* Set PRE_PERIPH_CLK to PLL1, 1200M */

    CLOCK_SetMux(kCLOCK_PrePeriphMux, 0x3);

    /* Set PERIPH_CLK MUX to PRE_PERIPH_CLK */

    CLOCK_SetMux(kCLOCK_PeriphMux, 0x0);

    for(int i = 0; i < 100; i++) { //test

        __NOP();

    }

#ifdef CONFIG_UART_MCUX_LPUART

    /* Configure UART divider to default */

    CLOCK_SetMux(kCLOCK_UartMux, 0); /* Set UART source to PLL3 80M */

    CLOCK_SetDiv(kCLOCK_UartDiv, 0); /* Set UART divider to 1 */

#endif

#ifdef CONFIG_I2C_MCUX_LPI2C

    CLOCK_SetMux(kCLOCK_Lpi2cMux, 0); /* Set I2C source as USB1 PLL 480M */

    CLOCK_SetDiv(kCLOCK_Lpi2cDiv, 5); /* Set I2C divider to 6 */

#endif

#ifdef CONFIG_SPI_MCUX_LPSPI

    CLOCK_SetMux(kCLOCK_LpspiMux, 1); /* Set SPI source to USB1 PFD0 720M */

    CLOCK_SetDiv(kCLOCK_LpspiDiv, 7); /* Set SPI divider to 8 */

#endif

#ifdef CONFIG_DISPLAY_MCUX_ELCDIF

    CLOCK_SetMux(kCLOCK_LcdifPreMux, 2);

    CLOCK_SetDiv(kCLOCK_LcdifPreDiv, 4);

    CLOCK_SetDiv(kCLOCK_LcdifDiv, 1);

#endif

#if CONFIG_USB_DC_NXP_EHCI

    CLOCK_EnableUsbhs0PhyPllClock(kCLOCK_Usb480M,

        DT_PROP_BY_PHANDLE(DT_INST(0, nxp_mcux_usbd), clocks, clock_frequency));

    CLOCK_EnableUsbhs0Clock(kCLOCK_Usb480M,

        DT_PROP_BY_PHANDLE(DT_INST(0, nxp_mcux_usbd), clocks, clock_frequency));

    USB_EhciPhyInit(kUSB_ControllerEhci0, CPU_XTAL_CLK_HZ, &usbPhyConfig);

    CLOCK_EnableUsbhs1PhyPllClock(kCLOCK_Usb480M,

        DT_PROP_BY_PHANDLE(DT_INST(0, nxp_mcux_usbd), clocks, clock_frequency));

    CLOCK_EnableUsbhs1Clock(kCLOCK_Usb480M,

        DT_PROP_BY_PHANDLE(DT_INST(0, nxp_mcux_usbd), clocks, clock_frequency));

    USB_EhciPhyInit(kUSB_ControllerEhci0, CPU_XTAL_CLK_HZ, &usbPhyConfig);

#endif

#if DT_NODE_HAS_STATUS(DT_NODELABEL(usdhc1), okay) && CONFIG_DISK_DRIVER_SDMMC

    /* Configure USDHC clock source and divider */

    CLOCK_InitSysPfd(kCLOCK_Pfd0, 0x12U);

    CLOCK_SetDiv(kCLOCK_Usdhc1Div, 0U);

    CLOCK_SetMux(kCLOCK_Usdhc1Mux, 1U);

    CLOCK_EnableClock(kCLOCK_Usdhc1);

#endif

#if DT_NODE_HAS_STATUS(DT_NODELABEL(usdhc2), okay) && CONFIG_DISK_DRIVER_SDMMC

    /* Configure USDHC clock source and divider */

    CLOCK_InitSysPfd(kCLOCK_Pfd0, 0x12U);

    CLOCK_SetDiv(kCLOCK_Usdhc2Div, 0U);

    CLOCK_SetMux(kCLOCK_Usdhc2Mux, 1U);

    CLOCK_EnableClock(kCLOCK_Usdhc2);

#endif

#ifdef CONFIG_VIDEO_MCUX_CSI

    CLOCK_EnableClock(kCLOCK_Csi); /* Disable CSI clock gate */

    CLOCK_SetDiv(kCLOCK_CsiDiv, 0); /* Set CSI divider to 1 */

    CLOCK_SetMux(kCLOCK_CsiMux, 0); /* Set CSI source to OSC 24M */

#endif

#ifdef CONFIG_CAN_MCUX_FLEXCAN

    CLOCK_SetDiv(kCLOCK_CanDiv, 1); /* Set CAN_CLK_PODF. */

    CLOCK_SetMux(kCLOCK_CanMux, 2); /* Set Can clock source. */

#endif

#if !(defined(CONFIG_CODE_FLEXSPI) || defined(CONFIG_CODE_FLEXSPI2)) && \

    defined(CONFIG_MEMC_MCUX_FLEXSPI) && \

    DT_NODE_HAS_STATUS(DT_NODELABEL(flexspi), okay)

    CLOCK_DisableClock(kCLOCK_FlexSpi);

    CLOCK_InitUsb1Pfd(kCLOCK_Pfd0, 24);

    CLOCK_SetMux(kCLOCK_FlexspiMux, 3);

    CLOCK_SetDiv(kCLOCK_FlexspiDiv, 2);

#endif

    /* Keep the system clock running so SYSTICK can wake up the system from

     * wfi.

     */

    CLOCK_SetMode(kCLOCK_ModeRun);

}

Best regards,

Doris

[Sam_Gao]

Great to hear that!