A53 core suspend

[Lakshmi_AG]

Hi,

I am using imxqm with only A53 cores enabled.

Presently trying to self suspend A53 cores(0,1,2,3) using below mentioned scu API but i see cores are not suspending and DDR also not in refresh mode.

/*************************************sleep sequence****************************************************/

sc_irq_enable(ipc_handle, 297U, SC_IRQ_GROUP_WAKE, SC_IRQ_PAD, SC_TRUE);
    /* Enable GPIO pad wakeup */
    sc_pad_set_wakeup(ipc_handle, 40U, 6U);//SC_PAD_WAKEUP_RISE_EDGE 6U

    /* make sure system sources power ON in low power mode by default */
    sc_pm_req_low_power_mode(ipc_handle, SC_R_A53_0, SC_PM_PW_MODE_ON);
    /* Config system interface HPM, LPM */
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_DDR, SC_PM_PW_MODE_ON, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_OCMEM, SC_PM_PW_MODE_ON, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_MU, SC_PM_PW_MODE_OFF, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_INTERCONNECT, SC_PM_PW_MODE_ON,
        SC_PM_PW_MODE_OFF);
    sc_pm_set_cpu_resume(ipc_handle, SC_R_A53_0,true, 0x80000000);//
    sc_pm_req_cpu_low_power_mode(ipc_handle, 1U,SC_PM_PW_MODE_OFF, SC_PM_WAKE_SRC_SCU);

__asm__("dsb sy");
    __asm__("wfi");

while(1);

/************************************************************************************************/

/***************************wake-up sequence******************************************/

/* Query SCU wakeup event status */
#if 1
    uInt32 status;
    sc_irq_status(ipc_handle, 297U, SC_IRQ_GROUP_WAKE, &status);
    /* Check for pad wakeup */
    if (status & SC_IRQ_PAD)
    {
        /* Check for GPIO pad wakeup */
        /* Note: SCFW updates pending pad wakeup config to SC_PAD_WAKEUP_OFF */
        uInt8 gpio_wakeup;
        //sc_pad_get_wakeup(ipc_handle, 40U, &gpio_wakeup);
        if (gpio_wakeup == SC_PAD_WAKEUP_OFF)
        {
            /* GPIO pad generated wake event */
            print(DEBUG_PRINT_ALWAYS, "GPIO wakeup\r\n");

        } }

Is the suspend sequnce all good, because if i add any print statemnt inside while(1), is it still executing that mean A53 are still alive.

Is there any other steps do we need to follow before suspending, or is there any method to check if cores are suspend mode.    

[qiang_li-mpu_se]

For iMX8QM, when suspend the A53 core, the SCFW will turn off the A53 core power.

I think you need reference to the ATF "arm-trusted-firmware" code, function imx_domain_suspend() of file "arm-trusted-firmware\plat\imx\imx8qm\imx8qm_psci.c". Before suspend, ATF will set the resume address to SCFW, it is the ATF base address in DDR, then after suspend, SCFW will track the GPIO PAD wakeup event, after get the wake up signal, SCFW will turn on the A53 power, ATF will start to run again, ATF will check, it is cold boot or resume boot, for resume it will call imx_domain_suspend_finish() "in imx8qm_psci.c" to do the resume tasks.

The ATF code can be found at https://github.com/nxp-imx/imx-atf

ATF suspend main sequence: imx_domain_suspend() -> imx_pwr_domain_pwr_down_wfi(), they are called from function psci_cpu_suspend_start() of file "arm-trusted-firmware\lib\psci\psci_suspend.c"

[Lakshmi_AG]

Thanks for the quick response.

I tried the sequence as mentioned in the ATF build, still core suspend and resume is not working.

is there any pre condition we need to have before sending commands to SCU for suspend

[qiang_li-mpu_se]

Hi @Lakshmi_AG ,

For quickly method, I think you can use the ATF directly, then put your own image to address 0x80020000 like the Uboot. The mkimage command can be "./mkimage_imx8 -soc QM -rev B0 -append mx8qmb0-ahab-container.img -c -scfw scfw_tcm.bin -ap bl31.bin a53 0x80000000 your_image.bin --data 0x80020000 -out flash.bin"

After ATF successfully booted, it will jump to address 0x80020000 to run your_image.bin.

Then in your own image, you can call ATF's psci APIs with smccc method. Uboot example can be followed:

#define PSCI_SYSTEM_SUSPEND_AARCH64 0xc400000e

int do_board_suspend(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])

{

    struct udevice *dev;

    uclass_get_device_by_name(UCLASS_FIRMWARE, "psci", &dev);

    invoke_psci_fn(PSCI_SYSTEM_SUSPEND_AARCH64, 0, 0, 0);

    return 1;

}

U_BOOT_CMD(

    suspend, 1, 1, do_board_suspend,

    "suspend\n",

    "system board suspend for i.MX 8 Quad devices \n"

);

If you can't use ATF directly, the you need porting the ATF code, it needs time to debug in your side. For such kind of debug, you can build scfw with "M=1" to enable the SCFW debug UART, then after run the suspend in your software, you can run command "power.r" in SCFW debug UART to check which module hasn't been turned off, then you need check the related code. 

[Lakshmi_AG]

Hi @qiang_li-mpu_se ,

I have started integrating the code to talk to atf,

 invoke_psci_fn(PSCI_SYSTEM_SUSPEND_AARCH64, 0, 0, 0);

defined the invoke_psci_fn as below.. but got a doubt how does it gets invoked with the bl31.bin 

is there any linker updates we need to do, or below code will work fine for me  

****************************************************************************

typedef unsigned long (psci_fn)(unsigned long, unsigned long,
    unsigned long, unsigned long);
static psci_fn *invoke_psci_fn;

#define SMCCC_VERSION 0x80000000
void suspend_using_atf(void)
{

    struct udevice *dev;
    invoke_psci_fn(SMCCC_VERSION , 0, 0, 0);
    

   return 1;
}

*********************************************************************************************

[qiang_li-mpu_se]

After ATF (bl31.bin) run, it will register service and handler for smccc. Then when kernel, Uboot or other software calls invoke_psci_fn(), it will goto arm_smccc_smc() as followed, the ATF handler and service will get the function call. No linker updates are needed.

ENTRY(__arm_smccc_smc)

    SMCCC smc

ENDPROC(__arm_smccc_smc)

[qiang_li-mpu_se]

Hi @Lakshmi_AG 

If your code is 32 bits, I think it can't work, because the ATF is 64 bits.

In Uboot, some files can be referenced for psci:

uboot-imx\arch\arm\cpu\armv8\psci.S

uboot-imx\arch\arm\cpu\armv8\smccc-call.S

uboot-imx\arch\arm\cpu\armv8\cpu.c

uboot-imx\drivers\firmware\psci.c

The smccc, psci method can be found from ARM website:

https://developer.arm.com/Architectures/Secure%20Monitor%20Calling%20Convention

https://developer.arm.com/documentation/den0028/latest/

From my understand, ATF has the handler to process smccc function calls. 

[Lakshmi_AG]

Hi @qiang_li-mpu_se 

If we are in 32bit mode, and if we cannot communicate with ATF(since atf running in 64 bit) with smccc method, is there any other way to do power management. 

or any other mode to talk to ATF from 32 bit system

Regards

Lakshmi

[qiang_li-mpu_se]

For your 32bits mode, I think you can only merge the ATF code into your own code. And before call SCFW to turn off the A core and system power, you must turn off all peripherals. Any pending interrupt will block the suspend. 

And when you fail to suspend the system, you can run command "power.r" from SCU debug UART to check which module blocks it, then modify your software to close the module.

[Lakshmi_AG]

Hi @qiang_li-mpu_se 

As per our current design, we are not planning to put A53 cores in shut down mode as on wake up it will take more time to return
due to reinitialization for cores and other peripherals. Hence we decided to put A53 cores in low power state by executing
WFI/WFE mode. If we put A53 cores in low power state, can SCU be able to wake up by sending any interrupt or wake event?

And does this too need ATF to be integrated into our build?

[qiang_li-mpu_se]

I means suspend to RAM, in this mode, DDR is in self refresh and ARM core power is turned off. SCFW recorded the A core resume address. After received the wakup event, such as GPIO, RTC, ON/OFF button, the A core can resume and continue to run. It is not WFI. 

For WFI, it is used for CPU IDLE, if you just need CPU IDLE, then no ATF and SCFW APIs are needed for your case.

There are big power comsunption difference between CPU IDLE and SUSPEND.

[Lakshmi_AG]

Hi  qiang_li-mpu_se

Thanks for the response.

Presently we are in 32bit mode,  can i get macro for 32bit,

and also Do we have a example or a document to talk to ATF.

or Uboot code path for reference i can use.

Regards

Lakshmi

[AldoG]

Hello,

Your sequence looks correct, just that you didn't set the A53 Core into low power mode, you may take the following sample code as reference.
https://github.com/nxp-imx/imx-atf/blob/lf_v2.6/plat/imx/imx8qm/imx8qm_psci.c#L195

So you're missing this one:
sc_pm_req_low_power_mode(ipc_handle, SC_R_A53, SC_PM_PW_MODE_OFF);

All other seems correct, please try it and let us know.

Best regards,
Aldo.

[Lakshmi_AG]

Hi Aldo,

I tried by adding the API but still the issue is same.

below is the updated sequence.

//suspend//

/* Register for PAD wakeup */
    sc_irq_enable(ipc_handle, 297U, SC_IRQ_GROUP_WAKE, SC_IRQ_PAD, SC_TRUE);
    /* Enable GPIO pad wakeup */
    sc_pad_set_wakeup(ipc_handle, 40U, 6U);//SC_PAD_WAKEUP_RISE_EDGE 6U

    /* make sure system sources power ON in low power mode by default */
    sc_pm_req_low_power_mode(ipc_handle, SC_R_A53, SC_PM_PW_MODE_OFF);
    /* Config system interface HPM, LPM */
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_DDR, SC_PM_PW_MODE_ON, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_OCMEM, SC_PM_PW_MODE_ON, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_MU, SC_PM_PW_MODE_OFF, SC_PM_PW_MODE_OFF);
    sc_pm_req_sys_if_power_mode(ipc_handle, SC_R_A53, SC_PM_SYS_IF_INTERCONNECT, SC_PM_PW_MODE_ON,
        SC_PM_PW_MODE_OFF);
    sc_pm_set_cpu_resume(ipc_handle, SC_R_A53_0,true, 0x80000000);//
    sc_pm_req_cpu_low_power_mode(ipc_handle, SC_R_A53_0,SC_PM_PW_MODE_OFF, SC_PM_WAKE_SRC_SCU);
    sc_pm_req_cpu_low_power_mode(ipc_handle, SC_R_A53_1, SC_PM_PW_MODE_OFF, SC_PM_WAKE_SRC_SCU);
    sc_pm_req_cpu_low_power_mode(ipc_handle, SC_R_A53_2, SC_PM_PW_MODE_OFF, SC_PM_WAKE_SRC_SCU);
    sc_pm_req_cpu_low_power_mode(ipc_handle, SC_R_A53_3, SC_PM_PW_MODE_OFF, SC_PM_WAKE_SRC_SCU);

__asm__("dsb");
    __asm__("isb");
    __asm__("wfi");
    //__asm__("wfi");
    while (1)
    {
        print(DEBUG_PRINT_ALWAYS, ".\r\n");
    } 

[AldoG]

Hello,

I see no problems with the sequence you have shared it looks practically the same with the sample code we have available I have shared previously.

Also, you may be able to enable Debug Monitor on the SCFW, so you may check the state of the resources while they are in LPM, just for debugging purpose.

Best regards,
Aldo.

[Lakshmi_AG]

Do we need to flush cache before executing wfi or do we any errata for A53 on why WFI does not work.

and also for enabling Debug Monitor, i went through the document and it specifies "The debug monitor
allows command-line interaction via the SCU UART" : which are hardware pins  we need connect in soc to for SCU UART.

[AldoG]

Hi,

As stated in the reference manual for the i.MX8QM, chapter 11.1.1.5 Interface:
Assuming that a simple standby state with no state loss is required, there is no preparation required, and the core may simply perform the WFI operation. The caches are kept coherent in this state, and resumption is caused by an interrupt to the core.

If transition is required to lower power states for the core, then before the WFI is signaled, the core must ensure that any architectural state is saved to memory, cache is disabled, flushed, and then removed from the coherency domain. The SCU must also have been told that the intent is to transition to this lower power state.

Also, regarding debug monitor, if you are using our MEK board, it has two serial ports, the first one (usually ttyUSB0) is used by the A cores (u-boot/Linux in this case), the second one is used by one of the M4 cores or the SCFW when debug monitor is enabled.

Best regards,
Aldo.

[Lakshmi_AG]

Hi,

Thanks for the confirmation,

To further understand why cores are not suspending after WFI, arm for a53 suggests as below

/*********************************/

On entry into WFI low-power state, STANDBYWFI for that core is asserted. Assertion of STANDBYWFI guarantees that the core is in idle and low-power state.

/***************************/

On imx8qm how to check the STANDBYWFI signal. is there any register available.

-Lakshmi

[AldoG]

Hello,

Sorry for the delayed response, please note that it is the SCFW which will handle the system power states, the SCFW also monitors the state of the system to enter the lowest possible power state.  SCFW uses the following to evaluate the system power state :

Executing environments (EEs) manage individual resources using SCFW APIs for resources (sc_pm_set_resource_power_mode).  After all resources within a subsystem are powered down, SCFW will automatically power down the entire subsystem. 

EEs manage shared system resources such as DDR, on-chip memory, and system interconnect using SCFW APIs for system interfaces (sc_pm_req_sys_if_power_mode).  EEs specify power requirements for these shared resources during high-power mode (HPM) and low-power mode (LPM).  If any EE is in HPM, SCFW will keep the system in HPM and set the system interfaces to the highest HPM power mode among all EEs currently in HPM.  If all EEs are in LPM, the SCFW will transition the system to LPM and set the system interfaces to the highest LPM power mode for all EEs.

Cortex-A EEs manage their LPM configuration using SCFW APIs (sc_pm_req_cpu_low_power_mode).  This API is also used to signal the SCFW to monitor individual CPU WFI signals and apply the LPM configuration once WFI has been reached.

There is the call sc_pm_get_resource_power_mode() to retrieve a resource current power mode, this for debugging purpose just to be sure the resource current state.

So I would like to know how are you testing going into low power mode?
Did you try by enabling debug monitor and check that it does not go into low power?

Best regards,
Aldo.

[Lakshmi_AG]

Hi Aldo,

Thanks for the explanation.

SCU API are all good.

what we suspect is we are not able to put cores in WFI state. We feel that cores are getting continuous interrupts in our system.

Can we identify which interrupt been served to cores? Do we have such provision?

[AldoG]

Hello,

I'll double check with internal team if we have something available.
Could you share how are you testing this?
Maybe sharing some rough steps to try to replicate.

Thank you,
Best regards,
Aldo.