Facing kernel hang issue at boot up in imx6ull based custom board.

[meetalipatel]

Hello NXP Team,

I am working on a custom board based on i.MX6ULL processor. We have a yocto image working fine with Linux 4.9.88 for this processor and now, we are working on updating the kernel to 6.1.22. 

With the new kernel version, I am facing an issue where the kernel hangs just sometime after the login prompt comes up. Following is the cpu0 device used in my custom dts file.

&cpu0 {
    operating-points = <
        /* kHz uV */
        528000 1175000
        396000 1175000
        198000 1175000
    >;
    fsl,soc-operating-points = <
        /* KHz uV */
        528000 1175000
        396000 1175000
        198000 1175000
    >;
    fsl,arm-soc-shared = <1>;
};

With some trial and error on the values of operating-points for cpu0 I found that when I configure the value to only 528000kHZ, it works fine and does not hang during boot up. But with any other frequency,(single value or multiple values) the kernel hangs. 

I am using the same dts file entry for Linux 4.9.88 where everything works fine. I don't understand why the same entry does not work for the newer kernel. Is there any difference related to CPU config and/or characteristics in these kernel versions?

Awaiting your response

Thanks & Regards

Meetali Patel

[meetalipatel]

Hi Jimmy, 

I enabled some debug prints in kernel and I found some new prints here.

[13036.803816] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13036.824055] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13037.363855] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13037.384036] cpu cpu0: 198 MHz, 1175 mV --> 528 MHz, 1175 mV
[13037.403941] cpu cpu0: 528 MHz, 1175 mV --> 396 MHz, 1175 mV
[13037.873851] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13037.894052] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13038.323965] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13038.344050] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13038.893840] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13038.914050] cpu cpu0: 198 MHz, 1175 mV --> 528 MHz, 1175 mV
[13038.933943] cpu cpu0: 528 MHz, 1175 mV --> 396 MHz, 1175 mV
[13039.403858] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13039.424056] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13039.913858] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13039.934041] cpu cpu0: 198 MHz, 1175 mV --> 528 MHz, 1175 mV
[13039.953939] cpu cpu0: 528 MHz, 1175 mV --> 396 MHz, 1175 mV
[13040.403938] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13040.424083] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13040.933841] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13040.954035] cpu cpu0: 198 MHz, 1175 mV --> 528 MHz, 1175 mV
[13040.973949] cpu cpu0: 528 MHz, 1175 mV --> 396 MHz, 1175 mV
[13041.443850] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13041.464058] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13041.843826] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13041.864050] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13042.281666] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13042.304048] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13042.323995] cpu cpu0: 396 MHz, 1175 mV --> 528 MHz, 1175 mV
[13042.343859] cpu cpu0: 528 MHz, 1175 mV --> 396 MHz, 1175 mV
[13042.483890] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13042.504053] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13042.973849] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13042.994056] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13043.035693] cpu cpu0: 396 MHz, 1175 mV --> 198 MHz, 1175 mV
[13046.084164]
[13046.084164] Bus freq set to 24000000 start...
[13046.090550] Bus freq set to 24000000 done!
[13046.109468] cpu cpu0: 198 MHz, 1175 mV --> 396 MHz, 1175 mV
[13046.151701]
[13046.151701] Bus freq set to 400000000 start...

It looks like when it switches the bus freq to 400000000, the CPU hangs. 

Can you please guide here further?

[meetalipatel]

Hi Jimmy,

Awaiting your response.

Here I am sharing a quick summary of the issue and debugging experiments that we have tried.

1. We are setting 3 frequencies and operating points in the cpu node of the dts file as follows:
   
   &cpu0 {
        operating-points = <
                   /* kHz  uV */
                   528000  1175000
                   396000  1175000
                   198000  1175000
           >;
           fsl,soc-operating-points = <
                   /* KHz  uV */
                   528000  1175000
                   396000  1175000
                   198000  1175000
           >;
   fsl,arm-soc-shared = <1>;
   };
   
   If we comment out 396 MHZ and 198 MHz frequencies and keep only 528 KHz, the CPU DOES NOT hang. But if we keep anyone from 396 and 198, the CPU hangs. If we use any 2 combinations of frequencies from these 3 frequencies, the CPU hangs. (We cannot let the CPU work on a single frequency as a solution, as we have a requirement of switching the frequencies as per CPU load.)
2. We tried disabling the kernel configs CONFIG_CPUFREQ_DT and CONFIG_ARM_CPUIDLE in the kernel, but still, the CPU hangs.
3. We are using the "Ondemand" governor as default. If we switch the governor to "Userspace" or "Performance", the CPU DOES NOT hang. With other governors, Conservative, Schedutil, and Powersave, the CPU hangs. (We are not sure about the consequences of this solution in the field. Also, we have a requirement for frequency switching with respect to the CPU load.)
4. If we define an LCD node in our DTS file (same as in the DTS file of EVK), the CPU DOES NOT hang. (But this is not a solution for us, as we are using some LCD pins as GPIOs in our custom application. Kindly note here that, we do not have any display device in our product.)
5. If we run another systemd service that turns an LED on and off at every 1 sec interval, the CPU DOES NOT hang.

Regards
Meetali Patel

[meetalipatel]

Hi Jimmy,

Any updates on this issue? I would like to know more details on why when using the ondemand governor, the device hangs but when using the userspace, it works fine.

Thanks & Regards

Meetali Patel

[meetalipatel]

Hi Jimmy,

I would like to use multiple frequencies for the CPU. Even if I change the voltages, the problem persists. I tried the following values that are same as of EVK.

&cpu0 {
    operating-points = <
        /* kHz uV */
        528000 1175000
        396000 1025000
        198000 950000
    >;
    fsl,soc-operating-points = <
        /* KHz uV */
        528000 1175000
        396000 1175000
        198000 1175000
    >;

    fsl,arm-soc-shared = <1>;
};

[jimmychan]

Hello,

Is there any error message? could you send me the log?

Best regards,

Jimmy

[meetalipatel]

There are no error logs. Attached are the logs for your reference.

[jimmychan]

So, where it stopped (hang)?

[meetalipatel]

After the command prompt. It's not accepting any input from the keyboard.

[meetalipatel]

Hi Jimmy,

I wanted to share something I observed today.

The current CPUFREQ governor used is "ondemand", when the hang issue is observed. If I change the default governor to "userspace", there's no hang issue seen. My device is battery-powered, so using the "userspace" governor is not an option for me. 

One more thing, the release note says, CPUFreq can be used for CPU frequency adjustment. The Interactive governor is added and enabled by default.

But I cannot see the interactive governor defined in the kernel. Attaching the release note I am referring. Is the interactive governor removed from the kernel?

Thanks & Regards

Meetali Patel

[jimmychan]

I found that you are using the old version of u-boot. As our BSP release L6.1.22, the u-boot version is V2023.0. Could you try to upgrade the u-boot to v2023.04?

[meetalipatel]

Yes, I am using an older version of U-boot. In our project, we are trying to upgrade the kernel from 4.9.88 to 6.1.22. We already have our devices in the field and we will be updating the firmware by OTA. The update will only happen for kernel and rootfs. So, we want to use the new kernel with the older u-boot.

Thanks & Regards
Meetali Patel

[jimmychan]

The i.mx6ull is using this dts file. It should work.

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/arch/arm/boot/dts/imx6ul.dtsi

[meetalipatel]

Hi Jimmy,

Yes, it should work, but somehow it's not working! Any suggestions on how can I debug more on this?

Thanks & Regards

Meetali Patel

[jimmychan]

"when I configure the value to only 528000kHZ, it works fine and does not hang during boot up."

As you mentioned, you could only set the operating point at 528000kHz. You don't need to set other points as you set the same operating voltage.