Slower boot with 5.10 kernel

[christopher_les]

We're building our embedded Linux distro for a custom i.MX6 PCB design using the Yocto hardknott version of meta-freescale.  Previously we'd been using the linux-imx 4.14.98 (zeus), then linux-imx 5.4.x (dunfell, gatesgarth, and earlier hardknott) kernels - but now recently the hardknott branch switched to 5.10.35 kernel. 

With the update to 5.10.x kernel we've observed a noticeable regression (slowdown) in the boot-time for our embedded device: loading of the linux kernel and driver modules now takes ~2X what it previously did on 4.14 or 5.4.x kernels:  the kernel binary startup increased from ~600msec --> 1.2msec, and overall application startup time (which includes loading of many kernel modules) increased from ~10sec --> 13sec.

After searching around a bit I found some discussions related to "fw_devlink" functionality now added in the 5.10 kernel which I believe is (at least partially) related.  For example, I found that I could add the kernel command-line: "fw_devlink=off" and observe the kernel startup reduce back down to ~600msec, as it previously was for 5.4 kernel.  However, the overall application startup time is still much slower on 5.10 versus what it is on 5.4.  There are no other changes to our Linux distro or application code in observing these boot time differences: both are built from the same Yocto hardknott version, only the linux-imx kernel version has been changed.

For now, we've decided to stick with the 5.4.x kernel with faster boot-time...  However, I'm still looking for ideas or insights on what could be causing a slow-down with the 5.10.x kernel and how I might resolve that for our build.  Has anyone else observed the same behavior?

[nreis]

@christopher_les I am currently stuck with a similar problem. Were you able to find a solution? Thanks in advance for any ideas!

[Bio_TICFSL]

Hello Christopher,

We have added a new GPU version in 5.10 and uboot, we know that this are going to up the speed, but not that far, however you can use your uboot and try the 5.10.35BSP or you can try the newest BSP 5.10.52 to see if that reduce the time of booting.

Hope this helps.

[christopher_les]

Thank you for the reply.

I did already test with the linux-fslc-imx_5.10 kernel, which is forked from the NXP 5.10.52 BSP, and currently at 5.10.69 upstream kernel version.  I observed identical performance with that version as with the imx 5.10.35 BSP: still noticeably slower than 5.4 kernel.

For uboot, we're currently using uboot-fslc_2020.04 version, and for consistency (and because we've added some local patches and customizations there), I have not changed the uboot version in our build when making boot-time comparisons between 5.10 and 5.4 kernel.  Can you think of anything specific that should be initialized differently, in uboot, for the 5.10 kernel to boot faster?

The only thing that's substantially changed in our build (to exhibit the observed slowdown) started with the `meta-freescale` update on ~9/11/2021, in which the linux-imx version changed from 5.4 to 5.10.  Any builds after that with (any) 5.10.x kernel are slower.  Any builds after that with (any) 5.4.x kernel are the same speed as they were before.  I noticed, for example, the GPU version was bumped from 6.4.3.p1.4 --> 6.4.3.p2.0 along with the 5.10 update, but that and any other current BSP changes do not seem to affect boot time at all with the 5.4 kernel.

My primary suspicion at this point is that there's some behavior in the upstream kernel that changed between 5.4 and 5.10... and I'm searching if there's anything that can be done about it to get back to same performance we were achieving with the 5.4 kernel.