High CPU load average with LSDK v21.08 Ubuntu 20.04

[kkrambo]

I have a custom board modelled after the LS1012A-FRWY and LS1012A-RDB.  I built a new Ubuntu 20.04 distribution for my board based on LSDK v21.08 and when I run this build the Linux CPU load average is 2.50.  This is much greater than load average with my older Ubuntu 18.04 distribution based on LSDK v20.04.

When I boot up and login to the Ubuntu20.04/LSDKv21.08 distro, the top command shows the load average is around 2.5 and the CPU is busy handling interrupts.  The cat /proc/interrupts command shows that the arch_timer interrupt is occurring about 250 times per second.

top - 05:00:48 up 4 min,  1 user,  load average: 2.48, 1.62, 0.70
Tasks:  84 total,   3 running,  81 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us, 68.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi, 31.4 si,  0.0 st
MiB Mem :    904.8 total,    720.2 free,     78.5 used,    106.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    760.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     10 root      20   0       0      0      0 S  32.9   0.0   1:18.38 ksoftir+
    140 root      20   0       0      0      0 R  32.9   0.0   0:32.72 kworker+
      5 root      20   0       0      0      0 R  30.3   0.0   0:55.21 kworker+
    265 root      20   0       0      0      0 I   2.6   0.0   0:30.32 kworker+
    758 root      20   0   13168   2956   2544 R   0.3   0.3   0:00.56 top
      1 root      20   0  101072   9668   7052 S   0.0   1.0   0:04.30 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd

When I do the same with my Ubuntu18.04/LSDKv20.04 distro, top shows the load average is near zero and cat /proc/interrupts shows the arch_timer interrupt is occurring about 10 times per second.

top - 09:36:29 up 20 min,  1 user,  load average: 0.01, 0.00, 0.00
Tasks:  75 total,   1 running,  38 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   928432 total,   793928 free,    53472 used,    81032 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   808084 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  885 root      20   0    5924   2976   2456 R  0.3  0.3   0:04.49 top
    1 root      20   0   13124   7524   5676 S  0.0  0.8   0:03.52 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
    3 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 rcu_par_gp

I found these related configuration settings in the linux kernel .config file but the settings are the same for both builds.

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem

# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

Why does the arch_timer interrupt occur 250 times per second with my Ubuntu20.04/LSDKv21.08 build, but only 10 times per second with my Ubuntu18.04/LSDKv20.04 build?  Is there another kernel configuration option that explains this?

[yipingwang]

I verified LSDK 21.08 re-built image, it works normally, please refer to my following log. The load average parameter is nearly 0.

root@localhost:~# top
top - 19:54:20 up 54 min, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 80 total, 1 running, 79 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 893.1 total, 597.7 free, 96.6 used, 198.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 729.8 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
791 root 20 0 13168 3108 2692 R 0.7 0.3 0:00.30 top
1 root 20 0 101260 9892 7172 S 0.3 1.1 0:05.44 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par+
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_perc+
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_tas+
10 root 20 0 0 0 0 S 0.0 0.0 0:00.23 ksoftir+
11 root 20 0 0 0 0 I 0.0 0.0 0:00.03 rcu_pre+
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migrati+
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmp+
15 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
16 root 20 0 0 0 0 I 0.0 0.0 0:00.09 kworker+
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_rea+
19 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeba+

Probably there is problem with your own rootfs filesystem, please try to do verification with the prebuilt ubuntu main rootfs filesystem downloaded from the following link.

$ wget https://www.nxp.com/lgfiles/sdk/lsdk2108/rootfs_lsdk2108_ubuntu_main_arm64.tgz

If your problem persists, please try the pre-built Kernel image in boot_LS_arm64_lts_5.10.tgz

$ wget https://www.nxp.com/lgfiles/sdk/lsdk2108/boot_LS_arm64_lts_5.10.tgz