[i.MX8ULP] linux clocksource precision

[martinetd]

Hi

We're building a product with an i.MX8ULP processor and one of our users noticed that the time drift is pretty bad (this is unit dependents, where some units are "only" a few seconds fast in about an hour, but some devices easily go fast one second per minute)

I've reproduced on our i.MX8ULP EVK, running the latest BSP (LF_v6.12.20-2.0.0), using chrony to measure the drift (requires internet)

imx8ulpevk login: root
# stop ntpd/timesyncd to avoid interfering... (why are both running?!)
root@imx8ulpevk:~# systemctl disable --now ntpd
root@imx8ulpevk:~# systemctl disable --now systemd-timesyncd
# run chronyd as a container for simplicity.
# tmpfs ensures drift state is not preserved accross restarts
root@imx8ulpevk:~# docker run --net=host --name ntp -d simonrupf/chronyd
Unable to find image 'simonrupf/chronyd:latest' locally
latest: Pulling from simonrupf/chronyd
d69d4d41cfe2: Pull complete 
5f93ef52bdbb: Pull complete 
3a84ecea1ef1: Pull complete 
Digest: sha256:6c2a693438b6c663f151516ccce21b409f0891df673a34cadf133c74b35b7b6b
Status: Downloaded newer image for simonrupf/chronyd:latest
15a0cdd35648868fe8592e98f7d653a6e9fd5abec5ff8c335b6edffc69617fe8

# (here need to wait a bit for the Frequency error to be computed
# based on measurements stability. If it doesn't compute it helps
# to run something like `chronyc burst 4/10`)
root@imx8ulpevk:~# docker exec ntp chronyc tracking 
Reference ID    : 2D4D1467 (v4.ntp.admtan.jp)
Stratum         : 3
Ref time (UTC)  : Tue Jul 22 07:26:57 2025
System time     : 0.343799949 seconds fast of NTP time
Last offset     : +0.068965137 seconds
RMS offset      : 0.129153565 seconds
Frequency       : 12426.411 ppm fast
Residual freq   : +0.000 ppm
Skew            : 100.904 ppm
Root delay      : 0.018474324 seconds
Root dispersion : 0.001048955 seconds
Update interval : 2.0 seconds
Leap status     : Normal

So the interesting line here is Frequency, described as "The ‘frequency’ is the rate by which the system’s clock would be wrong if chronyd was not correcting it. It is expressed in ppm (parts per million). For example, a value of 1 ppm would mean that when the system’s clock thinks it has advanced 1 second, it has actually advanced by 1.000001 seconds relative to true time."

In this case, that means 12426.411 * 10^-6 * 3600 = ~45 seconds would pass too fast in 1h, but this appears to change on reboot

So, a few questions:

- Where does the arch_sys_counter gets its clock from? Is that a problem with one of the crystals on the board being low precision? (... if so we copied that flaw from the EVK...)

- I've noticed there are two clocksources available, and switching to imx-tpm is much more stable (either `echo imx-tpm > /sys/devices/system/clocksource/clocksource0/current_clocksource` or setting `clocksource=imx-tpm` in boot args), is there any reason it's not the default? I'm thinking of sending a patch to increase the default from 200 to 500 in the 32-bit counter case (need higher than 400 to have priority over arch sys counter), but I'd like to understand why a lower value was chosen first, if there is any reason.

Thank you

[martinetd]

> root@imx8ulpevk:~# docker run --net=host --name ntp -d simonrupf/chronyd

Just correcting myself on this, I copied the wrong command and this is missing the tmpfs for /var/lib/chrony/ -- it needs something like this to avoid preserving the drift file (which if I understand this correctly would adjust the frequency and not report the offsets we see at chrony startup? that might have just been luck)

docker run --restart=always --net=host --name ntp -d --tmpfs /var/lib/chrony:rw,mode=1750,uid=100,gid=101 simonrupf/chronyd

with restart=always the container will restart when docker starts, which is on the first docker command as it is socket-activated by default, enabling docker.service will make the container run fresh at boot.

[JorgeCas]

Hello,

Do you mean that this helps with the timing issue? Or still present?

Best regards.

[martinetd]

Sorry, I just meant this to correct the "reproducing steps" I gave in my original post: without the --tmpfs option a docker volume would be used to persist the chrony drift file, and in the few tests I ran with such a file I observed lower `Frequency` values so I think it makes testing harder.

To be clear, we're running chrony (normally) on our boards and if there is an internet connection then there is no problem as the clock can be adjusted/corrected, but in offline mode then the drift is too big for some of our customers.
I don't know how exactly they use our board, but even if they don't display the system time directly it would for example make system logs harder to analyze when an issue occurs after a few days/weeks offline because timestamps are too far from one would expect -- the above example of 45 seconds per hour would drift over 2 hours in a week! For comparison 50ppm offset would only drift at most 30 seconds in the same week (50*10^-6*3600*24*7)


So I'm looking to improve this; it looks like using the tpm clocksource is better and I'm thinking of making it the default as a first solution, but I'd like to understand why the arch_sys_counter is so far off the (supposed) crystal frequency as that might cause other timing issues down the road.

Thanks

[JorgeCas]

Hello,

Let me check this with internal team since you are able to reproduce the issue in EVK.

I will check if this can be fixed with software.

Best regards.

[martinetd]

Thank you that is appreciated!
I'll keep looking a bit on my end as well and report if I figure anything.

[JorgeCas]

Hello,

Internal team as able to tun the same tests and confirmed that the arch_sys_counter delivers relatively poor precision, whereas the imx-tpm timer is markedly more accurate. As mentioned before, the ARM generic timer includes a system counter and set of per-core timers.

• The System Counter is an always-on device, which provides a fixed frequency incrementing system count.
• The system count value is broadcast to all the cores in the system, giving the cores a common view of the passage of time.
• On the 8ULP, the System Counter is implemented as the TSTMR module.
• It is driven by an internal 1 MHz LPO, whose frequency stability is inherently limited—hence the observed precision issues with arch_sys_counter.

If your application demands tighter timing accuracy, they can switch to the TPM as their clock source. Enabling TPM-based timing requires instantiating TPM timers on both CPU cores to preserve per-core event handling.

Here's the patch about dual TPM timers enablements. To test it you need to rebuild your ATF to generate new bl31.bin.

make -j8  PLAT=imx8ulp CROSS_COMPILE=aarch64-poky-linux- bl31 IMX8ULP_TPM_TIMERS=1

And you need to use imx8ulp-evk-tpm.dtb device tree file. 

Any questions, let me know.

Best regards.

[martinetd]

Thank you for the reply!

> It is driven by an internal 1 MHz LPO, whose frequency stability is inherently limited—hence the observed precision issues with arch_sys_counter.

Ah! That makes sense, thank you

> Here's the patch about dual TPM timers enablements. To test it you need to rebuild your ATF to generate new bl31.bin.

I'm on vacation this week so I'll check next week (in particular the atf code for IMX8ULP_TPM_TIMERS was added in lf-6.12.20-2.0.0 and we're on an older branch, so I'll need to rebase and validate first)

We've been running with a single tpm instance by just selecting the imx-tpm clocksource and I had not noticed this problem:

> Enabling TPM-based timing requires instantiating TPM timers on both CPU cores to preserve per-core event handling

Could you clarify what's wrong with enabling only a single tpm timer? "just" more load on cpu0 to keep the system clock up to date? Or are there other downsides?

Thank you again,

Dominique

[JorgeCas]

Hello,

Thank you for test this in EVK, this issue has not been reported before.

Regarding your questions:

The Generic Timer provides a standardized timer framework for Arm cores. The Generic Timer includes a System Counter and sets of per-core timers.

The System Counter is an always-on device, which provides a fixed frequency incrementing system count. The system count value is broadcast to all the cores in the system, giving the cores a common view of the passage of time. Which is implemented by SCTR.

The System Counter inputs two counter clock sources and outputs a gray coded counter value and interrupt signals (one per compare frame) to the platform’s interrupt controller.

After reset, the System Counter is disabled with count value reset to zero and base frequency selected.

Once the counter is enabled, it will increment the appropriate value on each rising edge of the selected clock.

Best regards.

[martinetd]

Hello,

thanks for the quick reply!
The crystals we're using for both 24MHz and 32768Hz have a tolerance of +/-20ppm, so I'm surprised we see similar (~5000-15000ppm) drifts on the system counter for our boards.

Is there anything you could advise me to check?

Thanks

[JorgeCas]

Hello,

We usually suggest a tolerance of +/-50ppm or less so, this is ok.

Best regards.