S32G3 PFE External Timestamp Mode

[NidasioAlberto]

Hello,

I'm working on an S32G-VNP-RDB3 board and I'd like to configure it as a PTP Boundary Clock on the pfe1 and pfe2 interfaces.

Running `ptp4l -H -i pfe1 -i pfe2 -m` (with the BSP44.0) results in this error:

ptp4l[2178.575]: selected /dev/ptp2 as PTP clock
ptp4l[2178.576]: port 2 (pfe2): PHC device mismatch
ptp4l[2178.576]: port 2 (pfe2): /dev/ptp2 requested, ptp3 attached
ptp4l[2178.576]: failed to open port pfe2
failed to create a clock

And that is ok because ptp4l requires the interfaces to use the same hardware clock. In S32G_PFE_LNX_DRV_UM section 2.3.1.2 External Timestamp Mode, it is explained that the clock tree can be configured such that "EMAC0 is the time source and feeds GMAC, EMAC1 and EMAC2" (dotted line in the figure).

NidasioAlberto_1-1754658772398.png

To apply this configuration (as explained in the chapter), I modify the device three (when building the image with yocto) as following:

diff --git a/image/s32g-pfe.dtsi b/image/s32g-pfe-new.dtsi
index 94f886b..0a6ba1c 100644
--- a/arch/arm64/boot/dts/freescale/s32g-pfe.dtsi
+++ b/arch/arm64/boot/dts/freescale/s32g-pfe.dtsi
@@ -53,6 +53,7 @@
"pfe-shared-pool", "pfe-bdr-pool";
nxp,fw-class-name = "s32g_pfe_class.fw";
nxp,fw-util-name = "s32g_pfe_util.fw";
+ nxp,pfeng-emac-ts-ext-modes = <PFE_PHYIF_EMAC_1>, <PFE_PHYIF_EMAC_2>;
nxp,pfeng-ihc-channel = <PFE_HIF_CHANNEL_0>;
status = "disabled";

@@ -175,4 +176,3 @@
};
};
};
-

In the bootlogs I see this:

[ 8.716896] pfeng 46000000.pfe pfe1 (uninitialized): Subscribe to HIF1
[ 8.723466] pfeng 46000000.pfe pfe1 (uninitialized): Host LLTX disabled
[ 8.730319] pfeng 46000000.pfe pfe1 (uninitialized): Enable HIF1
[ 8.736399] pfeng 46000000.pfe pfe1 (uninitialized): setting MAC addr: 00:04:9f:be:ef:01
[ 8.744584] pfeng 46000000.pfe pfe1 (uninitialized): PTP HW addend 0x80000000, max_adj configured to 46566128 ppb
[ 8.754886] pfeng 46000000.pfe: IEEE1588: Using external timestamp input
[ 8.761903] pfeng 46000000.pfe pfe1 (uninitialized): Registered PTP HW clock successfully on EMAC1
[ 8.771626] pfeng 46000000.pfe pfe1: registered
[ 8.776283] pfeng 46000000.pfe pfe2 (uninitialized): Subscribe to HIF2
[ 8.782837] pfeng 46000000.pfe pfe2 (uninitialized): Host LLTX disabled
[ 8.789684] pfeng 46000000.pfe pfe2 (uninitialized): Enable HIF2
[ 8.795771] pfeng 46000000.pfe pfe2 (uninitialized): setting MAC addr: 00:04:9f:be:ef:02
[ 8.803945] pfeng 46000000.pfe pfe2 (uninitialized): PTP HW addend 0x80000000, max_adj configured to 46566128 ppb
[ 8.814257] pfeng 46000000.pfe: IEEE1588: Using external timestamp input
[ 8.821221] pfeng 46000000.pfe pfe2 (uninitialized): Registered PTP HW clock successfully on EMAC2
[ 8.830823] pfeng 46000000.pfe pfe2: registered

But then `ethtool` still reports different ptp hardware clock for each pfe interface:

root@s32g399ardb3:~# ethtool -T pfe0
...
PTP Hardware Clock: 1
...
root@s32g399ardb3:~# ethtool -T pfe1
...
PTP Hardware Clock: 2
...
root@s32g399ardb3:~# ethtool -T pfe2
...
PTP Hardware Clock: 3
...

And I get the same output from `ptp4l -H -i pfe1 -i pfe2 -m`.

What am I missing in the configuration? Is there something to change in the pfe firmware such that is needs to be recompiled?

[NidasioAlberto]

From the PFE driver repository on GitHub, I saw that the code managing the external timestamp mode was commited by @jpetrous, maybe he can be of help here

[alejandro_e]

Hello @NidasioAlberto,

Thanks reaching out to us. I did the steps described in the BSP44 User guide and I was able to configure PTP timestamp with no problem, I used a precompiled image, which should be the same as the one build by yocto. Although with a different command syntax, I used:

ptp4l -2 -H -i end0 -m -s

And got the following output:

root@s32g399ardb3:~#  ptp4l -2 -H -i pfe2 -m -s
ptp4l[331.563]: selected /dev/ptp2 as PTP clock
ptp4l[331.600]: port 1 (pfe2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[331.600]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[331.600]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[337.781]: selected local clock 00049f.fffe.beef02 as best master
ptp4l[385.758]: port 1 (pfe2): link down
ptp4l[385.758]: port 1 (pfe2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[385.788]: port 1 (pfe2): assuming the grand master role
ptp4l[385.788]: port 1 (pfe2): master state recommended in slave only mode
ptp4l[385.788]: port 1 (pfe2): defaultDS.priority1 probably misconfigured

The Link down message appeared after disconnecting the eth cable.

I just needed to configure the IP of the P3A port before, with the following:

ip addr add 192.168.0.131/24 dev pfe2

alejandro_e_0-1754696762765.png

I connected it to my Window 11 PC.

You should get the same ouput for PFE1 with:

ptp4l -2 -H -i pfe1 -m -s

Please let me know if I understood your problem correctly

[NidasioAlberto]

Thanks @alejandro_e for answering, but my problem still persist. I'll try to explain it further.

The PFE peripheral of the S32G3 has 3 interfaces, one is connected to the SJA1110 switch, and the other two are exposed on the board (PFE_MAC0 and PFE_MAC1 in the diagram).

S32G-VNP-RDB3 ports simple diagram.png

My target application involves connecting clients to the board through both interfaces, and to synchronize with them with PTP. For this reason I need ptp4l to run with both interfaces, not just pfe1 or pfe2 separately.

From the LinuxPTP documentation of the ptp4l tools, if I want to use hardware time stamping I read that.

All ports specified by the -i option and in the configuration file must be attached to the same PTP hardware clock (PHC).

So for this reason I need to configure the board such that both interfaces use the same hardware clock.

[alejandro_e]

Hello @NidasioAlberto,

Thanks for the clarification, I was not understanding your problem correctly. 

For convenience, in the following example I used end0 and pfe2, however you should get the same effect with pfe1 and pfe2.

Each command execution of ptp4l can only support one interface, you will need to run two commands, you can achieve that with the following:

ptp4l -2 -H -i pfe2 -s -m > /var/log/ptp4l_pfe2.log & ptp4l -2 -H -i end0 -s -m > /var/log/ptp4l_end0.log &

The '&' at the end of each command will send each execution to the background, therefore you will get control of the terminal. the output will be saved in the files indicated in each command, this can be named however you want, I chose that name for convenience. 

For example:

s32g399ardb3 login: root
root@s32g399ardb3:~# ip addr add 192.168.0.131/24 dev pfe2
root@s32g399ardb3:~# ip addr add 192.168.0.130/24 dev end0
root@s32g399ardb3:~# ptp4l -2 -H -i pfe2 -s -m > /var/log/ptp4l_pfe2.log & ptp4l -2 -H -i end0 -s -m > /var/log/ptp4l_end0.log &
[1] 455
[2] 456
root@s32g399ardb3:~# jobs
[1]-  Running                 ptp4l -2 -H -i pfe2 -s -m > /var/log/ptp4l_pfe2.log &
[2]+  Running                 ptp4l -2 -H -i end0 -s -m > /var/log/ptp4l_end0.log &
root@s32g399ardb3:~# cat /var/log/ptp4l_pfe2.log
ptp4l[79.716]: selected /dev/ptp2 as PTP clock
ptp4l[79.744]: port 1 (pfe2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[79.744]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[79.745]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[85.948]: selected local clock 00049f.fffe.beef02 as best master
root@s32g399ardb3:~# cat /var/log/ptp4l_end0.log
ptp4l[79.716]: selected /dev/ptp3 as PTP clock
ptp4l[79.760]: port 1 (end0): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[79.760]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[79.760]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[85.911]: selected local clock b2ecbb.fffe.ef0375 as best master
root@s32g399ardb3:~#

First I assigned an IP for each interface, then I invoked ptp4l for each interface and finally I showed the contents of the log files to the terminal.

For this I connected devices to both P3B (PFE2) and P3A (GMAC0), otherwise some error logs may appear.

Let me know if this solves your question.

[NidasioAlberto]

By running two separate ptp4l instances (more so with the `-s` option that enable client only mode), you have that both interfaces can become clients at the same time. When this occurs, and the peripheral hardware clocks of both interface gets synchronized, which one should be used by phc2sys to synchronize the system clock?

What I want to achieve is to have one single instance of ptp4l running on multiple ports synchronizing one single hardware clock. I want to implement a boundary clock that synchronize to an upstream master on one port, and provides it's clock as a master on the others.

ptp4l can work with more than one interface. The LinuxPTP documentation says so and I've also achieved what I want to do with the SJA1110 switch. What I did was to enable the DSA driver for the SJA and to run `ptp4l -i p2 -i p3 -2 -m`. The DSA driver exposes the switch interfaces in linux, and they have the same hardware clock (ethtool also reports so, showing clock 4). Since the two ports have the same hardware clock, ptp4l can use both of them. In this configuration I see that ptp4l chooses one master over one of the two interfaces, and if that master fails, a new master is selected from the other. And the board itself can become the master if the BMCA algorithm chooses so. And with this I can also run `phc2sys -a -rr -m` to automatically synchronize the system clock to the hardware clock if the board is a slave, or the hardware clock to the system clock if it acts as the grandmaster.

Since I enabled the "External Timestamp Mode" for pfe1 and pfe2 (as described by my first post), pfe1 and pfe2 hardware clocks are derived by the hardware clock of pfe0. By running what you provided, that is two ptp4l instances in client only mode, when ptp4l needs to adjust the hardware clocks, it gives an error.

root@s32g399ardb3:~# ptp4l -i pfe2 -2 -m
ptp4l[72513.760]: selected /dev/ptp3 as PTP clock
ptp4l[72513.796]: port 1 (pfe2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[72513.796]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[72513.796]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[72514.342]: port 1 (pfe2): new foreign master 00049f.fffe.096f15-1
ptp4l[72518.342]: selected best master clock 00049f.fffe.096f15
ptp4l[72518.343]: port 1 (pfe2): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[72520.342]: master offset -1593121442 s0 freq +227123 path delay 788
ptp4l[72521.349]: failed to adjust the clock: Operation not supported
ptp4l[72522.342]: master offset -1593112737 s0 freq +231478 path delay -211
ptp4l[72523.349]: failed to adjust the clock: Operation not supported

And this does make sense becase if I tell the board that hardware clocks of pfe1 and pfe2 are derived from pfe0, then I cannot momdify them.

So my problem is. With the "External Timestamp Mode" enabled for pfe1 and pfe2, why in Linux the pfe1 and pfe2 peripherals are not related to hardware clock 1 but are instead related to their clocks (2 and 3 respectively). I mean, with the DSA driver for the SJA1110, different interfaces are related to the same hardware clock (in this case `/dev/ptp4`). Please refer to my first post for more details. So, is it a bug or do i need to configure something else? Maybe there is something else to modify in the device tree that I'm missing?

Would it also be possible to give some reference to documentation? Like, if ptp4l does not support multiple ports, where can I find this documented?

Btw, thanks a lot @alejandro_e for helping me.

[alejandro_e]

Hello @NidasioAlberto

The problem your are facing with the clocks, this is, using "External timestamp mode" and getting the wrong configuration with the ptp4l is out of my expertise. I will reach out to the internal team for better support on this topic. This might take some time, I appreciate your patience. 

Regarding the ptp4l documentation, please check the following:

Chapter 20. Configuring PTP Using ptp4l | System Administrator’s Guide | Red Hat Enterprise Linux | ...

alejandro_e_0-1755041461812.png

It does mention that a group of interfaces can be started at the same time with the same instance, however, that is related to the ptp4l program, which is not provided by NXP and therefore we cannot offer much support, other than what we currently provide. Sorry for the inconviniences.

I will get back to you regarding the first topic of this reply.

[NidasioAlberto]

Thanks @alejandro_e for the follow-up and for escalating the issue. I'll wait for a feedback.

[alejandro_e]

Hello @NidasioAlberto,

I have received the following feedback from the internal team:

"

If PFE_EMAC1 and PFE_EMAC2 share timestamp from PFE_EMAC0, it doesn't support to adjust PTP time of PFE_EMAC1 and PFE_EMAC2, you can think PTP time registers work in read-only mode. and reflect the PTP time of PFE_EMAC0. In this case in order to synchronize time you should use PFE_EMAC0.

" 

Let me know if this information helped with your problem

[NidasioAlberto]

Thanks @alejandro_e for getting back to me so quickly.

I understand what the internal team is saying.

What I don't understand is why, when configurationg "External Timestamp Mode" for pfe1 and pfe2, the hardware clock associated with pfe1 and pfe2 is not the same of pfe0 but still their own independent clock. With this I'm referring to what the pfe driver exposes to Linux, and thus to ptp4l.

What I would expect is that insteal of seeing this:

root@s32g399ardb3:~# ethtool -T pfe0
...
PTP Hardware Clock: 1
...
root@s32g399ardb3:~# ethtool -T pfe1
...
PTP Hardware Clock: 2
...
root@s32g399ardb3:~# ethtool -T pfe2
...
PTP Hardware Clock: 3
...

I see this:

root@s32g399ardb3:~# ethtool -T pfe0
...
PTP Hardware Clock: 1
...
root@s32g399ardb3:~# ethtool -T pfe1
...
PTP Hardware Clock: 1
...
root@s32g399ardb3:~# ethtool -T pfe2
...
PTP Hardware Clock: 1
... 

[alejandro_e]

Hello @NidasioAlberto,

My understanding is that the number of the clock in the output of ethtool is the id of the 'virutal device', and does not necessarily reflect the internal configuration of the module.

For each device a different clock device gets registered, it happens here linux-pfeng/pfeng-ptp.c#L247 

for example, using the base BSP44 image, you can see all the PFE_EMAC instances register a PTP clock:

root@s32g399ardb3:~# dmesg | grep "Registered PTP HW clock successfully on EMAC"
[    9.525257] pfeng 46000000.pfe pfe0 (uninitialized): Registered PTP HW clock successfully on EMAC0
[    9.593027] pfeng 46000000.pfe pfe1 (uninitialized): Registered PTP HW clock successfully on EMAC1
[    9.653784] pfeng 46000000.pfe pfe2 (uninitialized): Registered PTP HW clock successfully on EMAC2

This clock register process gets iterated here linux-pfeng/pfeng-netif.c#L1544, for each interface.

You could modify the source code so that all the interfaces use the same virtual clock. Which is related to the information I shared before:

alejandro_e_0-1755555300778.png

If all the PFE_EMAC interfaces register the same clock you may be able to use a single instance of ptp4l for all the interfaces you are interested in. 

For more details you can use the following commands:

root@s32g399ardb3:~# ls -l /sys/class/ptp/
total 0
lrwxrwxrwx 1 root root 0 Oct  9 15:51 ptp0 -> ../../devices/platform/soc/46000000.pfe/ptp/ptp0
lrwxrwxrwx 1 root root 0 Oct  9 15:52 ptp1 -> ../../devices/platform/soc/46000000.pfe/ptp/ptp1
lrwxrwxrwx 1 root root 0 Oct  9 15:52 ptp2 -> ../../devices/platform/soc/46000000.pfe/ptp/ptp2
lrwxrwxrwx 1 root root 0 Oct  9 15:59 ptp3 -> ../../devices/platform/soc/4033c000.ethernet/ptp/ptp3

root@s32g399ardb3:~# ls /sys/class/ptp/
ptp0  ptp1  ptp2  ptp3
root@s32g399ardb3:~# ls /sys/class/ptp/ptp0/
clock_name  dev  device  max_adjustment  max_vclocks  n_alarms  n_external_timestamps  n_periodic_outputs  n_programmable_pins  n_vclocks  power  pps_available  subsystem  uevent
root@s32g399ardb3:~# cat /sys/class/ptp/ptp0/clock_name
pfeng ptp
root@s32g399ardb3:~# cat /sys/class/ptp/ptp0/dev
251:0
root@s32g399ardb3:~# cat /sys/class/ptp/ptp0/max_vclocks
20

This last part matches the following structure linux/include/linux/ptp_clock_kernel.h#L166.

Let me know if this information was of help

[NidasioAlberto]

Thanks @alejandro_e, I'll try to patch the PFE driver then. 

[alejandro_e]

Hello @NidasioAlberto,

Perfect, that is exactly what I conclude is needed.

If you require more support please let me know and I will do my best effort to help you.

Given the extend of this post, if you need to follow this topic, please create a new post, you can reference this one to keep track of the context.

Thanks