IEEE1588 time-stamping issue

[christophe31]

Hi,

I'm trying to use IEEE1588 time-stamping capability on a Vybrid Tower kit TWR-VF65GS10.

I'm using the 50MHz external clock to clock the ENET module, thus I use an increment value (ENET_ATINC) of 20 (0x14). That way I can directly read values expressed in nano-second.

It seems to work correctly, but when I use the timestamp counter (ENET_ATVR) to measure some code performance I get a strange behaviour :

I make 2 succesive readings of ENET_ATVR (which first needs to set the CAPTURE flag of ENET_ATCR register), then I compare the 2 values.

On a very small piece of code, most of the time, the difference is an almost constant value (560 or 580 ns), but sometimes (?!) values are not consistant at all. I can observe various behaviors :

- Increment is not consistent :

  t1 = 0x7AD650B4

  t2 = 0x7AD6548C

  diff. = 984

==> the difference is greater that 580, but primarily it is not a multiple of 20 ?!

- The difference is negative :

  t1 = 0x0862D280

  t2 = 0x0862D0C4

  diff. = -444

  Here, the expected t2 value is 0x0862D4C4 (seems to be a defect on bit 10), but this is not always so trivial e.g. :

  t1 = 0x60EB193C

  t2 = 0x60EB135C

I also note another strange behavior : I launch my SW normally, then I break and disable time-stamping (I set ENET_ATCR=0) with my debugger (Lauterbach Trace32).

Then I set the CAPTURE flag many times to acquire some ATVR values. What I get is 2 different values separated by the increment value. Values are not coming in a regular flow, e.g. :

0x2EAECC10

0x2EAECC10

0x2EAECC10

0x2EAECC10

0x2EAECC24

0x2EAECC10

0x2EAECC24

0x2EAECC24

...

It seems that data come from a dual buffer. Could it be the origin of the problem ?

Here is my IEEE1588 registers configuration :

      ATCR    00000001

      ATOFF   00000000

      ATPER   80000000

      ATCOR   00000000

      ATINC   00001414

I experiment this on both ENET0 and ENET1

Did someone face such an issue ?

Thanks for any help

Regards

Christophe

[naoumgitnik]

Christophe,

Regarding "... 50MHz external clock to clock the ENET module...." - do you mean  the Ethernet PHY clock is not generated by our Vybrid?

/Naoum.

[christophe31]

Hi Naoum,

Thanks for these answers

for those interested in it, we recommend to use the MII interface instead of the existing RMII - the Tower PHY module does support both of them, as far as i know.

Hum, maybe this is the point, but I need 2 network interfaces. I'm using the 2 ENETs with a TWR-SER2 module. Thus, I have no choice than using RMII. Am I wrong?

Moreover, I don't understand what it changes... Time-stamp seems to be an ENET's sub-component. Could you explain what impact has MII/RMII on it?

Regarding "... 50MHz external clock to clock the ENET module...." - do you mean  the Ethernet PHY clock is not generated by our Vybrid?

Yes. My ENET clock is generated by the TWR-SER2 board, which clocks both PHY and Vybrid's ENET/MAC (via CLKOUT0). Do you think it would change something if I use the Vybrid internal clock?

Thanks again for your help !

Best regards,

Christophe

[naoumgitnik]

Dear Christophe,

Regarding "I need 2 network interfaces. I'm using the 2 ENETs with a TWR-SER2 module. Thus, I have no choice than using RMII. Am I wrong?" - as per '8.7 Communication interfaces', page 575 of 'Vybrid Reference Manual, Rev. 5, 07/2013' - "MII mode is supported for MAC0 only and when MII mode for MAC0 is enabled, RMII of MAC1 cannot be used." - You are NOT wrong.

As I learned, it is somehow related to the RMII signal jitter. No such problem with MII.

More details later this week.

Based on the Vybrid datasheet, below are relevant MII connections between it J17A (elevator) connector to be added onto the Tower board (planned to be implemented in the next board revision):

PTA6 – MII_TXCLK to J17A – B14

PTA21 – MII_RXCLK to J17A – A15

PTD17 – MII_TXERR to J17A – B16

PTD18 – MII_TX2 to J17A – B18

PTD19 – MII_TX3 to J17A – B17

PTD20 – MII_COL to J17A – B12

PTD21 – MII_CRS to J17A – A12

PTD22 – MII_RX2 to J17A – A18

PTD23 – MII_RX3 to J17A – A17

Regards, Naoum Gitnik.

[christophe31]

Naoum, Jiri,

Thanks for your involvement !

Maybe a workaround here :

  ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

  __asm("DSB");

  __asm("nop");

  __asm("nop");

  __asm("nop");

  __asm("nop");

  ts_counter_ns = ENET0->ATVR;

I did not make a lot of tests but it seems much better this way.

DSB makes the CPU (Cortex M4 here) wait until the capture flag is really set.

At least 4 NOP must be added in order to obtain a coherent value.

So I guess there are 2 problems :

- in some cases, the ATCR write might be treated after the ATVR read (I'm not sure if Cortex-M4/NIC301 architecture allows that, but it could explain some negative results). This is prevented with DSB instruction.

- the ATVR register is not updated synchronously. Additional NOPs allow to wait until all bits are properly set.

BTW, this is not perfect, because it makes me wait a "long" time (approx. 21 CPU cycles) for "nothing", which is not ideal for constrained real-time application, so I'm still expecting a better solution... if possible!

Regards,

Christophe

[naoumgitnik]

Dear Christophe,

I am not sure if workarounds help, based on the below information from our people who tried to deal with this feature. Please, find some excerpts below:

* The issue is confirmed on both Vybrid and Kinetis Tower Modules (with RMII connection). 

* The jitter observed with RMII interface is actually a known matter, i.e. this behavior is also found in other Ethernet controllers, e.g. the INTEL i210 datasheet (http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-data...). Table 7-62 on page 340 states that the jitter on 100-Mbps RX frame time-stamping is 80ns (2228 - 2148 = 80), and this is exactly what we have observed in our Ethernet Controller.

The jitter in RMII, which is bigger (in general) than the MII, is mainly due to the nature of the interface itself.

In MII, data and the clock are effectively forwarded to the MAC (i.e. they are source-synchronous), but that’s not the case for RMII; therefore RMII is understood to introduce greater jitter.

* ... MII (RGMII/GMII)  is preferred as far as 1588 time-stamp concerns.

* The TWR-Vybrid performance ... below +/-130ns (have the same 80ns jitter as Kinetis k60).

You can probably find more speculations about this on the Internet.

Regards, Naoum Gitnik.

==========================

Correction: what is called "jitter" in the above reply is rather "latency".

Sorry  - it took me additional thinking to realize that the term used was quite incorrect (signal jitter is something different).

[christophe31]

Hi Naoum,

Really interesting, I have to take care of that in my design, I can deal with a known jitter, but I fear that we don’t talk about the same thing!

For now, I don’t talk about reading the time-stamp of incoming frames. I just compare 2 values of ATVR, which is supposed (as far as I understand) to be representative of an internal time-stamp counter always increasing (up to its maximum value of course).

Or maybe I totally misunderstand: do you suggest that ATVR could be altered by incoming frames?

To be more specific, regarding my first post, my source code simply looks like:

ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

t1 = ENET0->ATVR;

…some code (to measure performance of)…

ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

t2 = ENET0->ATVR;

diff = t2 - t1;

where t1 and t2 values are not consistent, except with additional DSB/NOP as said above. This is the point!

Of course, this is temporary code, and I also use ATVR to compare with time-stamp of incoming frames. But at first I ‘d like to be confident with ATVR content…

One more question: could you tell me where does the last excerpt come from ? (The TWR-Vybrid performance ... below +/-130ns (have the same 80ns jitter as Kinetis k60).). If possible I'd like to have the full background.

Thanks and regards,

Christophe

[naoumgitnik]

Dear Christophe,

1.

That's a good point the Juan mentioned Kinetis - I found several threads it might make sense for you to look into (sorry, I am not a SW person, thus cannot judge how much they may help in this case):

• https://community.freescale.com/message/350709#350709
• https://community.freescale.com/message/93334#93334
• https://community.freescale.com/message/300916#300916
• https://community.freescale.com/message/300917#300917.

2.
Some details of the test using 2 different Freescale's boards, which are not Vybrid-based (cannot really comment on the below text - not an IEEE 1558 expert):

"I’m not running any 1588 software stack, simply set one board up to transmit 500 back-to-back frames by preparing 500 TxBDs and the second board receive all 500 frames by preparing 500 RxBDs.

... at the end of the test, I look at the differences in deltas of consecutive timestamps in the buffer descriptors to determine latency for each pair of frames, i.e. Latency = | (T_n – T_n-1) – (R_n – R_n-1)|.

Since the transmit frames are all back-to-back, I never saw any latency in the TxBD timestamps, i.e. T_n –T_n-1 was always the same for each test. Therefore, the latency values below represent the latency in the receive time-stamping:

1.    2-nd board RMII (Tx) -> 1-st board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

2.    1-st board RMII (Tx) -> 2-nd board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

3.    2-nd board MII (Tx) -> 1-st board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

4.    1-st board RMII (Tx) -> 2-nd board MII (Rx) – 20ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

5.    1-st board RMII (Tx) -> 2-nd board MII (Rx) – 10ns latency (IEEE 1588 timestamp clock source equals 100 MHz).

These results line up with what I would expect:

• For experiments 1-3, we see the 80ns latency when using the RMII as the receiver.
• For experiments 4 and 5, we see a reduced latency when MII is used as the receiver.

It must be noted that additional latency may be seen as a result of the MAC detecting the SFD in the MII clock domain and capturing the timestamp in the IEEE 1588 clock domain.

If two these clocks are from different sources, I expect additional latency equal to the period of the IEEE1588 timestamp clock:

• For experiments 1-3, when the receiver is in RMII mode, the 50 MHz crystal on the serial board is driving the XTAL pin, is the source of the RMII clock, and used to generate the 1588 timestamp clock. So, those two clock sources are the same, so no additional latency.
• For experiments 4 and 5 when the receiver is in MII mode the PHY is driving the MII Rx clock, which is recovered from the RX data and is essentially the Tx clock from the transmitter board and the crystal on the serial board is driving the XTAL  pin, which is used to generate the 1588 timestamp clock. So in this case the clock sources are different, so we expect additional latency, which we see.

... A simple message ... is there will be a possible latency of 80ns in 100Mb/s operation (and 800ns in 10 Mb/s) operation due to Rx to Tx clock synchronization.

• For RMII, the Rx clock is given to the PHY, so synchronization has to occur between the Tx and Rx clock fairly often.
• For MII, the Rx clock is driven from the PHY and can be recovered from the data, so can stay synchronized with the Tx clock."

Regards, Naoum Gitnik.

[christophe31]

Hi Naoum,

Thanks for these new information, they are of great interest for my work.

But there is still nothing related to the ATVR reading error.

However, thanks to Juan, I saw I’m not the only one having problem with that!

Do you think that MQX’s workaround is an official and/or an approved solution?

Regards,

Christophe

Edit : after re-reading CM4 ref. manual, I think that the second writing of the capture flag forces to wait until the first one is effective. This is no more than DSB instruction does. Thus QMX solution does not seem better than mine finally :-(

[naoumgitnik]

Dear Christophe,

Everything posted on our official web site is an "official and/or an approved solution".

The Community forum materials, though, cannot be considered as such; if it works or not is solely based on the specific thread contents.

Regards, Naoum Gitnik.

[christophe31]

Hi Naoum,

The problem is that my software is destined to avionics, and I have to justify every piece of code in the eyes of certification authorities. Thus I can hardly explain why I insert 4 NOPs if I don’t have an application note, or an errata, or an answer from HW designer, because the certifying officer will ask :

“Why only 4 NOPs? Are you sure there is no situation where 5 NOPs would be needed?...”

Even though there’s a workaround in MQX, it is not clear whether the solution will be approved or not…

Regards,

Christophe

[juangutierrez]

Hi

Well, here is the errata document for kinetis

http://www.freescale.com/docs/pcn_attachments/15543_KINETIS_8N30D_prelim2.pdf

Look for: e2579: ENET: No support for IEEE 1588, TS_TIMER, timestamp timer overflow interrupt

The description WA is there too.

This is working for kinetis, so it might be applied to Vybrid too.

[christophe31]

Hi Juan,

Thanks for this answer, but as I said above, I'm not concerned by this errata!

My problem is about reading ENET ATVR register, which returns aberrant values under certain conditions.

Regards,

Christophe

[naoumgitnik]

Hello Christophe,

I eventually received reply from my colleagues having IEEE1588 experience, and they confirmed the issue.

Briefly - NOPs are indeed required; the detailed reply to be provided in the nearest future.

Thanks for catching this issue!

Sincerely, Naoum Gitnik.

juangutierrez

[christophe31]

Hello Naoum,

Thanks for this anwser. I look forward to seeing a detailed explanation...

Regards,

Christophe

[naoumgitnik]

Dear Christophe,

It looks like you got confirmation (and blessing :smileyhappy:) for using NOPs from our side, and nothing is holding you back, right?

Based on my experience, our regular procedure is to issue an Erratum for this issue, and as far as I know:

• My colleagues are planning to work on it.
• Its contents will be very similar to those for Kinetis (provided earlier - would it suffice for now?).

Sincerely yours, Naoum Gitnik.

Juan Antonio Gutierrez Rosas

[christophe31]

Hi Naoum,

In fact, I’m still waiting for a detailed solution and/or explanation, possibly from HW designer guys: which is the correct sequence ? How many NOPs ? etc.

Moreover, as far as I know, there is no similar erratum for Kinetis !? (I mean related to ATVR consistency). All the errata provided earlier were related to other IEEE1588 issues...

Regards,

Christophe

[naoumgitnik]

Dear Christophe,

My colleagues are still working through the procedure; until they come with the workaround recommendation, may you use the number of NOPs you empirically found, please?

I also understand that, according to the avionics standards, you have to have this information from us - do you have a due date for it for your project, please? - With this date, it will be easier for me to communicate with my colleagues.

Regards, Naoum Gitnik.

[christophe31]

Hi Naoum,

Thanks again for your involvement! You perfectly understand my needs! Fortunately, I’m beginning with Vybrid and the certification audit is not planned yet. So it’s not an emergency, nevertheless it’s really crucial for me to obtain this information if I want to make my project "stamped".

Best regards,

Christophe

[naoumgitnik]

Dear Christophe,

My sincere apologies for the delay.

Just to be sure - as of now, this is information is preliminary and unofficial (actually a basis for the future formal erratum):

• When doing a “Capture Timer Value” command via setting the CAPTURE bit in the Timer Control Register, i.e. ENETx_ATCR[CAPTURE] = 1, the user must wait a set amount of time before reading the captured value in the ENETx_ATVR register. 
• The amount of time to wait is equal to 6 clock cycles of the slower clock out of the IPS Bus Clock and the Ethernet Time Sampling Clock.
• The IPS Bus Clock frequency is equal to the frequency of the IPG Bus Clock and is derived from (Platform Bus or Cortex-M4 Clock)/CCM_CACRR[IPG_CLK_DIV].
• The Ethernet Time Sampling Clock is selected via CCM_CSCMR2 [ENET_TS_CLK_SEL].
• E.g., if the IPG clock is running at 66MHz and the Ethernet TS clock is running at 50MHz, then the time to wait is 6*(1/50MHz) = 120 nanoseconds.

(BTW, the 'read' of the ENETx_ATVR register will consume some cycles and could be incorporated into the wait time. If the read took, say, 100 ns, then the user would only need to wait 20ns in the above example.)

Regards, Naoum Gitnik.

[naoumgitnik]

Hello Christophe,

I discussed this issue with some other colleagues of mine - those having IEEE1588 experience (not necessarily on Vybrid, though).

I am expecting their feedback within several days.

Sincerely, Naoum Gitnik.

juangutierrez