Vybrid ProcessorsのトピックRe: IEEE1588 time-stamping issue

IEEE1588 time-stamping issue

christophe31 — Wed, 08 Jan 2014 16:43:24 GMT

Hi,

I'm trying to use IEEE1588 time-stamping capability on a Vybrid Tower kit TWR-VF65GS10.

I'm using the 50MHz external clock to clock the ENET module, thus I use an increment value (ENET_ATINC) of 20 (0x14). That way I can directly read values expressed in nano-second.

It seems to work correctly, but when I use the timestamp counter (ENET_ATVR) to measure some code performance I get a strange behaviour :

I make 2 succesive readings of ENET_ATVR (which first needs to set the CAPTURE flag of ENET_ATCR register), then I compare the 2 values.

On a very small piece of code, most of the time, the difference is an almost constant value (560 or 580 ns), but sometimes (?!) values are not consistant at all. I can observe various behaviors :

- Increment is not consistent :

t1 = 0x7AD650B4

t2 = 0x7AD6548C

diff. = 984

==> the difference is greater that 580, but primarily it is not a multiple of 20 ?!

- The difference is negative :

t1 = 0x0862D280

t2 = 0x0862D0C4

diff. = -444

Here, the expected t2 value is 0x0862D4C4 (seems to be a defect on bit 10), but this is not always so trivial e.g. :

t1 = 0x60EB193C

t2 = 0x60EB135C

I also note another strange behavior : I launch my SW normally, then I break and disable time-stamping (I set ENET_ATCR=0) with my debugger (Lauterbach Trace32).

Then I set the CAPTURE flag many times to acquire some ATVR values. What I get is 2 different values separated by the increment value. Values are not coming in a regular flow, e.g. :

0x2EAECC10

0x2EAECC24

0x2EAECC10

0x2EAECC24

...

It seems that data come from a dual buffer. Could it be the origin of the problem ?

Here is my IEEE1588 registers configuration :

ATCR 00000001

ATOFF 00000000

ATPER 80000000

ATCOR 00000000

ATINC 00001414

I experiment this on both ENET0 and ENET1

Did someone face such an issue ?

Thanks for any help

Regards

Christophe

Re: IEEE1588 time-stamping issue

naoumgitnik — Sun, 12 Jan 2014 21:48:31 GMT

Dear Christophe,

We are aware of that this feature has an issue with the way it is implemented on the Tower kit.

for those interested in it, we recommend to use the MII interface instead of the existing RMII - the Tower PHY module does support both of them, as far as i know.

Please, expect more details within several days.

Regards, Naoum Gitnik.

Re: IEEE1588 time-stamping issue

naoumgitnik — Sun, 12 Jan 2014 21:52:51 GMT

Christophe,

Regarding "... 50MHz external clock to clock the ENET module...." - do you mean the Ethernet PHY clock is not generated by our Vybrid?

/Naoum.

Re: IEEE1588 time-stamping issue

christophe31 — Mon, 13 Jan 2014 08:02:15 GMT

Hi Naoum,

Thanks for these answers

for those interested in it, we recommend to use the MII interface instead of the existing RMII - the Tower PHY module does support both of them, as far as i know.

Hum, maybe this is the point, but I need 2 network interfaces. I'm using the 2 ENETs with a TWR-SER2 module. Thus, I have no choice than using RMII. Am I wrong?

Moreover, I don't understand what it changes... Time-stamp seems to be an ENET's sub-component. Could you explain what impact has MII/RMII on it?

Regarding "... 50MHz external clock to clock the ENET module...." - do you mean the Ethernet PHY clock is not generated by our Vybrid?

Yes. My ENET clock is generated by the TWR-SER2 board, which clocks both PHY and Vybrid's ENET/MAC (via CLKOUT0). Do you think it would change something if I use the Vybrid internal clock?

Thanks again for your help !

Best regards,

Christophe

Re: IEEE1588 time-stamping issue

jiri-b36968 — Mon, 13 Jan 2014 12:34:54 GMT

Hi Christophe,

You are right. If you need 2 Ethernet interfaces you need to use RMII due to share pins. When RMII is used we use external 50MHz clock from TWR-SER2 tower module. We have also observed some issues on IEEE1588 when RMII is used. Not sure if it help when clock is generated on Vybrid - unfortunately clockout for RMII is not suported on Vybrid tower module. We can check it once rev.H with MII interface and clockout is available.

/Jiri

Re: Re: IEEE1588 time-stamping issue

naoumgitnik — Tue, 14 Jan 2014 00:30:14 GMT

Dear Christophe,

Regarding "I need 2 network interfaces. I'm using the 2 ENETs with a TWR-SER2 module. Thus, I have no choice than using RMII. Am I wrong?" - as per '8.7 Communication interfaces', page 575 of 'Vybrid Reference Manual, Rev. 5, 07/2013' - "MII mode is supported for MAC0 only and when MII mode for MAC0 is enabled, RMII of MAC1 cannot be used." - You are NOT wrong.

As I learned, it is somehow related to the RMII signal jitter. No such problem with MII.

More details later this week.

Based on the Vybrid datasheet, below are relevant MII connections between it J17A (elevator) connector to be added onto the Tower board (planned to be implemented in the next board revision):

PTA6 – MII_TXCLK to J17A – B14

PTA21 – MII_RXCLK to J17A – A15

PTD17 – MII_TXERR to J17A – B16

PTD18 – MII_TX2 to J17A – B18

PTD19 – MII_TX3 to J17A – B17

PTD20 – MII_COL to J17A – B12

PTD21 – MII_CRS to J17A – A12

PTD22 – MII_RX2 to J17A – A18

PTD23 – MII_RX3 to J17A – A17

Regards, Naoum Gitnik.

Re: IEEE1588 time-stamping issue

christophe31 — Tue, 14 Jan 2014 16:12:55 GMT

Naoum, Jiri,

Thanks for your involvement !

Maybe a workaround here :

ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

__asm("DSB");

__asm("nop");

ts_counter_ns = ENET0->ATVR;

I did not make a lot of tests but it seems much better this way.

DSB makes the CPU (Cortex M4 here) wait until the capture flag is really set.

At least 4 NOP must be added in order to obtain a coherent value.

So I guess there are 2 problems :

- in some cases, the ATCR write might be treated after the ATVR read (I'm not sure if Cortex-M4/NIC301 architecture allows that, but it could explain some negative results). This is prevented with DSB instruction.

- the ATVR register is not updated synchronously. Additional NOPs allow to wait until all bits are properly set.

BTW, this is not perfect, because it makes me wait a "long" time (approx. 21 CPU cycles) for "nothing", which is not ideal for constrained real-time application, so I'm still expecting a better solution... if possible!

Regards,

Christophe

Re: Re: IEEE1588 time-stamping issue

naoumgitnik — Fri, 17 Jan 2014 00:24:54 GMT

Dear Christophe,

I am not sure if workarounds help, based on the below information from our people who tried to deal with this feature. Please, find some excerpts below:

* The issue is confirmed on both Vybrid and Kinetis Tower Modules (with RMII connection).

* The jitter observed with RMII interface is actually a known matter, i.e. this behavior is also found in other Ethernet controllers, e.g. the INTEL i210 datasheet (http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf). Table 7-62 on page 340 states that the jitter on 100-Mbps RX frame time-stamping is 80ns (2228 - 2148 = 80), and this is exactly what we have observed in our Ethernet Controller.

The jitter in RMII, which is bigger (in general) than the MII, is mainly due to the nature of the interface itself.

In MII, data and the clock are effectively forwarded to the MAC (i.e. they are source-synchronous), but that’s not the case for RMII; therefore RMII is understood to introduce greater jitter.

* ... MII (RGMII/GMII) is preferred as far as 1588 time-stamp concerns.

* The TWR-Vybrid performance ... below +/-130ns (have the same 80ns jitter as Kinetis k60).

You can probably find more speculations about this on the Internet.

Regards, Naoum Gitnik.

==========================

Correction: what is called "jitter" in the above reply is rather "latency".

Sorry - it took me additional thinking to realize that the term used was quite incorrect (signal jitter is something different).

Re: IEEE1588 time-stamping issue

christophe31 — Fri, 17 Jan 2014 13:29:45 GMT

Hi Naoum,

Really interesting, I have to take care of that in my design, I can deal with a known jitter, but I fear that we don’t talk about the same thing!

For now, I don’t talk about reading the time-stamp of incoming frames. I just compare 2 values of ATVR, which is supposed (as far as I understand) to be representative of an internal time-stamp counter always increasing (up to its maximum value of course).

Or maybe I totally misunderstand: do you suggest that ATVR could be altered by incoming frames?

To be more specific, regarding my first post, my source code simply looks like:

ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

t1 = ENET0->ATVR;

…some code (to measure performance of)…

ENET0->ATCR |= ENET_ATCR_CAPTURE_MASK;

t2 = ENET0->ATVR;

diff = t2 - t1;

where t1 and t2 values are not consistent, except with additional DSB/NOP as said above. This is the point!

Of course, this is temporary code, and I also use ATVR to compare with time-stamp of incoming frames. But at first I ‘d like to be confident with ATVR content…

One more question: could you tell me where does the last excerpt come from ? (The TWR-Vybrid performance ... below +/-130ns (have the same 80ns jitter as Kinetis k60).). If possible I'd like to have the full background.

Thanks and regards,

Christophe

Re: Re: IEEE1588 time-stamping issue

juangutierrez — Fri, 17 Jan 2014 18:56:05 GMT

By googling I found and errata for Kinetis processor

e2579: ENET: No support for IEEE 1588 timestamp timer overflow interrupt

Errata type: Errata

Description: The TS_TIMER interrupt signal is not connected to the NVIC and will not generate an interrupt

event. This interrupt is set when the 1588 counter overflows.

Workaround: Use one of the Timer Compare Capture Registers, TCCRs, to manually force a counter

overflow event in order to trigger a counter overflow interrupt.

Maybe something similar is happening here:

For MQX code, the WA is already implemented

You can check the code in mqx/source/io/enet/macnet/macnet_1588.c at function MACNET_ptp_start (Around line 20 below)

uint_32 MACNET_ptp_start(MACNET_PTP_PRIVATE *priv, boolean master_mode)
{
        MACNET_PTP_PRIVATE *fpp = priv;


#if defined(BSP_TWR_K60N512) || defined(BSP_TWR_K60D100M)
    /* Select the 1588 timer source clock - EXTAL clock */
    OSC_CR |= OSC_CR_ERCLKEN_MASK;
    SIM_SOPT2 = ((SIM_SOPT2) & ~SIM_SOPT2_TIMESRC_MASK) | SIM_SOPT2_TIMESRC(0x2);
#endif


        /* Enable module for starting Tmr Clock */
    fpp->MACNET_PTR->ATCR = ENET_ATCR_RESTART_MASK;
    fpp->MACNET_PTR->ATINC = MACNET_1588_CLOCK_INC;
    fpp->MACNET_PTR->ATPER = MACNET_1588_ATPER_VALUE;


#if (MK60_REV_1_0 || MK60_REV_1_1 || MK60_REV_1_2)
    /* Workaround for e2579: ENET: No support for IEEE 1588, TS_TIMER, timestamp timer overflow interrupt */
    fpp->MACNET_PTR->CHANNEL[MACNET_PTP_TIMER].TCCR = MACNET_1588_ATPER_VALUE - MACNET_1588_CLOCK_INC;
    fpp->MACNET_PTR->CHANNEL[MACNET_PTP_TIMER].TCSR = (((0x5)<<ENET_TCSR_TMODE_SHIFT)&ENET_TCSR_TMODE_MASK) |
                                                      ENET_TCSR_TIE_MASK;
    fpp->MACNET_PTR->CHANNEL[MACNET_PTP_TIMER].TCCR = MACNET_1588_ATPER_VALUE - MACNET_1588_CLOCK_INC;
#endif


    /* Period PIN & EVT bits must be set for rollover detect */
    fpp->MACNET_PTR->ATCR = ENET_ATCR_PEREN_MASK| ENET_ATCR_PINPER_MASK;


    /* Set the slave mode in case the MACNET is not handling the 1588timer */
    if(!master_mode)
            fpp->MACNET_PTR->ATCR |= ENET_ATCR_SLAVE_MASK;


    /* Start counter */
    fpp->MACNET_PTR->ATCR |= ENET_ATCR_EN_MASK;
    return 0;
}

Re: Re: IEEE1588 time-stamping issue

naoumgitnik — Fri, 17 Jan 2014 20:01:27 GMT

Dear Christophe,

That's a good point the Juan mentioned Kinetis - I found several threads it might make sense for you to look into (sorry, I am not a SW person, thus cannot judge how much they may help in this case):

2.
Some details of the test using 2 different Freescale's boards, which are not Vybrid-based (cannot really comment on the below text - not an IEEE 1558 expert):

"I’m not running any 1588 software stack, simply set one board up to transmit 500 back-to-back frames by preparing 500 TxBDs and the second board receive all 500 frames by preparing 500 RxBDs.

... at the end of the test, I look at the differences in deltas of consecutive timestamps in the buffer descriptors to determine latency for each pair of frames, i.e. Latency = | (T_n – T_n-1) – (R_n – R_n-1)|.

Since the transmit frames are all back-to-back, I never saw any latency in the TxBD timestamps, i.e. T_n –T_n-1 was always the same for each test. Therefore, the latency values below represent the latency in the receive time-stamping:

1. 2-nd board RMII (Tx) -> 1-st board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

2. 1-st board RMII (Tx) -> 2-nd board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

3. 2-nd board MII (Tx) -> 1-st board RMII (Rx) – 80ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

4. 1-st board RMII (Tx) -> 2-nd board MII (Rx) – 20ns latency (IEEE 1588 timestamp clock source equals 50 MHz),

5. 1-st board RMII (Tx) -> 2-nd board MII (Rx) – 10ns latency (IEEE 1588 timestamp clock source equals 100 MHz).

These results line up with what I would expect:

For experiments 1-3, we see the 80ns latency when using the RMII as the receiver.
For experiments 4 and 5, we see a reduced latency when MII is used as the receiver.

It must be noted that additional latency may be seen as a result of the MAC detecting the SFD in the MII clock domain and capturing the timestamp in the IEEE 1588 clock domain.

If two these clocks are from different sources, I expect additional latency equal to the period of the IEEE1588 timestamp clock:

For experiments 1-3, when the receiver is in RMII mode, the 50 MHz crystal on the serial board is driving the XTAL pin, is the source of the RMII clock, and used to generate the 1588 timestamp clock. So, those two clock sources are the same, so no additional latency.
For experiments 4 and 5 when the receiver is in MII mode the PHY is driving the MII Rx clock, which is recovered from the RX data and is essentially the Tx clock from the transmitter board and the crystal on the serial board is driving the XTAL pin, which is used to generate the 1588 timestamp clock. So in this case the clock sources are different, so we expect additional latency, which we see.

... A simple message ... is there will be a possible latency of 80ns in 100Mb/s operation (and 800ns in 10 Mb/s) operation due to Rx to Tx clock synchronization.

For RMII, the Rx clock is given to the PHY, so synchronization has to occur between the Tx and Rx clock fairly often.
For MII, the Rx clock is driven from the PHY and can be recovered from the data, so can stay synchronized with the Tx clock."

Regards, Naoum Gitnik.

Re: IEEE1588 time-stamping issue

christophe31 — Tue, 21 Jan 2014 08:11:29 GMT

Hi Juan,

Thanks for your help.

I'm not concerned by this errata, but thanks to you, I looked at MQX and I found another undocumented way to access ATVR. They set the capture flag twice :

macnet_ptr->ATCR |= ENET_ATCR_CAPTURE_MASK;

curr_time->NSEC = macnet_ptr->ATVR;

I guess this is another way to wait for ATVR counter to be ready...

I tested it, it takes ~100ns more than my workaround, but maybe it is an "approved" solution ?

Regards,

Christophe

Re: IEEE1588 time-stamping issue

christophe31 — Tue, 21 Jan 2014 08:21:50 GMT

Hi Naoum,

Thanks for these new information, they are of great interest for my work.

But there is still nothing related to the ATVR reading error.

However, thanks to Juan, I saw I’m not the only one having problem with that!

Do you think that MQX’s workaround is an official and/or an approved solution?

Regards,

Christophe

Edit : after re-reading CM4 ref. manual, I think that the second writing of the capture flag forces to wait until the first one is effective. This is no more than DSB instruction does. Thus QMX solution does not seem better than mine finally :-(

Re: Re: IEEE1588 time-stamping issue

naoumgitnik — Tue, 21 Jan 2014 17:31:35 GMT

Dear Christophe,

Everything posted on our official web site is an "official and/or an approved solution".

The Community forum materials, though, cannot be considered as such; if it works or not is solely based on the specific thread contents.

Regards, Naoum Gitnik.

Re: IEEE1588 time-stamping issue

christophe31 — Fri, 31 Jan 2014 14:04:56 GMT

Hi Naoum,

The problem is that my software is destined to avionics, and I have to justify every piece of code in the eyes of certification authorities. Thus I can hardly explain why I insert 4 NOPs if I don’t have an application note, or an errata, or an answer from HW designer, because the certifying officer will ask :

“Why only 4 NOPs? Are you sure there is no situation where 5 NOPs would be needed?...”

Even though there’s a workaround in MQX, it is not clear whether the solution will be approved or not…

Regards,

Christophe

Re: IEEE1588 time-stamping issue

juangutierrez — Fri, 31 Jan 2014 18:19:31 GMT

Well, here is the errata document for kinetis

http://www.freescale.com/docs/pcn_attachments/15543_KINETIS_8N30D_prelim2.pdf

Look for: e2579: ENET: No support for IEEE 1588, TS_TIMER, timestamp timer overflow interrupt

The description WA is there too.

This is working for kinetis, so it might be applied to Vybrid too.

Re: IEEE1588 time-stamping issue

christophe31 — Tue, 04 Feb 2014 07:29:50 GMT

Hi Juan,

Thanks for this answer, but as I said above, I'm not concerned by this errata!

My problem is about reading ENET ATVR register, which returns aberrant values under certain conditions.

Regards,

Christophe

Re: IEEE1588 time-stamping issue

naoumgitnik — Tue, 04 Feb 2014 23:19:30 GMT

Hello Christophe,

I discussed this issue with some other colleagues of mine - those having IEEE1588 experience (not necessarily on Vybrid, though).

I am expecting their feedback within several days.

Sincerely, Naoum Gitnik.

juangutierrez

Re: Re: IEEE1588 time-stamping issue

naoumgitnik — Wed, 12 Feb 2014 00:51:56 GMT

Hello Christophe,

I eventually received reply from my colleagues having IEEE1588 experience, and they confirmed the issue.

Briefly - NOPs are indeed required; the detailed reply to be provided in the nearest future.

Thanks for catching this issue!

Sincerely, Naoum Gitnik.

juangutierrez

Re: IEEE1588 time-stamping issue

christophe31 — Thu, 13 Feb 2014 08:33:34 GMT

Hello Naoum,

Thanks for this anwser. I look forward to seeing a detailed explanation...

Regards,

Christophe