MPC8349 TSEC snoop

[EAlepins]

Hi,

MPC8349 Three-Speed Ethernet Controller (TSEC) has register DMACTRL[TDSEN, TBDSEN] and ATTR[RDSEN, RBDSEN] which control Rx/Tx/RxBD/TxBD data snoop enable. However, there is no description in the MPC8349 Ref. Manual Rev 1 about the behavior of these bits. I don't understand where the TSEC has a cache...

In my understanding, enabling DMACTRL[TDSEN] "Tx data snoop enable" enables the TSEC to look at transactions performed in DDR on the Tx buffers and ... what? invalidate it's Tx buffers internal cache if so?

In my system, I have enabled the core's data cache in copy-back mode on Rx and Tx descriptors and buffers. Is that correct or I should completely disable cache on the Rx/Tx buffers and RxBD/TxBD?

I am getting IEVENT[BSY] error... meaning that the TSEC sees all buffers as full.

Thanks for your help,

Étienne

[Bulat]

The TSEC does not have a cache or course. Those snoop attributes relate to the core's data cache. Note that the TSEC (and associated DMA) works directly with the memory, while fresh data (e.g. TX buffers or BDs) can reside in the cache. 'Snoop enable' eliminates this problem and provides memory coherency for TSEC operations.
So the rule  for correct data exchange is simple, if the data cache is enabled, snooping attributes on the TSEC side must be enabled.

Regards,

Bulat

[EAlepins]

Any answer?

[EAlepins]

Hi,

Ok, but is there something to do on core's side if cache is enabled? (e.g. set the M bit of a BAT or PTE to 1) Also, are both copy-back and write-through supported?

Can you give an example of what will happen for Tx buffers if snooping is enabled and they reside in the core's copy-back cache? E.g. the core writes to Tx buffers in cache, but at that moment, the TSEC tries to read from the Tx buffers in DDR. The TSEC will send a snoop command to the core which will then do what? What I don't understand is overwriting: if the core responds by flushing it's cache line, it might overwrite data written by the TSEC (e.g. core sets TxBD[R] #1 and has TxBD[R] #2 to 1; TSEC is reading RxBD #2 and sets TxBD[R] #2 to 0; if the core flushes after that point, then TxBD[R] #2 will be overwritten by a 1).

Thanks,

Étienne

[Bulat]

Nothing is needed on the core side. Both copy-back and write-through are supported.

When snooping on the TSEC side is enabled, then any access from the TSEC to the memory (no matter read or write) is checked if cache contains modified data. If yes, then TSEC's access is terminated with retry condition, then cache line with modified data is being written to the memory and invalidated, then TSEC retries and completes the access to the memory.

"Overwriting" (from your message) is not possible, i.e. the core can not overwrite data written by the TSEC due to explanation above.

Regards,

Bulat

[EAlepins]

Hi,

Ok, I now better get the behavior of a retry. However, I'm still not sure about MEMORY_COHERENCY attribute in MMU. You are saying it's not needed, but I don't understand why. Let's summarize:

- A bus master (core or TSEC) sets the gbl (global) signal alongside their transaction when they want other masters to snoop (i.e. look) at their transaction and ensure coherency.

- All masters are always snooping, i.e. listening to bus transactions. It's only that they ignore those which are not marked as global. There is no "snooping enable" neither in core nor in the TSEC.

- TSEC ATTR[RDSEN and RBDSEN] enables read/writes RX transactions done by the TSEC to be signaled as global. TSEC DMACTRL[TDSEN and TBDSEN] do the same for TX read/writes transactions. There are NOT snooping enable, they are "global" enable. Meaning that the TSEC always looks/snoops that the core's global transactions, even if these bits are disabled.

Please tell me if that understanding is correct. If it is, then there is something I don't understand: in some version of my code, the core was running in copy-back cache mode on the RxBD & TxBD and on the RX/TX buffers. Only TSEC TX snooping was enabled (DMACTRL[TDSEN and TBDSEN]). The MMU is not enabling Memory Coherency bit anywhere. The SW algorithm does the following:

1- Set TxBD.Length and make the buffer pointer point on the data buffer

2- Set TxBD.Status[R]=1. Although, since the cache is in copy-back, this write does not reach the RAM and remains in cache. Also, from my understanding, since the MMU didn't enable the Memory Coherency bit, the R write transaction is not global and hence the TSEC does not see the transaction.

3- SW waits that the transmission is over by polling TxBD.Status[R]. This is done immediately after previous step in a small loop. So I don't think there is data eviction in cache due to other data introduced in. This algorithm works (it does not timeout) and according to my above explanations, I don't understand why. Indeed, R=1 transaction is not global, hence TSEC cannot see it, hence it cannot start transmission, hence R cannot go back to 0. But it does.

Please help,

Étienne

[Bulat]

- A bus master (core or TSEC) sets the gbl (global) signal alongside their transaction when they want other masters to snoop (i.e. look) at their transaction and ensure coherency.

This is correct. However the wording "other masters" in your statement is not well defined. Snooping master is one equipped with the cache. In case of the MPC8349 it is the core only. In other words TSEC does not snoop the bus (as well as other potential masters besides the core).

- All masters are always snooping, i.e. listening to bus transactions. It's only that they ignore those which are not marked as global. There is no "snooping enable" neither in core nor in the TSEC.

No. See above concerning "All masters".
When we say "snooping enable" in terms of TSEC, we enable the transactions be snooped by snooping masters (core in our case). And yes, "snooping enable" for TSEC means  TSEC's transactions are performed with GBL signal active.

- TSEC ATTR[RDSEN and RBDSEN] enables read/writes RX transactions done by the TSEC to be signaled as global. TSEC DMACTRL[TDSEN and TBDSEN] do the same for TX read/writes transactions. There are NOT snooping enable, they are "global" enable. Meaning that the TSEC always looks/snoops that the core's global transactions, even if these bits are disabled.

Yes for the first part, No for the last statement. Again, TSEC does not look/snoop the global transactions (no matter from the core or or other bus masters) because it does not have cache.

...However, I'm still not sure about MEMORY_COHERENCY attribute in MMU...

MEMORY_COHERENCY bit should only be set if you want core's transactions to the memory be snooped by other snooping masters. Since there are no snooping masters other than the core, there is no reason to set MEMORY_COHERENCY bit in the MMU.

Regards,

Bulat

[EAlepins]

Hi,

We are almost there. Thanks for having explained that only the core snoops and hence, because MPC8349 is a single core processor, MMU "M" bit is useless.

However, I still don't understand why my transmit SW algorithm works: the modifications made to the TxBD by the core are only performed in cache (since it is in copy-back mode), but still, the TSEC sees it and starts transmission! Why? Here is my transmit code:

E_RETURN_CODE = K_ETHN_ERROR; // Default values

B_TIMEOUT = K_ETHN_FALSE;

// Update the TX Buffer descriptor:
S_TX_BD[I_TX_IDX_BUFF].I_LENGTH = I_BUFFER_LENGTH;
S_TX_BD[I_TX_IDX_BUFF].I_BUFFER_PTR = (T_ETHN_UINT32)P_DATA_BUFFER;
S_TX_BD[I_TX_IDX_BUFF].I_STATUS = S_TX_BD[I_TX_IDX_BUFF].I_STATUS | K_ETHN_NIC_TXBD_READY | K_ETHN_NIC_TXBD_LAST | K_ETHN_NIC_TXBD_PADCRC | K_ETHN_NIC_TXBD_TC | K_ETHN_NIC_TXBD_INTERRUPT;

// Launch the DMA by clearing THLT bit of TSTAT register:
ETHN_CMN_WRITE_32 (K_ETHN_NIC_TSTAT_ADDR, K_ETHN_NIC_TSTAT_THLT);

// Wait for the buffer to be transmitted:
I_START_TIME = ETHN_CMN_READ_TIME();
do
{
    I_STATUS = S_TX_BD[I_TX_IDX_BUFF].I_STATUS & K_ETHN_NIC_TXBD_READY_MASK;
    
    // Check that the data have been transmitted:
    if (I_STATUS == 0x0000)
    {
        // Check that the status of the transmission buffer is valid:
        if ((I_STATUS & K_ETHN_NIC_TXBD_OK_MASK) == K_ETHN_NIC_TXBD_CLEAR)
        {
            E_RETURN_CODE = K_ETHN_NO_ERROR;
        }
    }
    
    // Check if the timeout is elapsed:
    B_TIMEOUT = ETHN_CMN_CHECK_ELAPSED_TIME(I_START_TIME, K_ETHN_TRANSMIT_10MS_TIMEOUT);
} while ((E_RETURN_CODE != K_ETHN_NO_ERROR) && (B_TIMEOUT != K_ETHN_TRUE));

Where S_TX_BD is the transmit buffer descriptor. The K_ETHN_TRANSMIT_10MS_TIMEOUT is a 10 ms timeout which is not reached (we would have seen it because the task in which this code runs has a deadline shorter than 10 ms). DMACTRL[WOP] is configured to 0 (poll; meaning that clearing TSTAT[THLT] here seems to be only for robustness, not to trigger the transmission). K_ETHN_NIC_TXBD_READY_MASK is 0x8000. When entering the function, all descriptors R bits are 0. This function is really transmitting the message (the receiver sees it). Finally, I don't see how S_TX_BD[I_TX_IDX_BUFF] could get evicted from the cache by the other data accesses... The operating system has no preemption capability so this function is not interrupted by any other code / ISR / tasks.

Since the timeout is not reached, can you explain why the TSEC seems to see the S_TX_BD[I_TX_IDX_BUFF].i_STATUS[R]=1 inside the core's copy-back cache?

Thanks!

Étienne

[Bulat · Accepted solution]

"However, I still don't understand why my transmit SW algorithm works: the modifications made to the TxBD by the core are only performed in cache (since it is in copy-back mode)...
...
Since the timeout is not reached, can you explain why the TSEC seems to see the S_TX_BD[I_TX_IDX_BUFF].i_STATUS[R]=1 inside the core's copy-back cache?"

TSEC can not see content of the cache, it can access memory data only. However snooping mechanism is specially implemented to provide memory coherency. We seem to return to previous discussion of the memory coherency:
"When snooping on the TSEC side is enabled, then any access from the TSEC to the memory (no matter read or write) is checked if cache contains modified data. If yes, then TSEC's access is terminated with retry condition, then cache line with modified data is being written to the memory and invalidated, then TSEC retries and completes the access to the memory."
In particular this means following, if TSEC accesses TxBD and it was modified by the core in the cache, then TxBD will be written to the memory due to actions described above, and TSEC  accesses the "fresh" data prepared by the core.

Regards,
Bulat

[EAlepins]

Ah! I get it: the TSEC reads will force a cache flush.

Thanks for your support on this case!

Étienne