LS1046A SerDes XFI Query

[KHW]

Note: This topic is related to https://community.nxp.com/t5/Layerscape/LS2088A-SerDes-XFI-amp-2-5G-Query/td-p/1607421 which was closed recently. I posted these questions in that topic, but thought to create a new topic just in case. 

Hi @yipingwang , or someone else that can comment on this,

Hope you are doing well. We are using LS1046A in one of our products and using XFI interface to 10G AQR113C PHY. 

1. Could you please let me know how to check if FLOW CONTROL is enabled on MAC side for LS1046A? Is it the XFI AN Advertisement Register 0? I am not able to understand where to check for it.

2. When you mean special provisioned firmware for AQRate PHY to support speeds lower than 10G, I assume what you mean it is different than setting PHY registers for setting up PAUSE capability among other parameters, am I correct? The firmware meaning PHY will ship with it out of the box, right?

3. On AQR113C PHY we have enabled PAUSE operation for full duplex links and Asymmetric PAUSE operation for full duplex links in the Autonegotiation Advertisement Register: Address 7.10. Do you know if there's anything else to rate adaptation on PHY side that we need to set in the form of registers or is it the special firmware of PHY that is a must?

4. Importantly, if MAC flow control/pause frame is not enabled, what kind of problems can be expected? Will it cause lower throughput or will the link not work altogether or will it cause auto-negotiation or link downgrade issues? 

Any advice would be much appreciated.

Thank you.

[stadium_aquino]

1. ethtool --show-pause eth0 (or eth1 etc.). Pause reception is enabled in the mEMAC COMMAND_CONFIG register when the PAUSE_IGN bit is cleared. Pause transmission is controlled by the Pause Quanta registers.

2. The 1046 does not support USXGMII, which is the conventional way to support 10G and e.g. 1G speeds on the same interface. To work around this, NXP added an Aquantia phy. This phy always uses 10GBase-R (XFI) when communicating with the 1046. When you plug in a slower-speed interface (such as 1000Base-T), it will send pause frames back to the 1046 to ensure that the line rate is reduced to 1G.

3. Generally, you must request firmware from Marvell which will be configured for a particular rate adaptation configuration. You can check that it is enabled by inspecting the Global System Configuration registers.

4. You will have a lot of dropped packets.

[KHW]

Hi stadium_aquino,

Thank you so much for your reply.

1. When I run the ethtool command you mentioned on the Aquantia PHY, I get the following:

node:~# ethtool --show-pause nic2
Pause parameters for nic2:
Autonegotiate: on
RX: on
TX: on
RX negotiated: off
TX negotiated: off

I am not sure what "off" means here and if it is expected. Should it be on for RX negotiated and TX negotiated as well?

2. Yes, I understand that 1046A doesn't support USXGMII, and hence I want to make sure things are set up correctly from both 1046A (MAC) side and AQR113C (PHY) side. In general, do all Ethernet PHYs sent PAUSE frames by default? For Aquantia, I believe Autonegotiation Advertisement Register: Address 7.10 is the register where we enable the same. Am I correct? Or is there anything else to it?

3. Okay, I will check what I get and let you know. I am not sure if we have the special firmware from Aquantia, so going to check with them as well.

4. I see, thanks. Are occasional link downgrade issues associated with it as well?

[stadium_aquino]

> I am not sure what "off" means here and if it is expected. Should it be on for RX negotiated and TX negotiated as well?

"off" means that flow control via pause frames is disabled.

That said, be careful here, since this only reflects the pause frame configuration and not the actual configured pause. In kernels 6.1 and newer, the DPAA driver uses phylink, which will automatically configure pause frame if necessary. On those kernels, if you have a 1G link plugged in you should see a message like "Link is Up - 1Gbps/Full - flow control rx". The "flow control" shows the actual configured pause frame, even if it is disabled in ethtool. If you are using an older kernel, ethtool reflects the actual pause frame values. You should disable autonegotiation (since the far end is not necessarily using pause frames) and manually enable pause frame reception.

> In general, do all Ethernet PHYs sent PAUSE frames by default?

AFAIK this is something only done by Aquantia.

> For Aquantia, I believe Autonegotiation Advertisement Register: Address 7.10 is the register where we enable the same. Am I correct? Or is there anything else to it?

No, you have to look at the "Global System Configuration" registers 1E.31B etc.

Actually, I believe by default aquantia phys ignore the standard advertisement registers so MACs which are unaware of the rate adaptation will continue to work (as otherwise they would disable speeds under 10G). So there is some bit (I forget which) you need to clear.

> Are occasional link downgrade issues associated with it as well?

You shouldn't see that.

[KHW]

Looking forward to some advice here. Thanks!

[yipingwang]

I discussed your questions with the AE team, please refer to the following update from them.

[1]
It was a bit on eTSEC based devices. I'm not sure for DPAAx. Let me inquire.

[2]
Correct. You need firmware for the PHY. Let me inquire about the instructions to load and what is available

[3]
I'm not certain how to enable/diable AQRate on the AQR PHY. I always believed it was always on. I can inquire with the SW driver team.

[4]
I can only state "boundedly undefined" behavior. Since we are XFI, we will always send data at 10Gbps. If the PHY negotiates a 1Gbps link side partner, it would want to avoid RX overrun. Hence, it sends PAUSE frames to the MAC (system side) to slow things down.

[KHW]

Hi yipingwang,

Thank you so much for the reply. Yes, please do provide any further information on [1], [2] and [3] if possible. It would be much appreciated.

[1] Looks like it is the bit "stadium_aquino" mentioned in their comments. I could find it in the LS1046A DPAA reference manual.

[3] Okay, but I don't wanted to enable/disabled AQrate behavior. And as you said, it is probably a fixed thing. I mainly wanted to confirm on the registers that we need to set the make enable PAUSE operation from PHY side.

[yipingwang]

[1]
In eTSEC, I believe there were just config register bits to enable PAUSE flow control.
How do we enable/disable on DPAAx and confirm the setting?

I imagine 'ethtool' in Linux would be a convenient way.

In memac the bits for PAUSE are PAUSE_IGN and RX_SECTION_EMPTY.

The tool used is ethtool:

ethtool -A eth1 autoneg off rx off tx off

[2]
Do you have any instructions on how to obtain and program Aquantia PHY firmware on our platforms? Do you know if there is a way to confirm what features have been enabled in a firmware release? e.g., USXGMII, XFI, SGMII support.

Check here: https://github.com/nxp-qoriq/qoriq-firmware-aquantia

[3]

Do you know if there is any extra step to enable the AQRate feature on AQ PHYs or that is the entire point of the PHY? It is always enabled?

This is a response to an older ticket:

“For the configuration in which the external phy (AQR) is connected to our MAC through XFI interface, the supported speed at MAC side is 10Gbps.(and this cannot be changed)

Our earlier code (I say earlier referring to the SDK 1703 and below, because the snippet of code updated by the customer seems to be from the SDK) made the following assumption:

-the interface type XFI uses a fixed speed that is 10Gbps. However, the implementation didn’t take into account that some phys -e.g AQR might support rate adaptation (through pause frames generated from the phy) such that at line side one could connect to a peer that had a 1Gbps phy.

According to the implementation, our driver removed the mode values advertised by the PHY to the system side, causing the failure of auto negotiation with a link partner lower that 10Gbps (e.g 1Gbps)

The WR proposed by SWI is ok in my opinion because what their change does is that it does not remove the modes advertised by the AQR phy.

In the upstream this fix is already present.

Now I will make an adjustment: I said above that “earlier code” had some incorrect assumptions. The same statement can be applied for the LSDK code: it does not support a phy with rate adaptation.

It means that only in upstream we can find something let’s say valid, for this particular case.

To understand my comments, see these patches:

https://github.com/torvalds/linux/commit/73a21fa817f0cc8022dc6226250a86bca727a56d

https://lkml.org/lkml/2020/2/10/633

“

[4]
I shared this with the CAS member.

" I can only state "boundedly undefined" behavior. Since we are XFI, we will always send data at 10Gbps. If the PHY negotiates a 1Gbps link side partner, it would want to avoid RX overrun. Hence, it sends PAUSE frames to the MAC (system side) to slow things down."

[KHW]

Hi yipingwang,

Thank you so much for your detailed comments and sorry my delayed reply.

1. As per my another comment in the topic, checking what you mentioned shows the following for the AQR PHY. Do the "off" fields signify some kind of a problem? When I read this for the PHY at the other side of the link, it was "on" for all the fields.

node:~# ethtool --show-pause nic2
Pause parameters for nic2:
Autonegotiate: on
RX: on
TX: on
RX negotiated: off
TX negotiated: off

2. Thanks, I will take a look. Regarding the firmware, I checked with the software team recently and understood we are already using a firmware from Aquantia for AQR113C. Although upon checking the firmware registers of PHY, I realized that we are using an older firmware. So, I have requested Aquantia for the new firmware to see if that helps. 

3. Sorry, I did not understand it properly. While I try to understand the information, it looks like they are talking about NXP Linux SDK and some fix of this issue in that? If so, and if possible, could you please help me to understand quickly what's the latest SDK version and if the latest SDK has some fix regarding the issue?

4. Okay, thanks. 

[yipingwang]

You could refer to the latest LSDK release YP 4.1–lf-6.1.1.

https://github.com/nxp-qoriq/yocto-sdk/blob/langdale/readme

[KHW]

Any suggestion would be much appreciated.