FMAN mEMAC: in-band-status supported for 10G?

[tmoos]

Hi,

we made a T1023 based board where MAC1 is connected to an SFP+ module via XFI. The SFP+ module is connected via I²C to the SoC. We are using SerDes Protocol 0x95. The OS is OpenWRT with a kernel 4.14.137.

The first thing I noticed is that the DPAA driver does not support phylink. Therefore, it cannot talk to the SFP+ module via I²C (only MDIO is supported).

Therefore, I want to operate MAC1 in phy-less mode and added this to my DTS file:

eth3: ethernet@e0000 {
    managed = "in-band-status";
   /*sleep = <&rcpm 0x80000000>;*/
};

U-Boot adds some properties, e.g. phy-connection-type=xgmii, compatible="fsl,fman-memac" and pcsphy-handle=<0x1b>. The boot messages say:

[   13.462738] fsl_mac ffe4e4000.ethernet: FMan MEMAC
[   13.467570] fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:00:5b:04:d2:17
[   13.482199] ------------[ cut here ]------------
[   13.493914] WARNING: CPU: 0 PID: 1 at drivers/net/phy/swphy.c:135 swphy_read_reg+0x120/0x1b0
[   13.502644] ---[ end trace 6db981a9a9b68335 ]---
[   13.507305] ------------[ cut here ]------------
[   13.519007] WARNING: CPU: 0 PID: 1 at drivers/net/phy/swphy.c:135 swphy_read_reg+0x120/0x1b0
[   13.527682] ---[ end trace 6db981a9a9b68336 ]---
[   13.566896] fsl_mac ffe4e0000.ethernet: FMan MEMAC
[   13.571713] fsl_mac ffe4e0000.ethernet: FMan MAC address: 00:00:5b:00:00:12
[   13.581331] fsl_mac ffe4e4000.ethernet eth0: Probed interface eth0
[   13.589992] fsl_mac ffe4e0000.ethernet eth1: Probed interface eth1

The warning in drivers/net/phy/swphy.c is due to an illegal speed set on the (virtual) phy. After boot, the interface eth1 is present, but unusable.

I also tried the fixed-link property in the DTS, but this seems to support only 10/100/1000M speeds. But I have 10G. So this didn't work, either.

My questions are:

1. Is my observation correct that DPAA does not support phylink (and hence talking to SFP+)?
2. Does  managed="in-band-status" support 10G XFI connections?
3. Does fixed-link support 10G XFI connections?

Kind regards, Tanjeff

[tmoos · Accepted solution]

Meanwhile I managed to solve the problem.

First, I had a problem on my board: the polarity of the XFI transmit signal was swapped, therefore no data could be sent from MAC to SFP+-Module. I corrected this and now the XFI connection works.

To answer my original questions:

1. Yes, DPAA does not support phylink (and hence talking to SFP+).
2. No, managed="in-band-status" does not work for 10G XFI connections.
3. Yes, fixed-link works for 10G XFI connections. Note that it is impossible to configure 10G speed, but with 1G speed configured, it works.

The solution to my problem is:

In the DTS file I configured the MAC with <fixed-link> and set it to full-duplex and 1000M speed (it is not possible to set 10G speed). The speed is ignored by the driver, and the interface is running. Of course, I don't get a valid link status this way. Also, the ethtool reports wrong data (e.g. it reports 10M speed for some reason). The solution is not perfect, but it suffices for me.

[bpe]

Is my observation correct that DPAA does not support phylink (and
hence talking to SFP+)?

[Platon] DPAA driver utilizes Linux native PHY abstraction layer
as described here:

https://source.codeaurora.org/external/qoriq/qoriq-components/linux/tree/Documentation/networking/ph...

Does  managed="in-band-status" support 10G XFI connections?

[Platon] In-band status is not applicable to XFI. It's a parallel interface signalling technique.

The on-chip SerDes expects a  specific autonegotiation response from the locally
connected device to complete the link setup. You need to check
what (if at all) your PHY device sends the local MAC when it establishes
the local link.

Does fixed-link support 10G XFI connections?

[Platon] No. But even if it did, fixed-link only skips the external
PHY initialization and does nothing to T1023 SerDes which still expects
an autonegotiation response from the locally connected device. So
understanding what your PHY sends to the SerDes during link establishment phase

is the key.

[tmoos]

I managed to read the PCS PHY registers now. I use the MDIO bus of MAC1, address 0. Remember that I'm using SerDes protocol 0x95, i.e. MAC1 is configured for XFI. The register values are:

• 0x3.0x20 = 0x1001 (MDIO_XFI_PCS_10GR_SR1)
  • Bit RX_LNK_STAT is 1 (link is up)
  • PCS_BLK_LK is 1 (PCS locked to receive blocks)
• 0x3.0x21 = 0b10bb_bbbb_eeee_eeee (MDIO_XFI_PCS_10GR_SR2)
  
  • L_BLK_LK is 1 (PCS has block lock)
  • LH_BER is 0 (PCS has not reported a high BER)
  • The counters BER (bb_bbbb) and  ERR_BLKS (eeee_eeee) are constantly counting.
• 0x7.0x0 = 0x0 (MDIO_XF_AN_CR)
  • AN_EN = 0: Auto-negotiation is off.

If I enable MDIO_XF_AN_CR.AN_EN, then MDIO_XF_AN_SR.AN_COMP remains 0, i.e. auto-negotiation does not work.

To be honest, I'm using the wrong cables (single mode fiber instead of multi mode fiber), so the connection is not entirely reliable. I'm waiting for the correct cables to fix that problem. Nevertheless, I have an XFI link.

My DTS file states:

    eth3: ethernet@e0000 {
    };

My U-Boot adds some info, e.g. compatible=fsl,fman-memac, a pcsphy-handle entry, phy-connection-type=xgmii and some more.

Now I tried to bring up the interface:

    # ifconfig eth1 192.168.100.1
    [ 2531.886291] fsl_mac ffe4e0000.ethernet eth1: init_phy() failed
    ifconfig: SIOCSIFFLAGS: No such device

It seems that the DPAA driver is missing a MAC. But you said that neither "in-band-status" nor "fixed-link" are appropriate for XFI.

Do you have some advice how to bring-up the MAC in phy-less mode (since DPAA doesn't support SFP+)?

[bpe]

What your Linux is missing, is PHY, not MAC. You can try following the ideas given in NXP AN12572. It is, however, for backplane, not for SFP+.

Have a great day,
Platon

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tmoos]

Hi Platon,

You are of course correct: the PHY is missing, not the MAC :smileyhappy:

I enabled 10GBASE-KR link training on the MAC side (MDIO_XFI_10GKR_PMD_CR.TRAIN_EN=1), but the training fails (MDIO_XFI_10GKR_PMD_SR.TRAIN_FAIL=1). I suppose that my SFP+ Module does not support 10GBASE-KR. In addition, my Linux kernel does not include the FSL_BACKPLANE driver. I believe it is not worth porting it.

Meanwhile I hacked the PHY Abstraction Layer to support 10G, but with limited success. The MAC can now receive packets, but it doesn't send. I believe that the DPAA driver still thinks that the link is down. I'm still working on that.

My impression is that nobody before me tried to use an SFP+ module with the DPAA driver, and that this configuration is not officially supported...