LS1027A with support to 1.25G SFP through SGMII

[igor_franco]

Hello,

I am using an LS1027A processor in a custom board. The SerDes protocol is configured as 0x9999 (all lanes as SGMII 1G through TSN Switch). Both Lane A and B are directly connected to hot-pluggable SFP devices. I have enabled the SFP Cage in the Kernel PHY drivers and set the following in my DTS (only the first SFP as an example):

\{

    sfp_neta: sfp-neta {
           compatible = "sff,sfp";
           maximum-power-milliwatt = <2000>;
            los-gpios = <&gpio2 29 GPIO_ACTIVE_HIGH>;
            mod-def0-gpios = <&gpio2 30 GPIO_ACTIVE_LOW>;
            tx-fault-gpios = <&gpio2 28 GPIO_ACTIVE_HIGH>;
            tx-disable-gpios = <&gpio3 14 GPIO_ACTIVE_HIGH>;
     };

};

&mscc_felix_port0 {
    status = "okay";
    phy-mode = "sgmii";
    label = "sfp1";
    sfp = <&sfp_neta>;
    fixed-link {
           speed = <1000>;
           full-duplex;
     };
};

However, when the Linux boot, I got the following error:

[ 0.423890] libphy: SFP I2C Bus: probed
[ 0.423955] sfp sfp-neta: Host maximum power 2.0W

[ 1.230941] sfp sfp-neta: module OEM RSMP851GL-05D rev 1 sn PQH429J dc 201206
[ 1.244952] mscc_felix 0000:00:00.5 sfp1: validation with support 00,00000000,00006440 failed: -22

I am using the Kernel LSDK-20.12-V5.4-RT. How can I solve this problem? Or the TSN Switch is not able to communicate with SFP cages?

Thank you in advance,

Igor

[igor_franco]

Hello yipingwang,

Thank you for the answer.

I must have erased accidentally my i2c-bus, find below the fixed DTS for the SFP:

\{

    sfp_neta: sfp-neta {
            compatible = "sff,sfp";
            maximum-power-milliwatt = <2000>;

            i2c-bus <&i2c0>;
            los-gpios = <&gpio2 29 GPIO_ACTIVE_HIGH>;
            mod-def0-gpios = <&gpio2 30 GPIO_ACTIVE_LOW>;
            tx-fault-gpios = <&gpio2 28 GPIO_ACTIVE_HIGH>;
            tx-disable-gpios = <&gpio3 14 GPIO_ACTIVE_HIGH>;
     };

};

And regarding the debug on the sfp.c.

I have already followed some functions to understand what the validation does.

The SFP cage I have tested shows information through the I2C that it supports a 1000Base-SX, where the sfp-bus.c (sfp_parse_support) set the ethtool bit 1000Base-X.

However, looking in the felix.c function felix_phylink_validate, which is the other side of the validation, the ethtool bit for the 1000Base-X is not set.

I have tested forcing the 1000Base-X in the felix function with no success.

Also, I have forced the sfp-bus to support a 1000Base-T Full with no success too.

The error message that I have highlighted in the first message disappeared in both tests.

But my interface was kept in LOWERLAYERDOWN even with all cables and cages correctly connected.

Does the TSN Switch support a 1000Base-X communication through SGMII?

Where can I find documentation about the supported links for the switch?

What else could cause the LOWERLAYERDOWN?

Best regards,

Igor

[yipingwang]

Does the TSN Switch support a 1000Base-X communication through SGMII?

Yes, it should be possible. Please check if the SGMII electrical specs as specified in datasheet are met by the SFP device.

For general reference you may refer an old app note AN3869

[igor_franco]

Hello yipingwang,

It does not seem a problem regarding electrical specs. 

I said that because if I remove the sfp = <&sfp_neta> parameter from the TSN Switch port DTS node, the SFP interface appears in Linux and it works properly.

I can perform a ping without any problem.

Do you have any other idea of any configuration that could be wrong?

Or else, do you have the TSN Switch datasheet to understand better how it works?

Best regards,

Igor Franco

[yipingwang]

From our expert:

If you want 1000base-x you shouldn't have SGMII in dts I think.

It  is not enough to just set a bit. You said that you want 1000base-X, at least this is I understand reading the response.

If so then you need to set the interface mode to 1000base-x to configure the PCS in 1000base-x mode.

Then check that the lynx pcs goes through: https://elixir.bootlin.com/linux/latest/source/drivers/net/pcs/pcs-lynx.c#L192

The idea is to trace that the PCS is set to the correct SFP mode:1000base-x.

I also believe that phy-mode property is useless. You do not have any PHY between the path:  MAC ----SERDES ----SFP.

Remove everything related to the PHY.

[igor_franco]

Hello yipingwang,

About removing the PHY configuration from DTS. I have removed the phy-mode and fixed-link parameters from the mscc_felix_port node. It has only the status, label, and sfp settings. I have the following error:

[ 0.868048] mscc_felix 0000:00:00.5: Failed to read phy-mode or phy-interface-type property for port 0
[ 0.868063] mscc_felix 0000:00:00.5: Failed to register DSA switch: -19

Regarding the lynx pcs, I can't find this file in my kernel. I'm using version 5.4.47-RT that comes from the LSDK 20.12-v5.4-RT tag. It's hard for me to update the kernel now.

And debugging a bit, I saw that the PCS init is in the felix_vcs9959.c (drivers/net/dsa/ocelot), which does not mention anything about 1000Base-X. Also, the LS1028A has nothing about this. It only says about 1000Base-KX that I understand as another thing.

Besides that, I have tried the 1000Base-X in the phy-mode of the mscc_felix node. It shows an "Unsupported PHY mode 1000base-x on port 0" message. That is expected considering the Felix driver has no configuration for 1000Base-X, only for 2500Base-X.

Have I made some mistakes in the configuration without PHY? Have somebody already used the SFP cage directly connected to the SerDes?

Best regards,

Igor Franco

[yipingwang]

There is a need to check whether LSDK2012 supports 1000Base-X.

1000Base-X is supported in the upstream kernel through the SerDes pcs driver.

Meanwhile please do the following:

Revert back to the fixed-link mode and add the code attached for this mode in vcs9959.c.

[igor_franco]

Sorry, I don't understand.

What code am I supposed to add to the felix_vcs9959.c?

The drivers used by the LS1028A have no support to the 1000Base-X, including the fixed-phy.

[yipingwang]

The code.txt consist of a tentative function that adds 1000Base-X mode in vcs9959.c.
You can take 2500Base-X as reference and make changes in the vcs9959.c for 1000Base-X functionality.
For example : create functions for 1000Base-X like the following and all other changes that are required.
static void vsc9959_pcs_link_state_2500basex(struct phy_device *pcs, struct phylink_link_state *state)

[igor_franco]

I have done as you said. I added the support to 1000Base-X in the felix_vcs9959.c and kept the support in the felix.c felix_phylink_validate function. I got the following log:

# dmesg | grep -i sfp
[ 0.425337] libphy: SFP I2C Bus: probed
[ 0.425396] sfp sfp-netb: Host maximum power 2.0W
[ 1.206930] sfp sfp-netb: module OEM RSMP851GL-05D rev 1 sn PQH429J dc 201206
[ 1.221592] mscc_felix 0000:00:00.5 sfp2: switched to inband/1000base-x link mode
[ 5.595040] mscc_felix 0000:00:00.5 sfp1: configuring for fixed/sgmii link mode
[ 5.595269] mscc_felix 0000:00:00.5 sfp1: Link is Up - 1Gbps/Full - flow control off
[ 5.595286] 8021q: adding VLAN 0 to HW filter on device sfp1
[ 5.595394] IPv6: ADDRCONF(NETDEV_CHANGE): sfp1: link becomes ready
[ 5.598704] mscc_felix 0000:00:00.5 sfp2: configuring for inband/1000base-x link mode
[ 5.598973] 8021q: adding VLAN 0 to HW filter on device sfp2

Here, I keep only the sfp2 interface to use the sfp driver. The sfp1 is only using the SGMII fixed-link. Check the nodes in my DTS:

/ {
    sfp_netb: sfp-netb {
        compatible = "sff,sfp";
        maximum-power-milliwatt = <2000>;
        i2c-bus = <&i2c5>;

        los-gpios = <&gpio2 29 GPIO_ACTIVE_HIGH>;
        mod-def0-gpios = <&gpio2 30 GPIO_ACTIVE_LOW>;
        tx-fault-gpios = <&gpio2 28 GPIO_ACTIVE_HIGH>;
        tx-disable-gpios = <&gpio3 15 GPIO_ACTIVE_HIGH>;
    };
};

&mscc_felix_port0 {
    status = "okay";
    phy-mode = "sgmii";
    label = "sfp1";
    fixed-link {
          speed = <1000>;
          full-duplex;
    };
};

&mscc_felix_port1 {
     status = "okay";
     phy-mode = "1000base-x";
     label = "sfp2";
     sfp = <&sfp_netb>;
     fixed-link {
           speed = <1000>;
           full-duplex;
     };
};

Even the driver seems to be OK, the sfp2 interface is still in LOWERLAYERDOWN instead of UP:

# ip addr | grep sfp
5: sfp1@eth1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
           inet 10.19.82.1/29 brd 10.19.82.7 scope global sfp1
6: sfp2@eth1: <NO-CARRIER,BROADCAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
           inet 10.19.82.9/29 brd 10.19.82.15 scope global sfp2

The ping does not work with the sfp2 interface but works with sfp1.

Did I do something wrong? What is this LOWERLAYERDOWN?

Just to know, you mentioned a code.txt. However, it has nothing attached to your reply.

[yipingwang]

Does it mean you haven't added the function "static void vsc9959_pcs_init_1000basex" and others to support 1000Base-X?
Or it means you haven't got the expected results even after adding the support for 1000Base-X?

[yipingwang]

Please refer to the update from the expert team.

1. Please ask the customer to share the felix_vcs9959.c having the support of 1000Base-X.
2. Also, ask them to make changes according to the code.txt.
The code.txt consist of a tentative function that adds 1000Base-X mode in vcs9959.c.
You can take 2500Base-X as reference and make changes in the vcs9959.c for 1000Base-X functionality.
For example : create functions for 1000Base-X like the following and all other changes that are required.
static void vsc9959_pcs_link_state_2500basex(struct phy_device *pcs, struct phylink_link_state *state)

[igor_franco]

Regarding the previous reply: I have implemented the 1000base-X support and did not have the expected behavior. My sfp2 interface is always in the LOWERLAYERDOWN link state.

I have applied the changes mentioned in the code.txt attached. Still, the sfp2 interface keeps the same lower down state. Find a zip with my modified code for the Felix driver attached.

For this test, I used the same DTS mentioned in my last reply. That uses the sfp1 configuring it as an SGMII without the SFP driver and the sfp2 is using the driver with the modification you sent and requested.

[yipingwang]

From our expert:

phy_write(pcs, PHY_SGMII_IF_MODE_1000X, if_mode);

the correct way is to replace the above with:

#define IF_MODE 0x14

...

u16 if_mode = PHY_SGMII_IF_MODE_1000X;

phy_write(pcs, IF_MODE, if_mode);

Please set the link timer to 0x1312d0 that is(it must follow the upstream):

phy_write(pcs, ENETC_PCS_LINK_TIMER2, 0x0013); phy_write(pcs, ENETC_PCS_LINK_TIMER1, 0x12d0);

Check also  the SFP status signals: TX_FAULT, TX_DISABLE, MOD_ABS and RX_LOS.

• • allow to detect if the SFP transceiver is installed to the cage (MOD_ABS).
• • notify that Transmitter Fault (TX_FAULT) and Loss of Signal (RX_LOS) errors are not detected.
• • notify that transmitter is enabled (TX_DISABLE is pulled low on the board).

Make sure to add some debug print to check that the values  were applied.

[igor_franco]

I applied the changes you asked for. According to the debug print, the value written is right for both timer and if_mode. I have also changed the dev_dbg on sfp.c of drivers/net/phy to dev_err. The attached image shows the logs from both. It seems the SFP driver wants to put the link up but the interface keeps it down. The signals look right: LOS goes up when I remove the input cable and the other signals are always zero (tx fault and MOD). We identified a problem in our hardware regarding these last two signals. They don't activate properly, but must not be an issue once they are only not identifying the faults.

[yipingwang]

From our expert:

Add some debug prints here:

vsc9959_pcs_link_state_resolve.

Transform phydev_dbg(pcs,

                    "mode=%s/%s/%s adv=%*pb lpa=%*pb link=%u an_enabled=%u an_complete=%u\n",

Into phydev_err.

Also add the following :

diff --git a/drivers/net/dsa/ocelot/felix_vsc9959.c b/drivers/net/dsa/ocelot/felix_vsc9959.c

index cd0c12492411..87ffc7c1f6d4 100644

--- a/drivers/net/dsa/ocelot/felix_vsc9959.c

+++ b/drivers/net/dsa/ocelot/felix_vsc9959.c

@@ -924,6 +924,8 @@ static void vsc9959_pcs_init(struct ocelot *ocelot, int port,

        linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Full_BIT, pcs->supported);

        linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Half_BIT, pcs->supported);

        linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, pcs->supported);

+       linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseX_Half_BIT, pcs->supported);

+       linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT,

+ pcs->supported);

You could comment out the line that says:

        if (pcs->interface == PHY_INTERFACE_MODE_1000BASEX)

                 linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT,

                                   pcs->supported);

We need to see if the link is set in PCS or not. If the link is set in PCS but the driver says that there is no link then it's something missing at driver level.

(the general idea is that: once the PCS has been configured in a mode, those parameters will trigger the hardware to start in that mode. If there is a software issue then we should see a link in PCS while the driver reports that there is no link)

[igor_franco]

I was not able to set the ETHTOOL_LINK_MODE_1000baseX_Half_BIT once this bit does not exist in the ethtool.h (see attached image with the compile error). Therefore, I only kept the Full version. The print you asked for returned the following:

[ 6.400855] mdio_bus 0000:00:00.5-imdio:01: mode=1000base-x/1Gbps/Unknown adv=00,00000200,000002ff lpa=00,00000000,00000000 link=0 an_enabled=1 an_complete=0

I found the duplex as Unknown strange and add the duplex as full in the vsc9959_pcs_link_state_1000basex function. That fixed the duplex but the link is still not working:

[ 6.697320] mdio_bus 0000:00:00.5-imdio:01: mode=1000base-x/1Gbps/Full adv=00,00000200,000002ff lpa=00,00000000,00000000 link=0 an_enabled=1 an_complete=0

Could that happen because of the auto-negotiation?

[yipingwang]

From our expert:
check the link partner to see if it is correctly configured. It must have autoneg on. If it does not have autoneg on then disable autoneg from the driver.
The problem seems here:
an_enabled=1 an_complete=0

The first step in the setup would be to use sgmii and see that everything works then apply the changes to see what is happening.
There is a step the customer can check in his setup:
Have two ports with sfps configured in SGMII (SGMII we know that works), connect those two ports between them with optic.
Create 2 netns and ping between the interfaces.

If ping works then go to the next step:
- configure both to 1000basex and restart the board then check if there is link and if ping works.
Basically we do not use any peer, we use the same board sending the traffic back to us.

Meanwhile I am also trying to create a setup locally to test all the changes.

[igor_franco]

I have tried to comment on the SFP option in the DTS, leaving a fixed link with 1000Base-X as phy-mode. In this setup, I was able to ping.

Just to understand the architecture: the SFP is connected to a switch and a computer is connected to the same switch through RJ45. Both fixed links with SGMII and 1000Base-X communicated successfully (ping), only using the SFP option remains with the problem.

[yipingwang]

From our expert:

Add some extra prints , starting the investigation from below. Probably you will need to go in sfp.c.

       /* Ignore errors if we're expecting a PHY to attach later */

        ret = phylink_validate(pl, support, &config);

        if (ret) {

                phylink_err(pl, "validation with support %*pb failed: %d\n",

                            __ETHTOOL_LINK_MODE_MASK_NBITS, support, ret);

                return ret;

        }

I observed that your SFP node does not have any i2c-bus. Usually the SFPs are controlled through a serial bus.

Check your schematics and see how the SFP can be reached.