network interface not working properly in ls1028a with specific serdes configuration

Hamza — Tue, 17 Oct 2023 10:19:00 GMT

Hi,

We are currently using the layerscape module ls1028a in our custom board these are the network configuration that we are interested to use in our project.

SRDS_PRTCL_S1 = 0x9999

Reset Configuration Word (RCW): 00000000: 34004010 00000030 00000000 00000000 00000010: 00000000 008e0000 0030c000 00000000 00000020: 01403150 00002580 00000000 000080de 00000030: 00000000 00000010 00000000 00000000 00000040: 00000000 00000000 00000000 00000000 00000050: 00000000 00000000 00000000 00000000 00000060: 00000000 00000000 000e7000 00000000 00000070: 99990000 03000000 Using SERDES1 Protocol: 39321 (0x9999)

Device tree:

&enetc_port0 { status = "disabled"; }; &enetc_port1 { status = "disabled"; }; &enetc_mdio_pf3 { mdio1_sgmii_phy01: ethernet-phy@1 { reg = <0x01>; }; }; /* l2switch ports */ mxl8153_port0: &mscc_felix_port0 { status = "okay"; phy-mode = "sgmii"; fixed-link { speed = <1000>; full-duplex; }; }; rj45_port1: &mscc_felix_port1{ status = "okay"; phy-mode = "sgmii"; phy-handle = <&mdio1_sgmii_phy01>; }; sfp_port3: &mscc_felix_port2 { status = "okay"; phy-mode = "1000base-x"; sfp = <&sfp_neta>; fixed-link { speed = <1000>; full-duplex; }; }; &mscc_felix_port3{ status = "disabled"; phy-mode = "sgmii"; };

~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1520 qdisc mq state UP group default qlen 1000 link/ether 00:d0:93:5a:58:89 brd ff:ff:ff:ff:ff:ff altname enp0s0f2 inet6 fe80::2d0:93ff:fe5a:5889/64 scope link valid_lft forever preferred_lft forever 4: swp0@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:d0:93:5a:58:89 brd ff:ff:ff:ff:ff:ff inet 192.168.1.99/24 brd 192.168.1.255 scope global swp0 valid_lft forever preferred_lft forever inet6 fe80::2d0:93ff:fe5a:5889/64 scope link valid_lft forever preferred_lft forever 5: swp1@eno2: <BROADCAST,MULTIpiCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:d0:93:5a:58:89 brd ff:ff:ff:ff:ff:ff inet 192.168.3.99/24 brd 192.168.3.255 scope global swp1 valid_lft forever preferred_lft forever inet6 fe80::2d0:93ff:fe5a:5889/64 scope link valid_lft forever preferred_lft forever 6: swp2@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:d0:93:5a:58:89 brd ff:ff:ff:ff:ff:ff inet 192.168.2.99/24 brd 192.168.2.255 scope global swp2 valid_lft forever preferred_lft forever inet6 fe80::2d0:93ff:fe5a:5889/64 scope link valid_lft forever preferred_lft forever

~# dmesg | grep felix [ 2.878838] mscc_felix 0000:00:00.5: Adding to iommu group 0 [ 3.938295] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 0 [ 3.945147] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 1 [ 3.951985] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 2 [ 3.982181] mscc_felix 0000:00:00.5: configuring for fixed/internal link mode [ 3.989402] mscc_felix 0000:00:00.5: Link is Up - 2.5Gbps/Full - flow control rx/tx [ 4.016168] mscc_felix 0000:00:00.5 swp1 (uninitialized): PHY [0000:00:00.3:01] driver [RTL8211F Gigabit Ethernet] (irq=POLL) [ 4.028004] mscc_felix 0000:00:00.5 swp2 (uninitialized): switched to inband/1000base-x link mode [ 7.462744] mscc_felix 0000:00:00.5 swp0: configuring for fixed/sgmii link mode [ 7.482654] mscc_felix 0000:00:00.5 swp0: Link is Up - 1Gbps/Full - flow control off [ 7.531142] mscc_felix 0000:00:00.5 swp1: configuring for phy/sgmii link mode [ 7.586585] mscc_felix 0000:00:00.5 swp2: configuring for inband/1000base-x link mode [ 7.663844] mscc_felix 0000:00:00.5 swp2: Link is Up - 1Gbps/Full - flow control rx/tx [ 10.629362] mscc_felix 0000:00:00.5 swp1: Link is Up - 1Gbps/Full - flow control rx/tx ~# dmesg | grep enetc [ 0.788902] pci 0000:00:00.0: enetc_fixup_clear_rss_rfs+0x0/0x5c took 105157 usecs [ 0.896919] pci 0000:00:00.1: enetc_fixup_clear_rss_rfs+0x0/0x5c took 105457 usecs [ 1.004395] pci 0000:00:00.6: enetc_fixup_clear_rss_rfs+0x0/0x5c took 104925 usecs [ 2.885141] fsl_enetc 0000:00:00.2: Adding to iommu group 1 [ 2.891055] fsl_enetc_mdio 0000:00:00.3: Adding to iommu group 2 [ 3.004108] fsl_enetc_mdio 0000:00:00.3: enabling device (0400 -> 0402) [ 3.014087] fsl_enetc_ptp 0000:00:00.4: Adding to iommu group 3 [ 3.020113] fsl_enetc_ptp 0000:00:00.4: enabling device (0404 -> 0406) [ 3.544114] fsl_enetc 0000:00:00.2: enabling device (0400 -> 0402) [ 4.037412] fsl_enetc 0000:00:00.2 eth0: entered promiscuous mode [ 7.215724] fsl_enetc 0000:00:00.2 eno2: renamed from eth0 [ 7.427418] fsl_enetc 0000:00:00.2 eno2: configuring for fixed/internal link mode [ 7.442037] fsl_enetc 0000:00:00.2 eno2: Link is Up - 1Gbps/Full - flow control rx/tx

~# ethtool swp1 Settings for swp1: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: Symmetric Receive-only Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on master-slave cfg: preferred slave master-slave status: slave Port: Twisted Pair PHYAD: 1 Transceiver: external MDI-X: Unknown Supports Wake-on: d Wake-on: d Link detected: yes

The problem that I’m facing is that I’m not able to ping (send/receive data) using the interface swp1 (RJ45).

We tried to remove the flow control rx/tx using the ethtool using the following command before assigning an ip address to the interface and I was able to ping :

~# ethtool -A swp1 autoneg off rx off tx off ~# [ 128.775236] mscc_felix 0000:00:00.5 swp1: Link is Down [ 131.557587] mscc_felix 0000:00:00.5 swp1: Link is Up - 1Gbps/Full - flow control off ping 192.168.3.222 PING 192.168.3.222 (192.168.3.222): 56 data bytes 64 bytes from 192.168.3.222: seq=0 ttl=128 time=0.861 ms 64 bytes from 192.168.3.222: seq=1 ttl=128 time=0.674 ms

Unfortunately, I lose the connection (ping) if I remove and replug the cable, if I try to test the interface from ethtool

~# ethtool -t swp1 The test result is FAIL The test extra info: 1. Carrier 0 2. PHY dev is present 0 3. PHY internal loopback, enab 0 4. PHY internal loopback, UDP -110 5. PHY internal loopback, MTU -110 6. PHY internal loopback, TCP -110 7. PHY internal loopback, disa 0

This behavior is the same for linux kernel version 5.15 , 6.1.35 and 6.5.7 (we wanted to make sure that it was not a driver issue that is fixed in a newer version)

Re: network interface not working properly in ls1028a with specific serdes configuration

Oswalag — Mon, 23 Oct 2023 20:56:36 GMT

Hello

" This behavior is the same for linux kernel version 5.15 , 6.1.35 and 6.5.7 (we wanted to make sure that it was not a driver issue that is fixed in a newer version) " - It isn't a driver issue, did you follow the steps in chapter 29.7.1.1 1G SGMII in the ls1028ARM? it looks like an issue related to the Auto-N.

Re: network interface not working properly in ls1028a with specific serdes configuration

Hamza — Wed, 15 Nov 2023 15:14:30 GMT

Hi @Oswalag

I confirm that this is issue is related to auto negotiation in fact, the uboot driver is setting up everything correctly but the problem starts when booting to kernel.

affected registers 29.6.9.2 SGMII Control (SGMII_CR) 29.6.9.18 SGMII IF Mode (SGMII_IF_MODE)

before enabling the network interface both registers values are the same as the uboot initialization.

After enabling the network interface both registers values changes and the interface became unusable.

ip link set dev <interface> up or ip link set dev <interface> down

when we restart the service systemd-networkd we lost the connection with the interface.

systemclt restart systemd-networkd

to restore the interface we need to launch these commands which basically reinitialize the values into the registers as described in (29.7.1.1 1G SGMII).

devmem2 0x1f8148034 w 0x34 devmem2 0x1f8148038 w 0x3 devmem2 0x1f8148034 w 0x20 devmem2 0x1f8148038 w 0x1200

Do you have any idea what is the reason behind this behavior ? if so, I would appreciate it if you can share some the solution to solve it.

Best regards,

Re: network interface not working properly in ls1028a with specific serdes configuration

Oswalag — Fri, 17 Nov 2023 19:17:15 GMT

Please check the configuration of the registers,

This initialization says that there are 2 processes A and B. A is auto-negation process itself as discovering counterparts capabilities and B process as serdes PCS configuration for SGMII in accordance to the result of A process.

So when USE_SGMII_AN=0 the PCS operation should be programmed with the register bit SGMII_SPEED and SGMII_DUPLEX. When set to 1, the PCS operation is automatically programmed with the Partner abilities advertised during Auto-Negotiation.

topic network interface not working properly in ls1028a with specific serdes configuration in Layerscape

network interface not working properly in ls1028a with specific serdes configuration

Re: network interface not working properly in ls1028a with specific serdes configuration

Re: network interface not working properly in ls1028a with specific serdes configuration

Re: network interface not working properly in ls1028a with specific serdes configuration