[LS1028a]: Ethernet not working in Linux

[kumar_086]

Dear Experts,

We have a custom LS1028a board, having 1G SGMII Ethernet connected to ENETC port 1/2.5G SGMII/10G-SXGMII in SGMII 1G configuration mode. Externally there is a Marvell PHY 88E1512. Identified and configured the 88E1512 to work in SGMII 1Gbps mode in U-Boot using mii commands.

Ethernet in U-Boot working when emdio-3 is selected,

>mii device emdio-3
>mii info
PHY 0x00: OUI = 0x5043, Model = 0x1D, Rev = 0x01, 1000baseT, FDX
PHY 0x10: OUI = 0x01C1, Model = 0x0A, Rev = 0x02, 10baseT, HDX
PHY 0x11: OUI = 0x01C1, Model = 0x0A, Rev = 0x02, 10baseT, HDX
PHY 0x12: OUI = 0x01C1, Model = 0x0A, Rev = 0x02, 10baseT, HDX
PHY 0x13: OUI = 0x01C1, Model = 0x0A, Rev = 0x02, 10baseT, HDX

88E1512 - PHY 0x00: OUI = 0x5043

>mii device
MII devices: 'enetc-0' 'emdio-3' 'enetc-2' 'felix-switch'

Tried enabling the same in linux dts @ fsl-ls1028a-rdb.dts at packages/linux/linux/arch/arm64/boot/dts/freescale (attached the same),

&enetc_port1 {
phy-handle = <&sgmii_phy0>;
phy-connection-type = "sgmii";

mdio {
#address-cells = <1>;
#size-cells = <0>;
sgmii_phy0: ethernet-phy@0 {
reg = <0x0>;
};
};
};

&enetc_port0 {
status = "disabled";
};

With this addition able to see the Ethernet ports eno1, eno2 in ifconfig. But the ping is not working with pc over eno1. (Observed eno2 RX packets incremented to 2/3 packs)

Have the following questions,
1. What is the 'emdio-3' as per ENETC at U-Boot Level?
2. What changes to be added to Linux dts or any other configs to enable the Ethernet interface?
3. Repeated PCIe Bus error observed, what are the implications of this?
[ 10.667961] pcieport 0002:00:00.0: AER: Multiple Corrected error received: 0002:00:00.0
[ 10.676000] pcieport 0002:00:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 10.686039] pcieport 0002:00:00.0: AER: device [1957:82c0] error status/mask=00000001/00006000
[ 10.694862] pcieport 0002:00:00.0: AER: [ 0] RxErr

Attached dts files, linux boot log

Thanks

[yipingwang]

Would you please provide the result of "=> mdio list" in u-boot?

[kumar_086]

Dear @yipingwang,

The response of mdio list command as follows,

=> mdio list
enetc-0:
emdio-3:
10 - Vitesse VSC8574 <--> swp0
11 - Vitesse VSC8574 <--> swp1
12 - Vitesse VSC8574 <--> swp2
13 - Vitesse VSC8574 <--> swp3
enetc-2:

felix-switch:

Regards

[yipingwang]

I didn't find obvious problem in your dts file and Linux Kernel booting up log.

I need to escalated this case to the SE team, please wait for the feedback from them.

[yipingwang]

Please refer to the following update from the AE team.

What pcie device is connected? I think this is another issue.

Why ip address in linux is different with that in u-boot? plus, remote address is also different?

[kumar_086]

Dear @yipingwang 

1. We have the following PCIe configuration with the processor,
A. PCIe 1x1 - IGLOO2 FPGA (Gen 1.0) - This device is getting enumerated in Linux (log: pci 0002:01:00.0: [11aa:1556] type 00 class 0x000000)
B. PCIe 1x1 - PCIe switch (PEX8604) - This device is not configured right now

What is the issue suspected with PCIe?

Can you share the steps if we can correct it or make it work?

2. The IP addresses got changed because of the PC auto configuration. The IP Address is fixed now for both U-Boot & Linux and the remote address. The new log is captured and attached to this thread.

3. One more point, I have logged the 'cat /proc/interrupts' and observed the tx interrupts incrementing for CPU1 not CPU0; any hints with this.

Please share your inputs.

Thanks

[yipingwang]

Please check phy link status in linux by "ethtool eno0"

[kumar_086]

Dear @yipingwang ,

Yes, the phy link is present, as Linux prints below status,

root@TinyLinux:~# ifconfig eno0 172.16.1.51 up
[ 26.533349] page 18, reg 20: 0x14: 0001 (exp: 1)
[ 26.538157] page 1, reg 0: 0140 (exp: 0x140)
[ 26.542598] page 0, reg 0: 0140 (exp: 0x0140)
[ 26.547402] Marvell 88E1510 0000:00:00.0:00: attached PHY driver [Marvell 88E1510] (mii_bus:phy_addr=0000:00:00.0:00, irq=POLL)

*bold highlights the Marvell family device 88E1512/0 is attached and italic shows the configs done at u-boot are holding properly.

Unfortunately, we dont have ethtool and couldn't find the utility in this rfs.

Thanks

[yipingwang]

The SE team requires the information of ethtool.

Please copy attached ethtool command to your target board through USB or SD card, then type the following command.

root@TinyLinux:~# ./ethtool eno0
Settings for eno0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Link detected: no
root@TinyLinux:~#

[kumar_086]

Dear @yipingwang,

Thanks for sharing the ethtool utility. Here is the output of it,

root@TinyLinux:~# ethtool eno0
Settings for eno0:
        Supported ports: [ TP MII FIBRE ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Link detected: yes

As this is a critical interface of the project, to speed up this bring up is it possible to set up a call or mail at your convenience, please?

Thanks

[yipingwang]

The SE team updated

Any possible to provide schematic or board for further investigation?

[kumar_086]

Dear @yipingwang,

Attached are the schematics along with lspci log.

Listed are some doubts which are related to Ethernet context,

1. How we can access the Ethernet controller registers in this processor? (As this processor has an Ethernet controller connected over PCIe)

2. Does this have any relation with the RCW configurations? ENETC/GPIO's?

3. Does U-Boot use interrupts for Ethernet packet transmissions?

4. Does Linux use interrupts for Ethernet packet transmissions?

5. Will there be any issue due to interrupt mapping? The interrupt no's are not changed and are directly used from the RDB dts. Does anything need to be changed for the custom target?

root@TinyLinux:~# cat /proc/interrupts
CPU0 CPU1
1: 0 0 GICv3 25 Level vgic
3: 4194 3361 GICv3 30 Level arch_timer
4: 0 0 GICv3 27 Level kvm guest vtimer
6: 0 0 GICv3 23 Level arm-pmu
8: 1 0 GICv3 57 Level 20c0000.spi
9: 1 0 GICv3 66 Level 2000000.i2c
12: 637 0 GICv3 64 Level ttyS0
13: 0 0 GICv3 88 Level eDMA
14: 0 0 GICv3 68 Level gpio-cascade, gpio-cascade
15: 0 0 GICv3 69 Level gpio-cascade
16: 0 0 GICv3 112 Level xhci-hcd:usb1
17: 3 0 GICv3 113 Level dwc3-otg, xhci-hcd:usb3
18: 0 0 GICv3 165 Level ahci-qoriq[3200000.sata]
19: 0 0 GICv3 45 Level arm-smmu global fault
20: 0 0 GICv3 46 Level arm-smmu global fault
21: 0 0 GICv3 47 Level arm-smmu global fault
22: 0 0 GICv3 48 Level arm-smmu global fault
23: 0 0 GICv3 243 Level arm-smmu global fault
24: 0 0 GICv3 244 Level arm-smmu global fault
25: 0 0 GICv3 245 Level arm-smmu global fault
26: 0 0 GICv3 246 Level arm-smmu global fault
97: 0 0 GICv3 252 Level galcore:0
98: 0 0 GICv3 115 Level audio-controller
100: 0 0 GICv3 140 Level PCIe PME
101: 0 0 GICv3 141 Level aerdrv
102: 0 0 GICv3 145 Level PCIe PME
103: 2 0 GICv3 146 Level aerdrv
104: 0 0 GICv3 77 Level 2810000.timer
110: 0 0 ITS-MSI 1 Edge eno0-rxtx0
111: 0 13 ITS-MSI 2 Edge eno0-rxtx1
115: 0 0 ITS-MSI 8192 Edge ptp_qoriq
116: 2 0 GICv3 172 Level 8010000.jr
117: 0 0 GICv3 173 Level 8020000.jr
118: 0 0 GICv3 174 Level 8030000.jr
IPI0: 1088 1306 Rescheduling interrupts
IPI1: 6 1 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 574 667 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0

Thanks

[yipingwang]

1. How we can access the Ethernet controller registers in this processor? (As this processor has an Ethernet controller connected over PCIe) It's not real PCIe, it's based on Integrated endpoint root complex, which is PCIe likely bus inside SoC., you can refer RM to access any reg.

 2. Does this have any relation with the RCW configurations? ENETC/GPIO's?

It's working on u-boot, so HW configuration is supposed to be fine.

 3. Does U-Boot use interrupts for Ethernet packet transmissions?

No

 4. Does Linux use interrupts for Ethernet packet transmissions?

Yes 

5. Will there be any issue due to interrupt mapping? The interrupt no's are not changed and are directly used from the RDB dts. Does anything need to be changed for the custom target?

Since there is TX counter, can you check if frames are really received at other side? Even it's due to the interrupt issue, other side should receive some frames at the beginning. Plus, any way to check TX/RX counter of MArvell PHY?

[kumar_086]

Dear @yipingwang,

Thank you for providing the answers. 

The other end PC is not receiving any packets sent from the LS processor. The Marvell phy 88E1512 doesn't have the Ethernet statistics.

Regards

Kumar

[yipingwang]

For SGMII connection, please try add "in-band-status", looking forward to your feedback.

Something like:
port@0 {
status = "okay";
phy-handle = <&qsgmii_phy1>;
phy-mode = "qsgmii";
managed = "in-band-status";
};

[kumar_086]

Dear @yipingwang,

Added 'managed = "in-band-status";' for SGMII in dts file and the results are same.

I'm happy to share that Ethernet started working when the &enetc_port1 is enabled with the phy information rather &enetc_port0, 

&enetc_port1 {
    phy-handle = <&sgmii_phy1>;
    phy-connection-type = "sgmii";
    mdio {
        #address-cells = <1>;
        #size-cells = <0>;
        sgmii_phy1: ethernet-phy@0 {
            reg = <0x0>;
        };
    };
};

&enetc_port1 {
     status = "enabled";
};

As per the LS1028A reference manual and RDB configurations SGMII port is connected to port 0. Any clue why is this behavior with port 1?

Thanks for your support and to all NXP SE Team & Mr Suresh

[yipingwang]

A little bit confused. Do you mean port0 is working in linux after port1 in dts is changed? Which port are you using actually?

[kumar_086]

Yes, exactly. When port 1 is specified in dts, port 0 is working (a bit strange, looking forward to why is this so).

root@TinyLinux:~# ping 169.254.35.151 -I eno0
PING 169.254.35.151 (169.254.35.151): 56 data bytes
64 bytes from 169.254.35.151: seq=1 ttl=128 time=0.709 ms
64 bytes from 169.254.35.151: seq=2 ttl=128 time=0.430 ms
64 bytes from 169.254.35.151: seq=3 ttl=128 time=0.511 ms
^C
--- 169.254.35.151 ping statistics ---
4 packets transmitted, 3 packets received, 25% packet loss
round-trip min/avg/max = 0.430/0.550/0.709 ms
root@TinyLinux:~#
root@TinyLinux:~#
root@TinyLinux:~# ifconfig
eno0 Link encap:Ethernet HWaddr 3e:70:99:61:e9:d4
inet addr:169.254.35.51 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::3c70:99ff:fe61:e9d4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8343 (8.1 KiB) TX bytes:2924 (2.8 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

The Ethernet scheme used in the custom board is the same as in RDB (except for the PHY chip make), LS1028A Ethernet Port 0 SGMII eno0 (attached the figure).

[yipingwang]

How about to apply your port1 configuration described previously onto port0? Something like:

&enetc_port0 {

    phy-handle = <&sgmii_phy0>;

    phy-connection-type = "sgmii";

    mdio {

        #address-cells = <1>;

        #size-cells = <0>;

        sgmii_phy0: ethernet-phy@0 {

            reg = <0x0>;

        };

    };

};

[kumar_086]

We tried this method as well before by mentioning port 0 in dts file which is working in RDB but not working in the custom board. 

[yipingwang]

The SE team(in Beijing)  asked whether you could send your custom board to them to do more investigation?