SFP module not working in Linux

[tzaman]

We're in the middle of bringing up a custom LS1046A-based board and we've gotten far enough to boot into Linux successfully (using lsdk2108_yocto_tiny_LS_arm64.itb).

The problem we're facing currently is that SFP modules will not establish a link with the switch, but only in Linux, in u-boot, it works fine.

Here's the output:

[    2.105567] sfp sfp-xfi0: Host maximum power 3.0W
[    2.110647] sfp sfp-xfi1: Host maximum power 3.0W
[    2.133460] fsl_dpaa_mac 1ae2000.ethernet eth0: Probed interface eth0
[    2.158141] fsl_dpaa_mac 1ae8000.ethernet eth1: Probed interface eth1
[    2.182874] fsl_dpaa_mac 1aea000.ethernet eth2: Probed interface eth2
[    2.207768] fsl_dpaa_mac 1af0000.ethernet eth3: Probed interface eth3
[    2.232690] fsl_dpaa_mac 1af2000.ethernet eth4: Probed interface eth4
[    2.239517] clk: Disabling unused clocks
[    2.244553] Freeing unused kernel memory: 3008K
[    2.249175] Run /init as init process
INIT: version 2.99 booting
Starting udev
[    2.358215] udevd[125]: starting version 3.2.10
[    2.434319] sfp sfp-xfi0: module FLEXOPTIX        P.8596.02        rev A    sn F7B2H4B          dc 240827
[    2.488055] sfp sfp-xfi1: module FLEXOPTIX        P.C30.1          rev 1.0  sn F7B0V11-B        dc 240722
[    2.495775] hwmon hwmon6: temp1_input not attached to any thermal zone
[    3.394590] random: crng init done
[    3.401138] udevd[126]: starting eudev-3.2.10
[    3.442182] fsl_dpaa_mac 1ae2000.ethernet fm1-mac2: renamed from eth0
[    3.483254] fsl_dpaa_mac 1ae8000.ethernet fm1-mac5: renamed from eth1
[    3.514694] fsl_dpaa_mac 1aea000.ethernet fm1-mac6: renamed from eth2
[    3.538708] fsl_dpaa_mac 1af0000.ethernet fm1-mac9: renamed from eth3
[    3.566712] fsl_dpaa_mac 1af2000.ethernet fm1-mac10: renamed from eth4

As you can see, both modules that are inserted are correctly detected, no issues reported on any of the interfaces either, if I bring them up, they work fine:

root@TinyLinux:~# ip link set fm1-mac9 up
[   14.332682] fsl_dpaa_mac 1af0000.ethernet fm1-mac9: configuring for inband/10gbase-kr link mode
root@TinyLinux:~# ip link set fm1-mac10 up
[   19.500673] fsl_dpaa_mac 1af2000.ethernet fm1-mac10: configuring for inband/10gbase-kr link mode
root@TinyLinux:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: fm1-mac2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:04:9f:08:06:4a brd ff:ff:ff:ff:ff:ff
3: fm1-mac5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:04:9f:08:06:4b brd ff:ff:ff:ff:ff:ff
4: fm1-mac6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:04:9f:08:06:4c brd ff:ff:ff:ff:ff:ff
5: fm1-mac9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether 00:04:9f:08:06:4d brd ff:ff:ff:ff:ff:ff
6: fm1-mac10: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether 00:04:9f:08:06:4e brd ff:ff:ff:ff:ff:ff

I don't understand why no-carrier, regardless which mode I try in the device tree, I've tried xfi, xsgmii, 10gbase-r, nothing works.

Please advise.

[tzaman]

For posterity; We fixed the issue, turns out that we forgot to configure RCW for the pin that was driving our clock mux low, which fed 100 Mhz clock into the Serdes reference clock input, rather than 156.25 MHz that it needs. After we fixed RCW, everything started to work properly.