U-Boot driver needed to make Ethernet work in Linux

[JohnKlug]

We are using Micrel Phy:

KSZ8091RNBCA

For some reason Linux will not work from a cold boot if U-Boot is not configured for the Micrel Phy. Also the debug port becomes very sluggish and has lots of errors.

fec 2188000.ethernet eth0: Unable to connect to phy
RTNETLINK answers: No such device

bash# mii-diag eth0
SIOCGMIIPHY on eth0 failed: Operation not supported

We are using i.MX6ULL which has a single Ethernet port.

Linux seems to recognize our phy when U-Boot is configured and everything is normal.

Micrel KSZ8041 2188000.ethernet-1:01: attached PHY driver (mii_bus:phy_addr=2188000.ethernet-1:01, irq=POLL)
fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx

We would like to eliminate the U-Boot Ethernet driver, since we never use it, and we think it is better for security and boot simplicity if we do not have it.  What could be causing Linux not to initialize the Ethernet properly?

For instance mii-diag is unable to contact the Phy on a cold boot without the U-Boot driver:

[JohnKlug]

I also tried disabling Ethernet in U-Boot in device tree, which did not work.  Now I added code in U-Boot to change the Ethernet pins to GPIO inputs. This made it worse, and it no longer works in a warm boot situation.

For instance we are using MX6UL_PAD_ENET1_TX_CLK__ENET1_REF_CLK1.

Here is U-Boot, setting the Ethernet pins to GPIO during U-Boot:

=> md.l 0x20e0368 1
020e0368: 0001b000                             ....
=> md.l 0x20e00dc 1
020e00dc: 00000005                             ....

Here are the same pins set to Ethernet in Linux, which does not work when U-Boot has somehow disabled Ethernet:

bash# devmem2 0x20e0368 w
/dev/mem opened.
Memory mapped at address 0x76f3a000.
Read at address  0x020E0368 (0x76f3a368): 0x00000020
bash# devmem2 0x20e00dc w
/dev/mem opened.
Memory mapped at address 0x76f78000.
Read at address  0x020E00DC (0x76f780dc): 0x00000014

In the U-Boot case the pins are now GPIO inputs.

Linux still has the correct pinctrl, and has defined the usage of the Ethernet mux, mux 4.  But I still have issues with Linux not reaching the PHY when U-Boot has disabled Ethernet:


But with the Ethernet pins set to GPIO inputs in U-Boot, now I am seeing a failure to reach the Ethernet PHY in both cold and warm boots.

[JohnKlug]

I disabled the following using U-Boot's menuconfig:

CONFIG_PHYLIB
CONFIG_PHY_MICREL
CONFIG_PHY_MICREL_KSZ8XXX
CONFIG_DM_MDIO
CONFIG_FEC_MXC
CONFIG_MII

This change worked on another i.MX6ULL board we have that used a TI phy (maybe not exactly the same case because it is not Micrel) when we disabled it in U-Boot.  I assume something is causing the CPU to enter an undefined state, because why else would the debug serial port go bad when in Linux? As I said it becomes ridiculously slow, full of bad characters and hangs, and TOP is not showing any processes hogging memory or CPU.  And putting the Ethernet setup back in U-Boot fixes everything.

I will look into changing the MDIO Ethernet pins to GPIO inputs in U-Boot and see if that fixes it.

[joanxie]

if you remove these from config files, maybe would cause enet clock enable issue, refer to the clock file, one can refer to the source code as below, if you just set disabled status in the dts file, does it have any issue? I didn't this PHY on the imx6ull to test for you, so it's hard to trace the root cause, I don't suggest that you remove the config file otherwise you can ensure this modification wouldn't  cause any other issue

 https://github.com/nxp-imx/uboot-imx/blob/lf_v2022.04/arch/arm/mach-imx/mx6/clock.c#L103

[JohnKlug]

Normally during boot I get a successful link in U-Boot, the link is brought down and Linux brings up the link again.

If I disable Ethernet in U-Boot device tree and leave the .config file for U-Boot alone, U-Boot no longer links.  However Linux does not link either, and I see the unable to connect to phy mesage.

This is my only change:

 &fec1 {
        phy-reset-gpios = <&gpio1 4 GPIO_ACTIVE_LOW>;
        phy-handle = <&ethphy1>;
+       status = "disabled";
        mdio {
                #address-cells = <1>;
                #size-cells = <0>;

What follows in Linux is:

fec 2188000.ethernet eth0: Unable to connect to phy


And the link lights never come on ever.

[joanxie]

what bsp version do you test? as you mentioned before, you tested other ethernet module which  is ok, only KSZ8091RNBCA？ and could you reproduce this on nxp imx6ull evk board? 

[JohnKlug]

The Linux we use is:

https://github.com/nxp-imx/mwifiex/tree/lf-5.15.71_2.2.0/mxm_wifiex/wlan_src

The U-Boot is lf_v2204.04:

https://github.com/nxp-imx/uboot-imx/tree/lf_v2022.04

I did not try the EVK board.  I have not done a build for the EVK in a long time. But we have another i.MX6ULL board that is running the same kernel, and with U-Boot disabled, Linux works.  This PHY is:

DP83825IRMQR

Disablement in U-Boot:

# CONFIG_PHYLIB is not set

# CONFIG_FEC_MXC is not set
# CONFIG_MII is not set

Note that on the working board with the TI phy the link status lights do not come on during U-Boot, but do come on in Linux (what we want). Also, when resetting the phy with the hardware reset pin, the phy loses link and comes back, just like what we want.

With the  Micrel board, if we disable U-Boot drivers (either in config or device tree), the link status lights never come on in U-Boot or Linux.

The kernel configuration is exactly the same for Linux on the two boards, but device trees differ.

Here are the PHY related entries for Linux:

CONFIG_SWPHY=y
# CONFIG_LED_TRIGGER_PHY is not set
CONFIG_FIXED_PHY=y

CONFIG_FEC=y

CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
CONFIG_FWNODE_MDIO=y
CONFIG_OF_MDIO=y
CONFIG_MDIO_DEVRES=y
CONFIG_MICREL_PHY=y
CONFIG_MICROCHIP_PHY=y
CONFIG_AT803X_PHY=y
CONFIG_SMSC_PHY=y
CONFIG_DP83822_PHY=y
CONFIG_USB_PHY=y
CONFIG_USB_MXS_PHY=y
CONFIG_GENERIC_PHY=y
CONFIG_PHY_FSL_IMX_PCIE=y

[JohnKlug]

I thought I would start to look at registers. If U-Boot does not enable the Ethernet, then it is not possible:

bash# devmem2 0x2188044 w
/dev/mem opened.
[HANG]

If U-Boot initializes the driver, devmem2 works:

bash# devmem2 0x2188044 w
/dev/mem opened.
Memory mapped at address 0x76f19000.
Read at address 0x02188044 (0x76f19044): 0x0000001A

The register I am trying to read:
MII Speed Control Register (ENET1_MSCR)

[JohnKlug]

I looked in /sys/kernel/debug/clk/clk_summary.

The following clocks are not enabled when things do not work:

pll6_enet
enet_ptp
enet_ahb
ipg->enet

All say "N" in the hardware enable column.  So what does one do to enable these clocks in Linux?  If U-Boot initializes Ethernet, the clocks are enabled.  If I use my board that works without U-Boot enabling Ethernet, all these clocks are enabled.

[joanxie]

do you mind sharing two clock dump when you enable ethernet in the uboot and disable the ethernet in the uboot?

this is clock driver in the kernel

https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/drivers/clk/imx/clk-imx6ul.c

this is ethernet clock settings in the kernel

https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/arch/arm/boot/dts/nxp/imx/imx6ul-14x14-evk.dtsi#L...

[JohnKlug]

I was able to fix the issue by changing the clocks line in our single phy from ENET to ENET2.

               micrelphy1: ethernet-phy@1 {
                        reg = <1>;
                        micrel,led-mode = <0>;
                        clocks = <&clks IMX6UL_CLK_ENET2_REF>;  // Fixes Issue
                        clock-names = "rmii-ref";
                };

Our i.MX6ULL on the non-working (when not configured by U-Boot) board has only one Ethernet as configured from the factory from NXP.  We had selected IMXUL_CLK_ENET_REF.  Linux was able to attach to the PHY when this was changed to IMX6UL_CLK_ENET2_REF.  How as one to know that the ENET2_REF was to be used?  The ENET2 is not connected otherwise.
Here are the pinmux settings:

// Need to add MDIO pins to pinctrl_enet1.
&pinctrl_enet1 {
    // Alternate drive strength:
    // MX6UL_PAD_ENET1_TX_CLK__ENET1_REF_CLK1      0x40000014
    // Reference design does not work:
    // MX6UL_PAD_ENET1_TX_CLK__ENET1_REF_CLK1      0x4001b031
    fsl,pins = <
        MX6UL_PAD_ENET2_RX_DATA1__ENET1_MDC        0x0001B008
        MX6UL_PAD_ENET2_RX_DATA0__ENET1_MDIO       0x0001B808
        MX6UL_PAD_ENET1_RX_EN__ENET1_RX_EN         0x00011000
        MX6UL_PAD_ENET1_RX_ER__ENET1_RX_ER         0x00011000
        MX6UL_PAD_ENET1_RX_DATA0__ENET1_RDATA00    0x00011000
        MX6UL_PAD_ENET1_RX_DATA1__ENET1_RDATA01    0x00011000
        MX6UL_PAD_ENET1_TX_EN__ENET1_TX_EN         0x00000018
        MX6UL_PAD_ENET1_TX_DATA0__ENET1_TDATA00    0x00000018
        MX6UL_PAD_ENET1_TX_DATA1__ENET1_TDATA01    0x00000018
        MX6UL_PAD_ENET1_TX_CLK__ENET1_REF_CLK1     0x40000020
        MX6UL_PAD_GPIO1_IO09__GPIO1_IO09           0x0001B0B0
    >;
};

And fec2 is disabled, which seems to correspond to ENET2.

The part we are using is:
MCIMX6Y1CVM05AB

[joanxie]

did you dump the clock to check? it seems the IMX6UL_CLK_ENET_REF is disable but IMX6UL_CLK_ENET2_REF works when you disable the ethenet in the uboot

[JohnKlug]

I apologize for my previous post which is not correct, as I did not look at the U-Boot I was using.  Unfortunately using IMX6UL_CLK_ENET2_REF Does not fix the issue.  I will attach the clk_summary from the kernel debugfs.

ubooteth.txt:     U-Boot sets up Ethernet
ubootnoeth.txt:   U-Boot has Ethernet disabled

The first clock that is off when U-Boot does not set up Ethernet is pll6_enet.

[joanxie]

it seems the clock is disable, refer to the reference manual, pls check if you enable the enet clock in the kernel, you can read CG6 of this register

CCM Clock Gating Register 0 (CCM_CCGR0)

Address: 20C_4000h base + 68h offset = 20C_4068h

[JohnKlug]

The pll6_enet clock is disabled by Linux during boot. If I turn the clock on manually, the following command disables the clock:
ip link set dev eth0 up
ip: SIOCSIFFLAGS: No such device
I suspect that if the driver cannot talk to the phy, the clock is then disabled by the driver.

I have three revisions of this board that are supposed to be functionally the same as far as Ethernet is concerned with the same CPU and Phy, but both copies of one of the revisions of the board fail. The other two board revisions work with Ethernet disabled in U-Boot.

This issue may be hardware implementation related. But the boards that require U-Boot initialization show the same performance with iperf3, 94MBits/S, and no boards show errors in /proc/net/dev.

[joanxie]

I have three revisions of this board that are supposed to be functionally the same as far as Ethernet is concerned with the same CPU and Phy

>so two revisions can work and one revision is failed, but 3 revisions are the same PHY and CPU, right? if yes, which PHY do you use? KSZ8041NL ? since you think this is HW issue, did you check your HW?

This shows that the driver will take the clock up and down, and this works even on boards that require U-Boot to initialize the PHY

> refer to the dump information and your description, if uboot didn't init this phy, the clock would be closed, then the driver in the kernel would lost the phy, so how about enable the clock in the driver to force the phy link up? you can consult KSZ8041NL  vendor to confirm what difference between KSZ8041NL  and KSZ8091RNB, since TI and KSZ8091RNB works when uboot didn't init, only KSZ8041NL is failed

[JohnKlug]

I have four different boards we have made. Two of the boards use the KSZ8091RNB, one used the TI, and the problem board uses the KSZ8041NL.

The most likely reason for this issue in my mind is that the Linux FEC driver is preserving settings done by U-Boot, and if they are not set by U-Boot, possibly the defaults are being used, which are not correct for this PHY. For instance in the past I have noticed that the SOC power supply driver for the i.MX CPU in Linux does not change driver settings set up by U-Boot.

Beyond reading the driver code, if we go back to Micrel I think we will need to verify the MDIO bus timing is correct for this PHY in Linux.

As you mention, I don't think the clock being down is the real issue. The clock is down because the driver cannot talk to the PHY, so the clock is brought down as a result.

[joanxie]

you can compare the phy driver to check the difference, it seems the KSZ8091RNB or TI driver would enable the clock in the driver, but KSZ8041NL wouldn't, so if you disable in the uboot, the clock would be disabled, then the phy is down, it seems this issue isn't related to the imx side, you need check the phy driver

[JohnKlug]

We have already verified electrically that the MDIO bus comes up and the MDIO bus attempts to get to the PHY.  The clock is brought down by the driver when the driver gets not response.  It cannot be the PHY driver if no phy is found at all.

[joanxie]

did you trace the code between KSZ8091RNB and KSZ8041NL? when the clock is disabled when board enter to the linux? try to print more debug information to confirm when the clock is enable in the good phy, if you don't want to trace this, just enable the clock before probe the phy

[JohnKlug]

With one of the boards that cannot reach the phy without U-Boot initialization I see:

bash# devmem2 0x20c4068 w  
/dev/mem opened.
Memory mapped at address 0x76f35000.
Read at address  0x020C4068 (0x76f35068): 0xC0C03F0F
bash# ip link set dev eth0 down
bash# devmem2 0x20c4068 w  
/dev/mem opened.
Memory mapped at address 0x76f2c000.
Read at address  0x020C4068 (0x76f2c068): 0xC0C00F0F
bash# ip link set dev eth0 up  
[  131.417830] Micrel KSZ8041 2188000.ethernet-1:01: attached PHY driver (mii_bus:phy_addr=2188000.ethernet-1:01, irq=POLL)
bash# [  132.491300] fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[  132.499259] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

bash# devmem2 0x20c4068 w  
/dev/mem opened.
Memory mapped at address 0x76f8d000.
Read at address  0x020C4068 (0x76f8d068): 0xC0C03F0F

This shows that the driver will take the clock up and down, and this works even on boards that require U-Boot to initialize the PHY.