LS1046A: No Ethernet after SerDes Lane reconfiguration

[tmoos]

I'm still working on getting the Broadcom "BCM84891L" 10GbE PHYs up and running (a related thread is  https://community.nxp.com/t5/Layerscape/How-to-connect-LS1046A-to-10G-PHY-using-USXGMII/m-p/1490248/...).

Meanwhile I am able to operate the PHY in 1G mode (using SGMII with SerDes protocol 0x3333) and in 10G mode (using XFI with SerDes Protocol 0x1133). Now I need to switch between 1G and 10G during runtime. Since I can't change the RCW during runtime, I try to reconfigure the SerDes lanes, but with no success so far.

Sorry for the lengthy post, but I believe that many details are neccessary to find the problem...

Both RCWs are nearly identical (differences are bold):

# 1G (single PLL):
0c140012 0e000000 00000000 00000000
33335559 c000500e 60000000 c1000000
00000000 00000000 00000000 00028c00
20004000 24401102 00000096 00000001

# 10G (separate PLLs):
0c140012 0e000000 00000000 00000000
11335559 c0005006 60000000 c1000000
00000000 00000000 00000000 00028c00
20004000 24401102 00000096 00000001

[solved] Problem 1: Can I switch PLL clocks used by the Lanes?

2023-05-03 Update: After reconfiguring, I measured SGMII signals with the correct frequency. The PHY reports "link up" on the SGMII link. I believe that I successfully solved this problem.

PLL1 is running with 125 MHz and PLL2 with 156.25 MHz.

The 1G-RCW sets RCW[SRDS_REFCLK_SEL_S1]=1, i.e. Lanes C and D use PLL2, and PLL2 uses PLL1=125MHz as source (this works).

The 10G-RCW sets RCW[SRDS_REFCLK_SEL_S1]=0, i.e. Lanes C and D use PLL2, and PLL2 uses 156.25MHz as source (this works).

Now I boot with the 10G RCW and then reconfigure Lane D to SGMII. In the process I set LNDGCR0[RPLL_LES]=1 and LNDGCR0[TPLL_LES]=1 to use PLL1. This differs from the configuration which the 1G-RCW uses (it uses LNDGCR0[RPLL_LES]=0). And it does NOT work.

Of course, I reconfigure a lot more than only RPLL_LES and TPLL_LES (see below). However I do not touch the PLL configuration, i.e. I don't need to obey the "PLL Reset and Reconfiguration" section.

Question: Is it allowed to switch Lane D from PLL2 to PLL1? Are there Pitfalls that I may have overlooked?

Problem 2: Lane reconfiguration: Is my approach okay?

I start the U-Boot with the 10G-RCW (XFI). The ping command in U-Boot works as expected (using a 10G remote PC). Then I reconfigure the lanes to SGMII. Thereafter ping does NOT work. Then I reconfigure back to XFI, and ping works again. Therefore I suspect that the reconfiguration sequence works, but that I miss something regarding the SGMII configuration.

Here is how I switch to SGMII-Configuration (see below for an explanation of set_bit32(), clear_bit32() and change_bits32()) :

mac_disable();

// Step 1: Put Lanes into reset
value = byteswap4(*serdes_lnagcr0);
clear_bit32(&value, LNxGCR0__TRST_B);
clear_bit32(&value, LNxGCR0__RRST_B);
*serdes_lnagcr0 = byteswap4(value);

value = byteswap4(*serdes_lnbgcr0);
clear_bit32(&value, LNxGCR0__TRST_B);
clear_bit32(&value, LNxGCR0__RRST_B);
*serdes_lnbgcr0 = byteswap4(value);

value = byteswap4(*serdes_lncgcr0);
clear_bit32(&value, LNxGCR0__TRST_B);
clear_bit32(&value, LNxGCR0__RRST_B);
*serdes_lncgcr0 = byteswap4(value);

value = byteswap4(*serdes_lndgcr0);
clear_bit32(&value, LNxGCR0__TRST_B);
clear_bit32(&value, LNxGCR0__RRST_B);
*serdes_lndgcr0 = byteswap4(value);

// Step 2: Wait at least 50 ns
ndelay(50);

// Step 3: Change the desired per-lane settings
value = byteswap4(*serdes_lndgcr0);
set_bit32(&value, LNxGCR0__RPLL_LES); // Use PLL 1 (instead of PLL 2)
set_bit32(&value, LNxGCR0__TPLL_LES); // Use PLL 1 (instead of PLL 2)
*serdes_lndgcr0 = byteswap4(value);

value = byteswap4(*serdes_pccrb);
change_bits32(&value, PCCRB__XFIA_CFG__MSB, PCCRB__XFIA_CFG__LSB, 0x0);
*serdes_pccrb = byteswap4(value);

value = byteswap4(*serdes_pccr8);
change_bits32(&value, PCCR8__SGMIID_CFG__MSB, PCCR8__SGMIID_CFG__LSB, 0x1);
*serdes_pccr8 = byteswap4(value);

value = byteswap4(*serdes_lndgcr0);
change_bits32(&value, LNxGCR0__RRAT_SEL__MSB, LNxGCR0__RRAT_SEL__LSB, 0x2);
change_bits32(&value, LNxGCR0__TRAT_SEL__MSB, LNxGCR0__TRAT_SEL__LSB, 0x2);
change_bits32(&value, LNxGCR0__PROTS__MSB, LNxGCR0__PROTS__LSB, 0x1);
clear_bit32(&value, LNxGCR0__IF20BIT_EN);
*serdes_lndgcr0 = byteswap4(value);

value = byteswap4(*serdes_lndgcr1);
change_bits32(&value, LNxGCR1__REIDL_TH_MSB, LNxGCR1__REIDL_TH_LSB, 0x1);
clear_bit32(&value, LNxGCR1__REIDL_EX_MSB);
change_bits32(&value, LNxGCR1__REIDL_EX_SEL__MSB,
                      LNxGCR1__REIDL_EX_SEL__LSB, 0x3);
set_bit32(&value, LNxGCR1__REIDL_ET_MSB);
change_bits32(&value, LNxGCR1__REIDL_ET_SEL__MSB,
                      LNxGCR1__REIDL_ET_SEL__LSB, 0x0);
*serdes_lndgcr1 = byteswap4(value);

value = byteswap4(*serdes_lndrecr0);
clear_bit32(&value, LNxRECR0__RXEQ_BST);
change_bits32(&value, LNxRECR0__GK2OVD__MSB, LNxRECR0__GK2OVD__LSB, 0xf);
change_bits32(&value, LNxRECR0__GK3OVD__MSB, LNxRECR0__GK3OVD__LSB, 0xf);
set_bit32(&value, LNxRECR0__GK2OVD_EN);
set_bit32(&value, LNxRECR0__GK3OVD_EN);
clear_bit32(&value, LNxRECR0__OSETOVD_EN);
change_bits32(&value, LNxRECR0__BASE_WAND__MSB,
                      LNxRECR0__BASE_WAND__LSB, 0x0);
change_bits32(&value, LNxRECR0__OSETOVD__MSB,
                      LNxRECR0__OSETOVD__LSB, 0x1f);
*serdes_lndrecr0 = byteswap4(value);

value = byteswap4(*serdes_lndtecr0);
change_bits32(&value, LNxTECR0__TEQ_TYPE__MSB,
                      LNxTECR0__TEQ_TYPE__LSB, 0x0);
clear_bit32(&value, LNxTECR0__SGN_PREQ);
change_bits32(&value, LNxTECR0__RATIO_PREQ__MSB,
                      LNxTECR0__RATIO_PREQ__LSB, 0x0);
clear_bit32(&value, LNxTECR0__SGN_POST1Q);
change_bits32(&value, LNxTECR0__RATIO_POST1Q__MSB,
                      LNxTECR0__RATIO_POST1Q__LSB, 0x0);
change_bits32(&value, LNxTECR0__ADPT_EQ__MSB,
                      LNxTECR0__ADPT_EQ__LSB, 0x30);
change_bits32(&value, LNxTECR0__AMP_RED__MSB, LNxTECR0__AMP_RED__LSB, 0x6);
*serdes_lndtecr0 = byteswap4(value);

value = byteswap4(*serdes_lndttlcr0);
change_bits32(&value, LNxTTLCR0__FLT_SEL__MSB,
                      LNxTTLCR0__FLT_SEL__LSB, 0x39);
*serdes_lndttlcr0 = byteswap4(value);

value = byteswap4(*serdes_sgmiidcr1);
set_bit32(&value, SGMIIxCR1__SGPCS_EN);
*serdes_sgmiidcr1 = byteswap4(value);

// Lane Step 4: Wait at least 120 ns
ndelay(120);

// Lane Step 5: Take the lane(s) out of reset
value = byteswap4(*serdes_lnagcr0);
set_bit32(&value, LNxGCR0__TRST_B);
set_bit32(&value, LNxGCR0__RRST_B);
*serdes_lnagcr0 = byteswap4(value);

value = byteswap4(*serdes_lnbgcr0);
set_bit32(&value, LNxGCR0__TRST_B);
set_bit32(&value, LNxGCR0__RRST_B);
*serdes_lnbgcr0 = byteswap4(value);

value = byteswap4(*serdes_lncgcr0);
set_bit32(&value, LNxGCR0__TRST_B);
set_bit32(&value, LNxGCR0__RRST_B);
*serdes_lncgcr0 = byteswap4(value);

value = byteswap4(*serdes_lndgcr0);
set_bit32(&value, LNxGCR0__TRST_B);
set_bit32(&value, LNxGCR0__RRST_B);
*serdes_lndgcr0 = byteswap4(value);

// Finally, enable MAC again
mac_enable();

After that I change some MDIO registers:

value = memac_mdio_read(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x14);
value &= ~0x1f; // Set all non-reserved bits to 0
value |= 0x9; // duplex=0, speed=0b10, an=0, en=1
memac_mdio_write(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x14, value);

value = memac_mdio_read(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x4);
value |= 0x1; // SGMII Mode=1
memac_mdio_write(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x4, value);

value = memac_mdio_read(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x0);
value &= ~0xffc0; // Set all non-reserved bits to 0
value |= 0x8000; // Reset=1
memac_mdio_write(dev_mdio_mac_9, 0x0, MDIO_DEVAD_NONE, 0x0, value);

Here is how I disable the MAC:

// COMMAND_CONFIG[RXSTP]=1
value = byteswap4(*emac9_command_config);
set_bit32(&value, COMMAND_CONFIG__RXSTP);
*emac9_command_config = byteswap4(value);

// wait for IEVENT[RX_EMPTY] to be set
while(!(byteswap4(*emac9_ievent) & (0x1 << IEVENT__RX_EMPTY)))
{
    byteswap4(*emac9_ievent));
}

//COMMAND_CONFIG[RX_EN]=0
value = byteswap4(*emac9_command_config);
clear_bit32(&value, COMMAND_CONFIG__RX_EN);
*emac9_command_config = byteswap4(value);

// Wait for IEVENT[TX_EMPTY] to be set
while(!(byteswap4(*emac9_ievent) & (0x1 << IEVENT__TX_EMPTY)))
{
}

// 3) Write COMMAND_CONFIG[TX_EN]=0
value = byteswap4(*emac9_command_config);
clear_bit32(&value, COMMAND_CONFIG__TX_EN);
*emac9_command_config = byteswap4(value);

Here is how I enable the MAC again:

// Set COMMAND_CONFIG[SWR]
value = byteswap4(*emac9_command_config);
set_bit32(&value, COMMAND_CONFIG__SWR);
clear_bit32(&value, COMMAND_CONFIG__RXSTP);
*emac9_command_config = byteswap4(value);

// Initialize IF_MODE register
value = byteswap4(*emac9_if_mode);
value = if_mode_value;
*emac9_if_mode = byteswap4(value);

// Set STATN_CONFIG[CLR]
value = byteswap4(*emac9_statn_config);
set_bit32(&value, STATN_CONFIG__CLR);
*emac9_statn_config = byteswap4(value);

udelay(1); // wait until reset is done (32 FMAN cycles needed, 1 usec is plenty!)

// Clear IEVENT register
*emac9_ievent = 0;

// Initialize IMASK register
*emac9_imask = 0;

// Set COMMAND_CONFIG[RX_EN], set COMMAND_CONFIG[TX_EN]
value = byteswap4(*emac9_command_config);
set_bit32(&value, COMMAND_CONFIG__RX_EN);
set_bit32(&value, COMMAND_CONFIG__TX_EN);
*emac9_command_config = byteswap4(value);

I my approach okay? Is it legitimate to change the Lanes that way? Do I correctly disable and re-start the MAC?

[solved] Problem 3: How to analyze the hardware state?

2023-05-03 Update: I read out the packet counters of BMI, MAC and PCS. The BMI counters increase for each sent packet. The MAC counters increase if the loopback is enabled (XGLP), otherwise they do not increase. The PCS counters never increase (even if the PCS loopback is enabled).

Therefore I believe that the packets reach the MAC, but they do not reach the PCS. The connection between MAC and PCS seems to be broken.

After configuring from XFI to SGMII I can see that U-Boot is feeding packets into the BMAN, but the packets are not transmitted by the MAC. Can I determine if BMAN is properly "connected" to the MAC? Is there a "Link" status or something? Are there other important status bits?

Appendix: Helper functions and naming conventions

Macro names:

• First part is the register name, 2nd part is the bit name, sometimes there is a 3rd part for MSB/LSB. The parts are separated by double underlines.
• Example: SGMIIxCR1__SGPCS_EN: SGMIIxCR1 are the registers (SGMIIaCR1, SGMIIbCR1, etc.), and SGPCS_EN is the bit within the register.
  
• Example: LNxTECR0__ADPT_EQ__MSB: LNxTECR0 is the register, ADPT_EQ are the bits within that register. Since ADPT_EQ spans multiple bits I use MSB and LSB for the highest and loweset bit.

Bit manipulation:

• set_bit32() and clear_bit32() perform a read-modify-write to set/clear a single bit. The parameter is the bit number as stated in LS1046ARM, e.g. LNDGCR0[TPLL_LES] is bit 4.
• change_bits32() works similar, but changes multiple adjacent bits. E.g. change_bits32(&value, LNxTECR0__ADPT_EQ__MSB, LNxTECR0__ADPT_EQ__LSB, 0x30) changes bits 18 to 23 to the value 0x30 without touching other bits in the register.

[tmoos]

Meanwhile I gave up. I didn't manage to switch between SGMII and XFI. However, NXP posted on a mailing list[1] that there are writable (!) debug register for the RCW where the RCW can be overwritten at runtime. Maybe that provides a solution. Note that I did not try this, and it is not guaranteed that these register solve the problem.

I managed to switch between SGMII and 2.5G SGMII. This enables me to implement a system with 100M, 1G and 2.5G.

[1] https://lore.kernel.org/linux-phy/20230811163637.bs7a46juasjgnmf4@skbuf/T/#m1339f385d9a331df753426c0...

[stadium_aquino]

I also attempted to solve this problem, but no dice. I was able to do "minor" reconfiguration (same PROTS) but I was unable to get "major" reconfiguration (different PROTS). From what I can tell, this is a PCS issue. The serdes gets CDR_LOCK, and the mEMAC loopback works (XGLP), but the PCS never shows a link up. My efforts in this area are at

https://lore.kernel.org/linux-phy/20230413160607.4128315-1-sean.anderson@seco.com/T/#m4a4bb770d2c123...

I ended up using two separate RCWs for 1G and 10G and rebooting in between. I use my driver to reconfigure the clocks, since the clocking for 3333 is incompatible with 1133. You will need to ensure RESET_REQ_B is disconnected from HRESET_B because the SoC tries to reset itself if it can't lock the PLLs (ugh). I found a generic GPIO worked great to request resets.

[tmoos]

I reproduced the loopback behavior: When setting XGLP, the MAC is working.

However, I just switched from XFI to SGMII, i.e. the MAC should be in GMII mode (not 10G mode), and the XGLP bit should not be effective. Loopback should have been enabled with IF_MODE[RLP] instead. But RLP does not work, only XGLP does.

It seems that the MAC is still in 10G mode?!

[stadium_aquino]

> However, I just switched from XFI to SGMII, i.e. the MAC should be in GMII mode (not 10G mode), and the XGLP bit should not be effective. Loopback should have been enabled with IF_MODE[RLP] instead. But RLP does not work, only XGLP does.

RLP is only for RGMII. From the MAC's perspective, everything else uses XGMII. You can tell because for IFMODE there is only 00 (10G) and 10 (RGMII)*. So I think it is normal that only XGLP works, since you are using SGMII and not RGMII.

*(If you look at the T4240 DPAARM you can see that IFMODE 01 is MII.)

[tmoos]

I just found your patch some days ago, and I can confirm that loopback works. But no packets are sent out to the Ethernet PHY. It seems I have the exact same situation as you had. But I have no solution either.

Thanks for your comment on this!