Reconfiguring SerDes for XFI or SGMII

[stadium_aquino]

I would like to select between XFI and SGMII at runtime on an LS1046A. I have managed to get both XFI and SGMII working independently by setting the PRCTL in the RCW. However, I have not been able to reconfigure the protocol. I am aware that this is not specifically supported, but I believe I am performing all steps as documented in the reference manual.

For prototyping, I am using an LS1046ARDB. I am testing with lane 1/C (MAC10), which has an SFP slot. I use the PRCTLs 0x1133 and 0x1333 to test XFI and SGMII (respectively). I am using a 10BASE-T/100BASE-TX/1000BASE-T module to test SGMII, and a 10GBASE-T module to test XFI. I have had no issues with link integrity.

To determine whether a lane is up, I perform a ping test. Curiously, even when the ping test fails, the PCS still shows the link as up (as determined from the pcs/an stat1 c45 mdio registers when using XFI, or from the BMSR c22 mdio register when using SGMII). Additionally, the far end of the link also shows the link as up. This is confusing to me, because if the SFP can talk to the far end, and the PCS can talk to the SFP, then packet transfer should work.

To reprogram lane C to XFI, I perform the following writes:

write(0x11000000, 0x1ea0220); // Disable SGMIIC
write(0x000000bf, 0x1ea1824); // Clear SGMIICCR1.SGPCS_EN
write(0xaa011080, 0x1ea0880); // Put lane C into reset
ndelay(50);
write(0x21000000, 0x1ea022c); // Enable XFIA
write(0x33051500, 0x1ea0880); // Program lane C for SGMII
write(0x10000019, 0x1ea0884);
write(0x0000045f, 0x1ea0890);
write(0x10233007, 0x1ea0898);
write(0x00000400, 0x1ea08a0);
ndelay(120);
write(0x33651500, 0x1ea0880); // Bring lane C out of reset

Note that all writes are read-modify-writes, leaving any reserved bits as they are.

To do the inverse operation (reprogramming to XFI), I perform the following writes:

write(0x01000000, 0x1ea022c); // Disable XFIA
write(0x33051500, 0x1ea0880); // Put lane C into reset
ndelay(50);
write(0x11100000, 0x1ea0220); // Enable SGMIIC
write(0x0000083f, 0x1ea1824); // Set SGMIICCR1.SGPCS_EN
write(0xaa011080, 0x1ea0880); // Program lane C for XFI
write(0x101c4019, 0x1ea0884);
write(0x0f0fc01f, 0x1ea0890);
write(0x00003006, 0x1ea0898);
write(0x39000400, 0x1ea08a0);
ndelay(120);
write(0xaa611080, 0x1ea0880); // Bring lane C out of reset

Once again, note that all writes are read-modify-writes, leaving any reserved bits as they are. I have simply omitted the reads for brevity.

I am unsure of the correct ordering for the PCCR and SGMIICCR1 writes. The datasheet does not specify whether they should happen before/during/after the lane reset. I have opted to disable/disconnect the protocol controller before resetting the lane, and to enable/connect the controller during the lane reset. However, I also tried disabling the protocol controller while the lane is reset, which is what is done by the Linux Lynx 28g phy driver. I also tried the reset sequence in https://community.nxp.com/t5/Layerscape/LS1043A-SERDES-PLL-reset-causes-PHY-link-fail/m-p/1375359#M9... but I did not have success with that.

To investigate this problem, I performed several memory dumps for each configuration and compared them. In the following three-way diff, files 1 and 2 are from boots where PRCTL=0x1333 and no reconfiguration was performed. File 3 is from a boot where PRCTL=0x1133 and lane C was reconfigured to XFI. When registers are different for all three boots, we can assume that they always change from boot-to-boot (such as calibration results). However, if registers match for boots 1 and 2, but different for boot 3, this represents a misconfiguration of some kind.

====1
1:4c
0000000c: 008c0000 ....
2:4c
3:4c
0000000c: 008b0000 ....

This is a difference in PLL1CR2, which is reserved. Interestingly, it is the same for both a reconfiguring boot and a non-reconfiguring boot. I suspect this difference is not interesting.

====2
1:41c
3:41c
000000a0: 28008000 (...
2:41c
000000a0: 48008000 H...

This is a difference in a reserved bit of RCALCR.

====3
1:518c
2:518c
00000814: 0f0f1128 ...(
3:518c
00000814: 0f0f10a8 ....
====
1:525,526c
00000830: 60418828 `A.(
00000834: 00000e00 ....
2:525,526c
00000830: 6055d96d `U.m
00000834: 00000a00 ....
3:525,526c
00000830: 6096dd7d `..}
00000834: 00000080 ....

These are all differences in reserved or calibration registers.

====
1:534c
00000854: 0f0f0f28 ...(
2:534c
00000854: 0f0f1028 ...(
3:534c
00000854: 0f0f0fa8 ....
====
1:541,542c
00000870: 60d7d14d `..M
00000874: 00007f80 ....
2:541,542c
00000870: 604db8eb `M..
00000874: 00007480 ..t.
3:541,542c
00000870: 60e00210 `...
00000874: 00007f00 ....
====3
1:550c
2:550c
00000894: 0f0f0f28 ...(
3:550c
00000894: 0f0f0ea8 ....
====
1:557,558c
000008b0: 60828c38 `..8
000008b4: 00000b40 ...@
2:557,558c
000008b0: 604db8eb `M..
000008b4: 000017c0 ....
3:557,558c
000008b0: 60bf7fc7 `...
000008b4: 00007a00 ..z.
====
1:566c
000008d4: 0e0030a8 ..0.
2:566c
000008d4: 0e0031a8 ..1.
3:566c
000008d4: 0e003128 ..1(
====
1:573,575c
000008f0: 60f45355 `.SU
000008f4: 00007d00 ..}.
000008f8: 40800000 @...
2:573,575c
000008f0: 606e3af3 `n:.
000008f4: 00005f40 .._@
000008f8: 01800000 ....
3:573,575c
000008f0: 6008a49a `...
000008f4: 00007ec0 ..~.
000008f8: 01800000 ....

I also performed the another comparison for the other way around. In this three-way diff, files 1 and 2 are from boots where PRCTL=0x1133 and no reconfiguration was performed. File 3 is from a boot where PRCTL=0x1333 and lane C was reconfigured to SGMII. To reduce repetition, I have ommitted differences in LNmRECR1, LNmTCSR0, LNmTCSR1, and LNmTCSR2.

====3
1:41c
2:41c
000000a0: 48008000 H...
3:41c
000000a0: 28008000 (...

This is the same as above.

====3
1:1546c
2:1546c
00001824: 0000003f ...?
3:1546c
00001824: 000000bf ....

This is a difference in a reserved bit of SGMIICCR1.

Aside from these differences in reserved bits (and in the test/calibration registers), there is no difference between boots where a serdes is recalibrated and boots where it is not. This suggest some kind of ordering or timing issue. Alternatively, it could be that there is some (undocumented) register which I need to set when switching PLLs. I could also have mis-entered a field, though I don't think that is likely since the registers match in the working case.