How to configure Ethernet (SGMII@MAC9) on LS1043A

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to configure Ethernet (SGMII@MAC9) on LS1043A

Jump to solution
15,438 Views
andreloddenkemp
Contributor II

Hello everybody,

 

I am here looking for help with the Ethernet configuration of our board. The CPU on our board is a QorIQ LS1043A ARMv8. Serdes Lane A is connected to a Broadcom Ethernet switch on the board via SGMII. There is no real PHY device involved here, the LS1043A Serdes is directly connected to the switch Serdes. The switch is capable of auto-negotiating with SGMII and 1000BaseX connections and by default set to SGMII.

 

We are using the Yocto Linux SDK 2.0. We are deploying an u-boot bootloader, a Linux kernel and a filesystem via SD card to our board. As a starting point I used the LS1043A-RDB BSP, from the NXP evaluation board.


Unfortunately the Ethernet interface in the Linux is still not working properly. In the Linux I can bring it up with "ifconfig" and I can also see an increasing "Tx counter" there, but that's it, nothing else. There seems to be no link and when I look at the interface with ethtool, I get the following:

Settings for fm1-mac9:
Supported ports: [ MII ]
Supported link modes: Not reported
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Advertised link modes: Not reported
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Speed: 10Mb/s
Duplex: Half
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0xffffffff (-1)
drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000
Link detected: no


So here are my questions:

 

1. Is it correct that we have to configure it as an internal PHY, because of how the Ethernet interface is connected to the switch? I think in the reference manual of the LS1043a this is referred to as internal or PCS PHY.

 

2. The fman.txt document in the device tree binding documentation (http://lxr.free-electrons.com/source/Documentation/devicetree/bindings/powerpc/fsl/fman.txt?v=4.6), tells me that I have to use the "fsl,fman-internal-mdio;" parameter for my internal mdio bus as well as the "pcsphy-handle" parameter for my MAC. However, these device tree parameter only exist in newer kernels (4.6+), I think. Am I misunderstandn something here? Or is it really that new? We are at kernel 4.1x, because that is the most current kernel out of the Freescale git repository.

3. How do I need to configure my device tree? Currently I have set it up like this (only the relevant entries):

fman0: fman@1a00000 {
   #address-cells = <1>;
   #size-cells = <1>;
   cell-index = <0>;
   compatible = "fsl,fman", "simple-bus";
   ranges = <0x0 0x00 0x1a00000 0x100000>;
   reg = <0x00 0x1a00000 0x0 0x100000>;
   interrupts = <0 44 0x4>, <0 45 0x4>;
   clocks = <&clockgen 3 0>;
   clock-names = "fmanclk";

   fman0_10g_rx0: port@90000 {
      cell-index = <0>;
      compatible = "fsl,fman-port-10g-rx";
      reg = <0x90000 0x1000>;
   };

 

   fman0_10g_tx0: port@b0000 {
      cell-index = <0>;
      compatible = "fsl,fman-port-10g-tx";
      reg = <0xb0000 0x1000>;
      fsl,qman-channel-id = <0x800>;
   };

 

   fm1mac9: ethernet@f0000 {
      cell-index = <0>;
      compatible = "fsl,fman-memac";
      reg = <0xf0000 0x1000>;
      fsl,port-handles = <&fman0_10g_rx0 &fman0_10g_tx0>;
      /* FM1 mEMAC9: 1AF_000h, LS1043ARM_Rev2.pdf, p. 548 */
      /* SGMII 1000 @ MAC9 */
      ethernet@f0000 {
         phy-handle = <&sgmii_phy>;
         phy-connection-type = "sgmii";
      };
   };

 

      /* MDIO9 for EMAC9: 1AF_1000h, LS1043ADAARM.pdf, p. 500 */

   mdio9: mdio@f1000 {
      #address-cells = <1>;
      #size-cells = <0>;
      compatible = "fsl,fman-memac-mdio";
      reg = <0xf1000 0x1000>;
      fsl,fman-internal-mdio;
      sgmii_phy: ethernet-phy@0 {
         reg = <0x0>;
      };
   };

};

 

4. I also tried to set the ethernet interface up as a fixed link. It was nearly the same result. I could bring the interface up, but only TX counter increased, nothing else. It looked like this:

fm1mac9: ethernet@f0000 {
   cell-index = <0>;
   compatible = "fsl,fman-memac";
   reg = <0xf0000 0x1000>;
   fsl,port-handles = <&fman0_10g_rx0 &fman0_10g_tx0>;
   /* FM1 mEMAC9: 1AF_000h, LS1043ARM_Rev2.pdf, p. 548 */
   /* SGMII 1000 @ MAC9 */
   ethernet@f0000 {
      fixed-link {
         speed = <1000>;
         full-duplex;
      };
      phy-connection-type = "sgmii";
   };
};

 

5. We only have lane A connected to the switch, all other lanes are unconnected. For that I am using Serdes Protocol 3333, because it is one of the many, which provide SGMII on lane A. More specifically, it provides SGMII on all the lanes. It seems only mac9 can be used on this lane. Any concern regarding this? I think I should be fine here. My RCW is as follows:

00000000: 08100010 0a000000 00000000 00000000
00000010: 33330002 00004002 60025000 c1002000
00000020: 00000000 00000000 00000000 0003effc
00000030: 00000000 00001102 00000096 00000001


If anyone already has experience with the LS1043A or with the topic in general, I would be very glad to get some feedback on my issues. Am I on the wrong track? Or is just something misconfigured? Do I need to use the newer kernel version?

 

Attached to this post, you also find some more (hopefully useful) information regarding our board. See "dev_tree.dts" for my device tree. See "bootlog.txt" for a part of the bootlog, where the mdio bus and the mac is probed. Do you need anything else? I will gladly provide more information, if that helps.

 

Best regards,
Andre Loddenkemper

Original Attachment has been moved to: dev_tree.dts.zip

Original Attachment has been moved to: bootlog.txt.zip

Labels (1)
1 Solution
7,666 Views
andreloddenkemp
Contributor II

First off, thanks for the answers! Even though they did not solve my problem, they are greatly appreciated and may help others.

Regarding my problem: It is solved for a while now. We didn't need to configure an internal phy or anything like that. We configured our MAC device in the device tree as this:

/* FM1 mEMAC9: 1AF_000h, LS1043ARM.pdf, p. 696 */
/* SGMII 1000 @ MAC9 */
ethernet@f0000 {
      phy-connection-type = "sgmii";
      fixed-link {
         speed = <1000>;
         full-duplex;
   };
};

We do not have a mdio bus or phy assigned to it. The reason why it was not working was the configuration of our Serdes PLLs. I have created a startup script, which is executed after the boot as this:

# First we need to calibrate the PLLs to enable the ethernet interface
echo "Calibrating PLLs"
# Reset PLL1
devmem 0x1ea0000 32 0xE8454786
# Disable PLL2
devmem 0x1ea0020 32 0xc7454726

After this it works fine. I do not have all the details from the top of my head right now, but if anybody is interested in the details, just say so and I will talk to our hardware engineer to get a memory boost again. Basically we only use Serdes PLL1 and thus we have to first reset PLL1 and then disable the PLL2. We do so by writing into the according CPU registers.

Best regards,

Andre

View solution in original post

0 Kudos
7 Replies
7,667 Views
andreloddenkemp
Contributor II

First off, thanks for the answers! Even though they did not solve my problem, they are greatly appreciated and may help others.

Regarding my problem: It is solved for a while now. We didn't need to configure an internal phy or anything like that. We configured our MAC device in the device tree as this:

/* FM1 mEMAC9: 1AF_000h, LS1043ARM.pdf, p. 696 */
/* SGMII 1000 @ MAC9 */
ethernet@f0000 {
      phy-connection-type = "sgmii";
      fixed-link {
         speed = <1000>;
         full-duplex;
   };
};

We do not have a mdio bus or phy assigned to it. The reason why it was not working was the configuration of our Serdes PLLs. I have created a startup script, which is executed after the boot as this:

# First we need to calibrate the PLLs to enable the ethernet interface
echo "Calibrating PLLs"
# Reset PLL1
devmem 0x1ea0000 32 0xE8454786
# Disable PLL2
devmem 0x1ea0020 32 0xc7454726

After this it works fine. I do not have all the details from the top of my head right now, but if anybody is interested in the details, just say so and I will talk to our hardware engineer to get a memory boost again. Basically we only use Serdes PLL1 and thus we have to first reset PLL1 and then disable the PLL2. We do so by writing into the according CPU registers.

Best regards,

Andre

0 Kudos
7,666 Views
williamanderson
Contributor III

Andre,

   Thanks for the information.  Based on it we started looking at our SERDES status register and see that the PLL1 is not locked.  I would like to hear more details from the hardware engineers if possible.  We are trying to connect to a port  on the 88E6185 that is in chip-to-chip mode.

Thanks

Bill

0 Kudos
7,666 Views
cswarth
Contributor III

Andre,

I'm not sure this will be helpful, but I have a similar configuration on a related processor, the LS1046A. Our SERDES interface is connected directly to a switch without an intervening PHY. In our case the switch is a Marvell part, and I'm not sure of its auto negotiation capabilities.

Unlike you I am not trying to jump directly into Linux. I would like to get the enet functional under u-boot first. I am using u-boot sources directly from the QorIQ repository. I have found these infinitely easier (and faster) to work with than the yocto-based build in the NXP SDK.

Although not directly applicable, I have found that some of the source files for the LS1046AQDS platform helpful in pointing the way to a PHY-less configuration. The eth.c source for that board file demonstrates how to add "fixed link" devices to the flattened device tree (FDT) passed to Linux. Those fixed link devices are critical to PHY-less configuration under Linux.

1. Is it correct that we have to configure it as an internal PHY,    because of how the Ethernet interface is connected to the    switch? I think in the reference manual of the LS1043a this is    referred to as internal or PCS PHY.

Yes, my experience under u-boot is that one must configure the SGMII interface via the internal MDIO Ethernet Management Interfaces. (See the FMAN memory map in the Data Path Acceleration Architecture (DPAA) Reference Manual for the offset to those MDIO interfaces.) Not only did I have to create a MDIO control interface for the base address of one of those MDIO control sets, I also had to modify the code that initializes the MDIO interface. The existing u-boot MDIO interface code in memac_phy.c configures CLKDIV values that won't work on internal MDIO interfaces.

I also had to choose a unique PHY identifier and program it into the DEVAD field of appropriate SGMII control register (in my case that was SGMIIBCR1 for MAC5).

Now I can communicate with our switch under 1000Base-X Gigabit mode, but not yet under SGMII mode.

  • Chris
7,664 Views
williamanderson
Contributor III

Hi Chris,

  I am using the LS1021A and trying to connect with the Marvell 88E8165 Switch over a fixed link.  Were you able to et SGMII connection to work under linux?  

Thanks

Bill

0 Kudos
7,664 Views
cswarth
Contributor III

Yes, William, I was able to get it partially working under u-boot and under Linux.  I am not able to reliably talk directly to our switch (Marvell 88E6185), but I am able to talk over SERDES to a remote PHY.  

We are using the LS1046A which has four lanes on SerDes1.  Our board physically only uses lanes A & C.  We configure RCW SRDS_PRTCL_S1 with '3333' which configured SGMII.9 on lane A and SGMII.5 on lane C.

To get it working under Linux I had to add the appropriate fixed-links to the device tree that Linux sees. In the NXP SDK yocto-based build, I found the ls1046a-rdb device tree specification,

./build_ls1046ardb/tmp/work-shared/ls1046ardb/kernel-source/arch/arm64/boot/dts/freescale/fsl-ls1046a-rdb.dts‍‍

To populate this directory I had to do `bitbake -c clean virtual/kernel` and `bitbake -c patch virtual/kernel`.

Then the `./build_ls1046ardb/tmp/work-shared/ls1046ardb/kernel-source` was populated.

I copied the fsl-ls1046a-rdb.dts file and changed the &fman0 entry to include only the two eMACS I am using and to turn them into fixed-links.

&fman0 {          /* EMAC5 (2.5/1Gbps) (mEMAC) */         ethernet@e8000 {                 phy-handle = <&sgmii_phy1>;                 phy-connection-type = "sgmii";                 fixed-link {                     speed = <1000>;                     full-duplex;                 };         };          /* EMAC9(10/2.5/1Gbps) (mEMAC 9) */         ethernet@f0000 { /* 10GEC1 */                 phy-handle = <&aqr106_phy>;                 phy-connection-type = "xgmii";                 fixed-link {                     speed = <1000>;                     full-duplex;                 };         };          mdio@fc000 {                 rgmii_phy1: ethernet-phy@1 {                         reg = <0x1>;                 };                 rgmii_phy2: ethernet-phy@2 {                         reg = <0x2>;                 };                 sgmii_phy1: ethernet-phy@3 {                         reg = <0x3>;                 };                 sgmii_phy2: ethernet-phy@4 {                         reg = <0x4>;                 };         };          mdio@fd000 {                 aqr106_phy: ethernet-phy@1 {                         compatible = "ethernet-phy-ieee802.3-c45";                         interrupts = <0 131 4>;                         reg = <0x0>;                 };         }; };

It was critical to look at the QorIQ LS1046A Data Path Acceleration Architecture (DPAA) Reference Manual.  You'll have to find the equivalent for your processor.  From that manual I began to understand that each MAC has an internal MDIO bus.  There are also external MDIO buses, but you want to use the internal MDIO bus for each MAC. The  ethernet@e8000 and ethernet@f0000 in the device tree above point to the INTERNAL MDIO interfaces for those MACs.  That is crucial.

7,666 Views
cswarth
Contributor III

Sorry about the formatting of that patch.  This posting system really sucks.  stackoverflow is a much better place to edit and answer questions like this.

0 Kudos
7,664 Views
williamanderson
Contributor III

Chris for the information. Hope you get it working on your board. Have you tried setting the Marvell to SGMII mode versus chip to chip mode?

Thank you very much.

Bill

0 Kudos