LS1012A custom board with Broadcom switch, dsa config

[jclasquin]

Hello,

I have the first MAC connected via SGMII to a single Marvell PHY chip.
The second MAC is connected via RGMII to a Broadcom BCM53128 switch chip.

I'm using the DSA version of the b53 driver and SPI instead of MDIO.  I haven't been able to ping a PC connected to any of the 8 ports on the switch even though all the ports are showing up when I do ifconfig or ip commands.  The driver loads OK and the switch is found and configured.    

This is my DTS.

&pcie {
	status = "okay";
};

&duart0 {
	status = "okay";
};

&esdhc0 {
	sd-uhs-sdr104;
	sd-uhs-sdr50;
	sd-uhs-sdr25;
	sd-uhs-sdr12;
	status = "okay";
};

&esdhc1 {
	mmc-hs200-1_8v;
	status = "okay";
};

&i2c0 {
	status = "okay";
};

&sata {
	status = "okay";
};

&pfe {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;

	pfe_mac0: ethernet@0 {
		compatible = "fsl,pfe-gemac-port";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x0>;	/* GEM_ID */
		fsl,pfe-gemac-if-name = "eth0";  
		fsl,mdio-mux-val = <0x0>;
		phy-mode = "sgmii";
		phy-handle = <&sgmii_phy>;
	};

	pfe_mac1: ethernet@1 {
		compatible = "fsl,pfe-gemac-port";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x1>;	/* GEM_ID */
		fsl,pfe-gemac-if-name = "eth1";
		phy-mode = "rgmii-txid";
		fixed-link {
		    speed = <1000>;
         	    full-duplex;
     	 	};
	};
	mdio@0 {
		#address-cells = <1>;
		#size-cells = <0>;

		sgmii_phy: ethernet-phy@1 {   // Marvell PHY at address 1
			reg = <0x1>;         
		};
	};
};

&dspi {
	status = "okay";
	bus-num = <2>;
	pinctrl-names = "default";	

	switch0: ethernet-switch@0 {
		compatible = "brcm,bcm53128";
		reg = <0x0>;  // Chip select 0
		//dsa,member = <0 0>;
		dsa,ethernet = <&pfe_mac1>;
		spi-max-frequency = <20000000>;
		spi-cpol;
		spi-cpha;
		status = "okay";
			
			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				
				port0@0 {
					reg = <0>;
					label = "lan1";
				};
				port1@1 {
					reg = <1>;
					label = "lan2";
				};
				port2@2 {
					reg = <2>;
					label = "lan3";
				};
				port3@3 {
					reg = <3>;
					label = "lan4";
				};
				port4@4 {
					reg = <4>;
					label = "lan5";
				};
				port5@5 {
					reg = <5>;
					label = "lan6";
				};
				port6@6 {
					reg = <6>;
					label = "lan7";
				};
				port7@7 {
					reg = <7>;
					label = "lan8";
				};
				port8@8 {
					reg = <8>;
					label = "cpu";
					ethernet = <&pfe_mac1>;
					fixed-link {
						speed = <1000>;
						duplex-full;
					};
				};
			};
	};
};

The output of "ip link" is shown below. I was expecting eth1 to be connected to the 8 LAN ports, but each "lan" port shows "@eth0". Why? Where is that association made? Can it be changed?  Am I correct in assuming this is why I can't ping?

Thanks,
Joel

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:04:9f:06:75:9f brd ff:ff:ff:ff:ff:ff
4: lan1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: lan2@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: lan3@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: lan4@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: lan5@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: lan6@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: lan7@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
11: lan8@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

[jclasquin]

Thanks for the reply.  I changed my DTS file to match your suggestion, but the result is the same.  "ip link" still shows lan1@eth0, lan2@eth0, etc.  

I will continue to use "of_find_net_device_by_node(ethernet->parent)", but I would like to eventually find a way to fix this without modifying open source code.

Thanks again,
Joel

[yipingwang]

t should not work by moving the order of pfe_mac because it does not use the pfe_mac compatible at all in the driver.

So workaround can be like below and customer could have a try to verify this issue.

diff --git a/drivers/staging/fsl_ppfe/pfe_eth.c b/drivers/staging/fsl_ppfe/pfe_eth.c

index 513438fe6e8c..dc255ece91d5 100644

--- a/drivers/staging/fsl_ppfe/pfe_eth.c

+++ b/drivers/staging/fsl_ppfe/pfe_eth.c

@@ -2526,7 +2526,8 @@ int pfe_eth_init(struct pfe *pfe)

        else

                pfe_errata_a010897 = false;

-       for (ii = 0; ii < NUM_GEMAC_SUPPORT; ii++) {

+

+       for (ii = NUM_GEMAC_SUPPORT -1; ii >= 0; ii--) {

                err = pfe_eth_init_one(pfe, pfe_info, ii);

                if (err)

                        goto err_eth_init;

If this is verified, we will make a plan for formal patch to fix this issue.

[jclasquin]

I made the suggested change.  Unfortunately, ip link still shows the LAN ports at "eth0".  But I can see a difference during boot up.  I added a lot of print statements.  See below.  It is starting with "ii" = 1, and calling pfe_eth_init_one.  Then "ii" = 0.   But "eth0" is still the first device added.  Then "eth1".   The BCM53128 switch is found during the "eth0" initialization.    But the switch is connected to pfe_mac1 via RGMII, and pfe_mac1 is named "eth1".

I still have the pfe_mac sections reversed in my DTS file, as suggested yesterday.  I don't think that matters, but I can change it back if needed.

Thank you for continuing to work on this.

  137.413058] calling pfe_eth_init_one, ii = 1
[  137.417353] ********** Called pfe_eth_init_one with id = 1
[  137.423036] in netdev_register_kobject, ndev->name = eth0
[  137.428582] device: 'eth0': device_add
[  137.436697] device: 'wakeup3': device_add
[  137.448469] calling pfe_phy_init from pfe_eth_init_one
[  137.457975] pfe_phy_init phy_id = ls1012a-0:00
[  137.464942] interface = 11
[  137.472735] configuring MDIO from PAD
[  137.479297] pfe_phy_init interface b
[  137.485464] calling of_phy_connect JAC **********
[  137.501237] called of_phy_find_device, phy_np->name is ethernet1
[  137.510539] bus_find_device_by_of_node returned d = 951476224
[  137.519404] calling to_mdio_device
[  137.522975] mdiodev->modalias is
[  137.526405] mdiodev->addr is 0
[  137.529546] mdiodev->flags is 1
[  137.532841] calling to_phy_device
[  137.536175] calling phy_connect_direct
[  137.540034] libphy: in phy_probe, writing 0 to lock.owner to force init
[  137.546882] in b53_spi_probe
[  137.550130] phy_connect_direct returned ret = 0
[  137.554864] b53_common: in b53_switch_register
[  137.559470] b53_common: dev->pdata is NULL, forcing chip ID
[  137.565185] pfe_eth_init_one: created interface, baseaddr: 000000009a20dbd6
[  137.572216] b53_common: found switch(dsa): BCM53128, rev 0
[  137.577733] calling pfe_eth_init_one, ii = 0
[  137.582016] in dsa_switch_parse_of
[  137.585512] in dsa_switch_parse_ports_of
[  137.589532] ********** Called pfe_eth_init_one with id = 0
[  137.595189] in netdev_register_kobject, ndev->name = eth1
[  137.600694] OF: in of_parse_phandle and of_parse_phandle_with_args returned !0, returning NULL
[  137.609418] device: 'eth1': device_add
[  137.617350] device: 'wakeup4': device_add
[  137.621424] in dsa_port_parse_of, name = lan1
[  137.632397] in dsa_switch_parse_ports_of
[  137.636613] calling pfe_phy_init from pfe_eth_init_one
[  137.644378] OF: in of_parse_phandle and of_parse_phandle_with_args returned !0, returning NULL
[  137.654734] pfe_phy_init phy_id = ls1012a-0:00
[  137.671004] interface = 4
[  137.680127] in dsa_port_parse_of, name = lan2
[  137.686064] configuring MDIO from PAD
[  137.692125] pfe_phy_init interface 4
[  137.695806] in dsa_switch_parse_ports_of
[  137.700891] calling of_phy_connect JAC **********
[  137.705694] OF: in of_parse_phandle and of_parse_phandle_with_args returned !0, returning NULL
[  137.714429] called of_phy_find_device, phy_np->name is ethernet-phy
[  137.720950] in dsa_port_parse_of, name = lan3

[jclasquin]

Comparing the boot messages before and after the suggested code change.  This is the first call to pfe_eth_init_one.

BEFORE code change:
[ 15.441336] calling pfe_eth_init_one, ii = 0
[ 15.445630] ********** Called pfe_eth_init_one with id = 0
[ 15.451306] in netdev_register_kobject, ndev->name = eth0
[ 15.456824] device: 'eth0': device_add

AFTER code change:
[ 137.413058]calling pfe_eth_init_one, ii = 1
[ 137.417353] ********** Called pfe_eth_init_one with id = 1
[ 137.423036] in netdev_register_kobject, ndev->name = eth0
[ 137.428582] device: 'eth0': device_add

[yipingwang]

The modification is used to make the eth0 as RGMII interface which connected to the switch, so it is what we need that eth0 is connected to switch, and customer need to check whether the switch work under this condition.

[jclasquin]

I understand now.  Yes, this seems to have connected the switch to eth0 and the single port to eth1.

When I plug the cable into the single port I see messages about the eth1, and when I plug into one of the 8 ports on the switch I see eth0 status change to up.  

I'm still able to successfully ping when I connect to the single port, but ping always fails when I use one of the switch's ports.  There must be something else wrong, but this was a step in the right direction.  

Thanks

[yipingwang]

It should not master = of_find_net_device_by_node(ethernet->parent);

It is a little special that the network devices point to pfe device node, which will result that it cannot find the pfe_mac1 device correctly, but with ethernet->parent, it would find the pfe_mac0 device, so you will get this issue.

Need more investigation on how to fix this issue.

[yipingwang]

As said before, both pfe_mac0/pfe_mac1 point to the "pfe" device node, that is why it cannot find the network device with of_find_net_device_by_node(ethernet); but could find it by of_find_net_device_by_node(ethernet->parent); but it seems both pfe_mac0/pfe_mac1 are point to "pfe", so it may get the first network device that point to "pfe", obviously it is the pfe_mac0 then customer see that the eth0 is connected to the switch.

So a workaround can be tried as below:

1. keep to use of_find_net_device_by_node(ethernet->parent); instead of   of_find_net_device_by_node(ethernet);
2. change the order of pfe_mac0/pfe_mac1 in dts, such as:

        pfe_mac1: ethernet@1 {

                 compatible = "fsl,pfe-gemac-port";

                 #address-cells = <1>;

                 #size-cells = <0>;

                 reg = <0x1>;    /* GEM_ID */

                 fsl,pfe-gemac-if-name = "eth1";

                 phy-mode = "rgmii-txid";

                 fixed-link {

                     speed = <1000>;

                   full-duplex;

                  };

        };

        pfe_mac0: ethernet@0 {

                 compatible = "fsl,pfe-gemac-port";

                 #address-cells = <1>;

                 #size-cells = <0>;

                reg = <0x0>;    /* GEM_ID */

                 fsl,pfe-gemac-if-name = "eth0"; 

                 fsl,mdio-mux-val = <0x0>;

                 phy-mode = "sgmii";

                 phy-handle = <&sgmii_phy>;

        };

[yipingwang]

I didn't find obvious problem in your dts file.

For Broadcom BCM53128 switch definition in the dts file, please refer to the following.

arch/arm/boot/dts/sun7i-a20-lamobo-r1.dts

arch/arm/boot/dts/bcm47094-linksys-panamera.dts

Documentation/devicetree/bindings/net/dsa/dsa.txt

[jclasquin]

Thank you for replying.  

I forgot to mention one thing.  The switch is detected only if I modify code in dsa2.c.  The function is shown below.

I have to change the line:
master = of_find_net_device_by_node(ethernet);

To:
master = of_find_net_device_by_node(ethernet->parent);

Otherwise the switch is not detected and none of the lan ports appear.

of_find_net_device_by_node calls class_find_device which tries to find a device for the class "net_class".  A device is found, and that device is compared with the device that is passed into of_find_net_device_by_node (the part I have to change to make this work).

If I do not change the line of code shown above, then the devices do not match.  As you can see in my DTS, the phandle 'ethernet' points to 'ethernet@1'.  Apparently, the device found in class_find_device is 'pfe'.  They don't match.

If I do change that line of code and pass in the parent of 'ethernet@1' -- which is 'pfe' -- then there is a match.  The switch is detected and all of the lan ports appear.

I know something is wrong.  I should not need to do this.  Any ideas why these devices don't match?  Is this why eth0 is getting attached to the lan ports instead of eth1?

Again, thank you for the assistance.

Joel 

static int dsa_port_parse_of(struct dsa_port *dp, struct device_node *dn)
{
	struct device_node *ethernet = of_parse_phandle(dn, "ethernet", 0);
	const char *name = of_get_property(dn, "label", NULL);
	bool link = of_property_read_bool(dn, "link");

	dp->dn = dn;

	if (ethernet) {
		struct net_device *master;
		const char *user_protocol;

		master = of_find_net_device_by_node(ethernet);
		if (!master)
			return -EPROBE_DEFER;

		user_protocol = of_get_property(dn, "dsa-tag-protocol", NULL);
		return dsa_port_parse_cpu(dp, master, user_protocol);
	}

	if (link)
		return dsa_port_parse_dsa(dp);

	return dsa_port_parse_user(dp, name);
}