Traffic become broadcast between cascaded SJA1105Q switch chips

detory2023 · ‎05-17-2023

Hi, Experts,

I are working on setup of 2 cascaded SJA1105Q chips w/ imx8mp main board. We manage to upgrad to linux kernel 6.1.1 to get more SJA1105Q DSA driver updates. Now we can make the cascaded DSA setup work but found a weird phenomenon that all packets cross SJA1105 had been converted to broadcast packet.

Basically, we use 2 SJA1105 chips cascaded, swp0 ~ swp2 are on the first chip, and swp3 ~ swp6 are on the 2nd chip. You can refer to attached DTS snippet for more connection details:

The swp0 is so-called "upper port" which connects to the PC and is supposed to be sent mirrored packet to. I used the below TC commands and tried to mirror all packets to/from some specific port(swp3 in this case) to swp0.

tc qdisc add dev swp3 clsact
tc filter add dev swp3 ingress matchall skip_sw action mirred egress mirror dev swp0
tc filter add dev swp3 egress matchall skip_sw action mirred egress mirror dev swp0

The configuration is as below. I use laptops behind these ports to ping each other and then capture packets on swp0 to verify the result.

ip link set eth0 up
ip link set swp0 up
ip link set swp1 up
ip link set swp2 up
ip link set swp3 up
ip link set swp4 up
ip link set swp5 up
ip link set swp6 up 
ip link add name br0 type bridge
ip link set dev swp0 master br0
ip link set dev swp1 master br0
ip link set dev swp2 master br0
ip link set dev swp3 master br0
ip link set dev swp4 master br0
ip link set dev swp5 master br0
ip link set dev swp6 master br0

ip addr add 172.100.9.128/24 dev br0
ip link set dev br0 up

The test results are as below:

If the monitored port is on the first chip, the port-mirroing works fine.

In more detail:

a. If both ends are on the first chip (the only case is: swp1 ping swp2, mirror swp1 to swp0), it works fine;

b. If one end of the communication is NOT on the first chip (e.g. swp1 ping swp3, mirror swp1 to swp0) , I can capture the ICMP packets on swp0 and swp2 even W/O port-mirroring configuration. (This behaviour may suggest that cross-chip packets are broadcasted?)

If the monitored port is NOT on the first chip (e.g. swp4 ping swp3, mirror swp4 to swp0), the port-mirroring can NOT work.

But in further tests, I found out if I changed the target to port on the same chip (e.g. swp4 ping swp3, mirror swp4 to swp5), the port-mirroing still works fine.

I wonder if I used the right configuration and if it's the case, are all the above behaviours expected?

Any suggestions and hints will be appreciated. Looking forward to your response and thanks in advance!

kaden · ‎05-30-2023

Hi @detory2023 ,

depending on the switch configuration, your observations are expected. For instance, if the switch receives an ethernet frame (that is not a broadcast frame) on a port and it's destination MAC address is not yet know (i.e. is not contained in the L2 Address Lookup Table), the ethernet frame is forwarded to those ports set in the FL_DOMAIN field. Please see the UM11040 for further explanation of the FL_DOMAIN field and switch configuration. This explains your observation described in b).

a) is also as per expectation and it seems mirroring on device #1 is configured correctly.

For case c): If you want to mirror traffic from a specific port on device #2 to a specific port on device #1, you would first need to mirror the traffic from device #2 to the cascaded port and then mirror the cascaded port on device #1 to the desired mirror port (swp0 in your case).

How do you configure the static configuration of the switch?

BR

Kevin

detory2023 · ‎05-30-2023

Hi, Kevin

Thanks a lot your explanation. The mirror part looks reasonable, I will try it out later and let you know the result. For the FL_DOMAIN, the static configuration of that field in my setup is all port set (e.g. for port 0, the bits of port1/2/3/4 all set which means unknown destination traffic from port0 will be broadcast to all other ports. But in my understanding, it should only happen at first packet, right? the L2 Address Lookup Table need to be updated to have the new destination MAC address once the response back. But what I observed is that the broadcast stable for the whole stream life. What do you think?

For the static configuration, as I mentioned in previous thread, I didn't find easy way to dump the whole register map w/ DSA driver mode but I can manage to get some snippet data if you can name the specific registers or fields you want to check.

BR

Kent

kaden · ‎05-30-2023

Hi @detory2023 "But in my understanding, it should only happen at first packet, right? the L2 Address Lookup Table need to be updated to have the new destination MAC address once the response back."

--> Your understanding is correct. However, depending on the configuration, address learning might be disabled or there is not enough memory space assigned to dynamically learn new L2 Lookup entries.

Is it possible you could dump the static configuration tables of the switch? I marked the ones of interest here:

BR

Kevin

detory2023 · ‎06-12-2023

Hi, Kevin

Sorry for response late, I was stuck in other priorites last couple of weeks. I managed to have a try w/ cross switch mirror configuration you suggested but found out it's kind of tricky: I can not find dev port to set, when tried to mirror traffic to cascaded port. The tc command only take "dev port" as paramter but cascaded port has no dev level port num since it's invisible in DSA mode. e.g. in my setup, you can only find swp0 ~5 which mapping respectively to port 0~2 on switch1 and port 1~4 on switch2. Let me know if it's the case and any suggestion you may have.

For the static configuration data you mentioned in last mail. I finally find a dump function which can dump the static configuration in "generic loader format". It means we can decode these 32-bit double word value according to table format from UM11040. Not sure if any tool can be used to make this process easier. I attached the dump output of 2 switches for your reference. Pls do let me know if it works or any further info you need.

Thanks in advance!

kaden · ‎08-03-2023

Hi @detory2023

Do you have the possibility to read the respective switch configuration tables via dynamic reconfiguration?

In the application hints for SJA1105PQRS (AH1704), chapter 7.2.1 you can find an example showing how to read the switch tables during runtime.

BR

Kaden

detory2023 · ‎08-06-2023

Hi, Kevin

I use the SJA1105 in linux DSA mode which seems not compatible w/ the Python script you suggested in AH1704. But I finially found a API (sja1105_fdb_dump) which comfired to eventually call spi read in driver code. This API will be called w/ standard linux command "bridge fdb show". Below is the output of this command on my setup. Pls do let me know if it's helpful, thanks in advance!

root@OK8MP:~# bridge fdb show
76:d0:f7:c5:62:24 dev eth0 self permanent
01:00:5e:00:00:01 dev eth0 self permanent
01:80:c2:00:00:00 dev eth0 self permanent
01:80:c2:00:00:03 dev eth0 self permanent
01:80:c2:00:00:0e dev eth0 self permanent
76:d0:f7:c5:62:24 dev swp6 vlan 1 master br0 permanent
76:d0:f7:c5:62:24 dev swp6 master br0 permanent
01:00:5e:00:00:6a dev br0 self permanent
01:00:5e:00:00:01 dev br0 self permanent

root@OK8MP:~# bridge link
6: swp0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 100
7: swp1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5
8: swp2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5
9: swp3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5
10: swp4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5
11: swp5@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5
12: swp6@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 5

kaden · ‎09-04-2023

Hi @detory2023

Sorry for the late response. Unfortunately, this is not the configuration dump I asked for. Please have a look at this tool https://github.com/vladimiroltean/mv88e6xxx_dump

It can be used to dump the switch configuration.

BR

Kevin

detory2023 · ‎09-06-2023

Hi, Kevin

Thanks for your suggestion. The tool seems work fine. I attached the output of all 2 devices (I have 2 sja1105q in setup) and a debug output w/ option -D for your reference. Hope it helps this time,

Looking forward to your response.

detory2023 · ‎09-25-2023

Hi, Kevin

Have you got chance to look into it? Let me know if anything else you need, thanks in advance!

detory2023 · ‎06-24-2023

@kaden

Hi, Kevin

Have you got chance to take a look at my previous mail? Let me know if any further configuration or info you need, thanks in advance!

kaden · ‎10-17-2023

Hi @detory2023

I checked all of the provided information from your side. However, there is still some connections unclear to me because they actually deviate between DTS, switch configuration files and the description. Is it maybe possible to provide a block diagram to show how all the connections are done in your setup?

So that we are clearly on the same page here.

BR

Kevin

detory2023 · ‎10-25-2023

Hi, Kevin

Thanks for looking into the data. It has taken me a while to recover the setup and double confirm the connections. Pls check the attached picture for connection diagram. Basically the broadcast phenomenon I mentioned in the first message only occur when ping test between DUT1 and DUT3 in which the connection path cross two switch chips.

Pls do let me know if anything unclear or any further info you needed, thanks in advance!

BR

kaden · ‎11-17-2023

Hi @detory2023

Does it not appear when you ping from DUT2 to DUT3?

This should not be the case.

One solution to solve this could be to add static routing (L2 Lookup Table) to SJA1105-1 and SJA1105-2.

E.g. for SJA1105-1:

DUT3 MAC-ADDR -> DEST PORT = 4

E.g. for SJA1105-2:

DUT1 MAC ADDR -> DEST PORT = 0

DUT2 MAC ADDR -> DEST PORT = 0

Also please check to set the CASC_PORT in the general parameters table of both switches to the correct value. It seems that at least for one switch there is a wrong port number.

BR

Kevin

detory2023 · ‎11-20-2023

Hi, Kevin

The "broadcast" phenomenon DID appear too when pinging from DUT2 to DUT3. I just use DUT1 to DUT3 as a specific example. The further tests show any packets cross the switchs will be broadcasted on both sides. e.g. if I ping from DUT1 to DUT3 (vice versa), then the ping request packet can be received on all other ports of these 2 switches. But if I ping from DUT1 to DUT2 (both ports in same switch) then only destination port can receive the ping request packet.

So it did look to me like something wrong w/ the forwarding table on both switches. But to my understanding, these configurations(e.g. cascaded port etc) should be only specified in the BTS file and nowhere else. The routing information in forwarding table should be recorded and maintained automatically according packets received if configuration is correct. Have you find anything unusual in the configuration dump I provided last time?

I also attached snippet of BTS file we used for your reference. Pls help to check if anything wrong or missed, thanks in advance!

kaden · ‎11-28-2023

Hi @detory2023

Your understanding is not quite correct. You need the device tree file so that the Linux kernel knows which driver it shall use for the specific device which is in your case SJA1105.

Once the linux kernel knows which driver to use for the SJA1105, it can then use the drivers to load the switch configuration into the switch via SPI. The static switch configuration is necessary in order for the switch to forward any traffic.

Do you have a binary file which gets loaded into the SJA1105 at startup? I assume so since both of your devices hold a valid configuration. This binary file contains the static configuration of the sja1105's.

--------------

Few comments from what I saw from the provided configuration files:

From the configuration you sent to me, I see that the CASC_PORT setting for sja1105_spi11.txt is set to 5 although it is port 0 according to your block diagram.

Also MIRR_PORT is set to port 5 in sja1105_spi01.txt - this should be set to 0 according to your block diagram.

BR

Kevin

detory2023 · ‎12-03-2023

Hi, Kevin

As I explained before in thread, unfortunately, there is no external binary file of static configuration in DSA architecture like the old SDK style. The DSA driver will create its own static configuration data according to DTS and other pre-defined value then pack it into configuration format of SJA1105 and write to chip through SPI interface.

But your finding about mirror/cascade port configuration is inspiring. At least we now got some clues to follow. I will manage to debug into the static configuration creation process and see what I can get. Will let you posted once some progress.

kaden · ‎05-22-2023

Hi @detory2023

what is the static configuration you loaded into the devices?

BR

Kevin

detory2023 · ‎05-23-2023

Hi, Kaden

Thanks for response. The tricky part is that in DSA architecture, the sja1105 don't have such clear/separate configure value block/set like in old driver style. The configuration scattered in different ops functions in which compute different value according to type and predefined variables. We didn't change this part which means we use the default value/logic, the only input choice we made is in dts which I attached in previous thread. I also wonder if there is any way to dump configuration(both static and dynamic) in DSA mode。

The corresponding code part is in /drivers/net/dsa/sja1105/sja1105_static_config.h/c.

Hope this can answer your question or give some hints.

detory2023 · ‎05-21-2023

Hi, Experts

Anyone can help out? Any comments or hints will be really appreciated!