How to mirror network traffic with S32G-RDB2 PFE driver?

[Gerry15]

Dear Community members,

my goal is to investigate incoming networktraffic going through the Ethernet ports of the S32G-RDB2 board, on the host linux BSP running on it (For now all traffic -> Not using/benefitting from offloading). Using tcpdump on the host to listen to pfe0, for example, does not show the traffic exchanged between two devices connected through pfe0. Therefore I wanted to solve this by mirroring traffic by using the PFE mirroring functionality, after failing with theSJA1110 mirroring functionality. Unfortunately I also had some problems in configureing the PFE driver correctly:

I tried using the mirror-add functionality of the PFE libfci_cli library pre-installed with the linux 36.0 BSP. I only found one case, how to read ping traffic between 2 devices connected through the board. I connected one device on PFE2 and another device on PFE0. Now I created the Mirror rule “MirrTo2”, which mirrors to emac2, and “MirrTo0”, which mirrors to emac0. I added the rules to the rxmirr0 variables of the physical interfaces. emac0 - rxmirr0: MirrTo2 and emac1 - rxmirr0: MirrTo0. This way both devices could communicate with each other, and I could read the ping traffic (PFE2 pings to PFE0) of the devices as shown in the following picture. I also activated Promiscuous mode on both emacs for simplicity. 

Connecting both devices to the PFE0 and using the rule emac0 - rxmirr: MirrTo2 does not show any traffic of the devices unfortunately when listening with tcpdump on the host. Why is this the case? I can also not follow why I can see the Ping request on PFE2 and the Ping reply on PFE0 in our first experiment. I tried to summarize my thoughts about this on the second attached picture. Additionally, it is not clear to me why I can see the traffic of the devices in first place, as other mirroring does not show the traffic to the host. Can you give an explanation?

Also, if this is the wrong approach to solve my goal, I would be grateful to receive some guidance on how to configure the PFE driver to forward all incoming traffic to my host. After achieving this goal of inspect all traffic with the host cpu, my goal will be to create classification rules fitting to my usecase to offload network traffic inspection to the PFE driver step by step. 

Thank you very much for reading!

[Daniel-Aguirre]

Hi,

Let us verify if your procedure is as expected for the mirror functionality to work as intended.

We have one note though, you are saying the following:

" I connected one device on PFE2 and another device on PFE0. Now I created the Mirror rule “MirrTo2”, which mirrors to emac2, and “MirrTo0”, which mirrors to emac0. I added the rules to the rxmirr0 variables of the physical interfaces. emac0 - rxmirr0: MirrTo2 and emac1 - rxmirr0: MirrTo0"

Which we assume the blacked part is a typo, since you were referring to emac2 in the previous sentence. Just to confirm this.

Please, let us know.

[Gerry15]

Hey,

thank you for your your Question, it was a typo and should be emac2 instead of emac1.

[Daniel-Aguirre]

Hi,

We apologize for the delay.

We have received the following comments from the internal team:

"The simplest way to monitor traffic on the physical interfaces is to enable an auxiliary interface and have the mirrors point to there. As a quick test you can modify the device tree from u-boot to enable aux0:

setenv enableAux0 'run loadfdt; fdt addr $fdt_addr; fdt resize; fdt set /soc/pfe@46000000/ethernet@13 status "okay"; '
setenv mmcboot 'echo Booting from mmc ...; run mmcargs; if run enableAux0; then ${boot_mtd} ${loadaddr} - ${fdt_addr}; else echo WARN: Cannot load the DT; fi;'
saveenv
reset

After logging in:

ifconfig aux0 up
ifconfig pfe0 up
ifconfig pfe2 up

libfci_cli mirror-add --mirror="toHif1" -i hif1
libfci_cli phyif-update -i emac0 --rxmirr0="toHif1"
libfci_cli phyif-update -i emac2 --rxmirr0="toHif1"

tcpdump -i aux0 should now show traffic entering through pfe0 and pfe2. You can verify that it's mirrored by removing the mirrors with the commands:

libfci_cli phyif-update -i emac0 --rxmirr0=""
libfci_cli phyif-update -i emac2 --rxmirr0=""

Note that aux0 will receive traffic that has not been marked as entering through a physical port, in this case it's the mirrored packet replica.

Mirroring shouldn't really be relied on to bridge traffic between physical interfaces. You should either use a fastpath or slowpath bridge or enable slowpath IP forwarding and iproute."

Please, let us know.

[Gerry15]

Hey, thank you!

Using aux works for me, but it only works after bridging pfe0 and pfe2 as described in my question threat. Unfortunately my configured mirroring already showed the traffic between both devices on tcpdump, but I didn't know why it worked and it seemed not meant for this kind of use. What possibilities are there to set up a bridge or even slow path bridge? 

Another issue I had, is that it is not possible to read the traffic between 2 devices connected via PFE0 Port2 & Port3. It seems to me, that the SJA switch already forwards the packtes before entering PFE0 and therefore nothing is mirrored to aux0. How is it possible to mirror the traffic of PFE0 port connected devices? This would eliminate the need of a bridge.

How can I activate slowpath IP forwarding and iprouting? The only way I found out was through editing the sysctl.conf file, but there is no distinguishing between fast or slow path. 

Kind Regards

[Daniel-Aguirre]

Hi,

Below will be some comments on your questions:

Q1 >> How is it possible to mirror the traffic of PFE0 port connected devices?

A1 >> Just to confirm, when you are referring to "PFE0 Port2 & Port3", what does Port2 and Port3 means under your context?

We seem to understand that this might not be possible, but we might be misunderstanding your question. We will verify this situation.

Q2 >> How can I activate slowpath IP forwarding and iprouting? 

A2 >> We will confirm if there is any guide of some sort for this situation.

Please, let us know.

[Gerry15]

UPDATE:

Hello,

I was able to mirror traffic of devices connected over the SJA1110A to one of the PFE0 ports by configuring the Switch via the SJA SDK.

Unfortunately the solution is not complete as I have to physically wire the MIRROR port to the eth0 interface port to be able to see the traffic on tcpdump. The data is mirrored and sent out via egress of the MIRROR port. I tried setting the MIRROR port to SJA1110 itself (port 4), but I was not able to see any data traffic.

Is there some way to get the mirroring configured correctly through the SDK? 

Sincerely 

[Daniel-Aguirre]

Hi,

Thanks for your feedback. We have received the following update:

"To create a fastpath bridge between pfe0 and pfe2 with mirroring to aux0, boot as before and execute the following commands

ifconfig aux0 up
ifconfig pfe0 up
ifconfig pfe2 up

libfci_cli bd-update --vlan=1 --uh=FORWARD --um=FLOOD --mh=FORWARD --mm=FLOOD
libfci_cli bd-update --vlan=0 --uh=FORWARD --um=FLOOD --mh=FORWARD --mm=FLOOD
libfci_cli bd-insif --vlan=1 -i emac0 --tag=OFF
libfci_cli bd-insif --vlan=1 -i emac2 --tag=OFF
libfci_cli bd-insif --vlan=1 -i hif1 --tag=OFF
libfci_cli bd-insif --vlan=0 -i emac0 --tag=ON
libfci_cli bd-insif --vlan=0 -i emac2 --tag=ON
libfci_cli bd-insif --vlan=0 -i hif1 --tag=ON
libfci_cli phyif-update -i emac0 --E --mode=VLAN_BRIDGE --P=ON --bs=NORMAL
libfci_cli phyif-update -i emac2 --E --mode=VLAN_BRIDGE --P=ON --bs=NORMAL
libfci_cli phyif-update -i hif1 --E --mode=VLAN_BRIDGE --P=ON --bs=NORMAL

libfci_cli mirror-add --mirror="toHif1" -i hif1
libfci_cli phyif-update -i emac0 --rxmirr0="toHif1"
libfci_cli phyif-update -i emac2 --rxmirr0="toHif1"

This will bridge traffic between pfe0 and pfe2 while mirroring the rx traffic on both ports to aux0.

For more information on this I suggest looking at the FCI API reference manual in the PFE driver docs.

Slowpath just refers to Linux based bridging and forwarding, while fastpath refers to FCI API setup of the PFE. There is nothing specific to PFE when configuring slowpath and there are many examples of how to do this in Linux available.

You cannot get the PFE to mirror traffic that is not seen at a PFE physical interface. I assume that by PFE port2 & port3 you are referring to traffic that is bridged by an external switch. If the board you are using is an RDB then you would need to get the SJA1110 SDK and setup mirroring to the host port on the switch itself."

We understand that you are now looking into the SJA1110 SDK, if so, we don't handle this SDK under the S32G community. Since SJA1110 documentation seems to be confidential, help us contacting your local NXP FAE, since they should be able to provide further support on this part. We do apologize.

Please, let us know.

[Gerry15]

Hello, 

I am refering to the following figure from the "S32G-VNP-RDB2 User Guide" for example.  

Thank you very much for the support!

[Daniel-Aguirre]

Hi,

Thanks for the feedback. Let us comment this with the internal team, to see what recommendations they provide.

Since the internal team has a limited bandwidth, the answer could be delayed, for which we apologize.

Please, let us know.