LX2160a dpsw not working with 2 dprcs

[phoffmann]

I'm trying to get several DPDK applications, each with their own DPRC (with a DPNI) to use a DPSW that is connected to DPMAC.3. In this use case, I would expect that the DPSW should be in the root DPRC (dprc.1).

In the simple example below, I have found that the DPDK application works when the dpsw is in dprc.2, but it does not work when the dpsw is in dprc.1. I've upgraded to the latest LSDK (20.04). I see packets leaving the DPNI (using restool), but don't see packets leaving the DPMAC.3. I don't see errors from restool or DPDK in either case.

Why does this not work?

restool -m # MC firmware version: 10.20.4
restool --version # restool LSDK-20.04

ROOT_DPRC=dprc.1

# Create container that has dpni to be used by DPDK application.
export DPRC=$(restool -s dprc create $ROOT_DPRC --label="DPDK Container" --options=DPRC_CFG_OPT_SPAWN_ALLOWED,DPRC_CFG_OPT_ALLOC_ALLOWED,DPRC_CFG_OPT_TOPOLOGY_CHANGES_ALLOWED,DPRC_CFG_OPT_IRQ_CFG_ALLOWED)

# Create dpni and put it in the created dprc.
ls-addni --container=$DPRC --no-link --options=DPNI_OPT_SINGLE_SENDER,DPNI_OPT_CUSTOM_CG,DPNI_OPT_HAS_KEY_MASKING,0x80000000 --num-tcs=1 --num-queues=16 --fs-entries=1 --vlan-entries=16 --qos-entries=64 --num-cgs=1
export DPNI=dpni.2 # result from running ls-addni

# Create dpsw
ls-addsw -i=9 dpmac.3 $DPNI --container=$DPRC # Works - can receive and send packets
ls-addsw -i=9 dpmac.3 $DPNI # This will use $ROOT_DPRC. It does NOT work: it cannot send or receive packets

# Problem: the dpsw should be in dprc.1, but it doesn't work there.

[phoffmann]

yipingwang‌, we are trying to evaluate the LX2160A to use for our next gen board. Creating L2 switches are essential for our application.

I was able to work around the container issue with experimenting: I had to remove some "restool sync"s to get the switch to work in the root container. I don't understand when syncs should and should not be used. Any insight or reference would be welcomed.

However, I am now seeing issues with performance when trying to replace a DPDMUX with a DPSW. And in the experiment, that is the only change.

I have 2 LX2160A boards connected via MAC.3. Each board has 2 apps running on separate CPUs. On each board, the apps both connect to a DPDMUX/DPSW (what is selected depends on the experiment). Each app sends 8 packets to the corresponding app on the opposite board every 125 usec (packets are about 1500 bytes long).

I have attached the commands that were run for the setup for the DPDMUX and DPSW.

• Experiment 1: If a DPMUX is used on each board, I see no drops.
• Experiment 2: If I take one board and use a DPSW instead: 1) not all packets are received by the board with the DPMUX, but 2) all the packets sent through the DPMUX are all received by the board with the DPSW.

So I suspect there is some configuration / transmission issue at the DPSW.

I have looked at the restool stats for the DPNIs and DPMACS: I do not seen any drops reported. But I do see drops reported by my application.

The setup for both the DPMUX and DPSW is the same. Both are tested with the same application binary. The only difference is the creation of a DPMUX or a DPSW.

I've experimented with params via restool on the DPSW to try get rid of the drops but have had no luck. What could cause these performance problems with a DPSW?

[yipingwang]

when the DPSW is in dprc.1 then it is driven by the Linux kernel driver which uses the switchdev kernel framework so that it exposes an interface for each switch port. Without adding the switch interfaces in a bridge, no switching will happen between the switch ports. This is the convention when dealing with switches in Linux. In Linux, no switching is happening if the interfaces are not added in the same bridge. So, once the ports are added to a bridge interfce, DPDK application work when DPSW in dprc.1.

Please refer the steps used in the attached log.

[phoffmann]

yipingwang,

Thanks for the information. But I still am seeing packet drops. The reason I am confused is I have not seen any drop issues with DPMUXes even when handling heavier loads (8 uplinks). And the same application is being used.

I've updated my script to match yours, but have changed the script so:

• I am only using DPDK-owned DPNIs.
• I had to cut down the number of DPIOs and DPBPs when adding more DPNIs

Each app on RDB 1 is pinned to a CPU and does the following:

• sends 8 Ethernet frames to the corresponding CPU on RDB 2
• receives 8 Ethernet frames from the corresponding CPU on RDB 2

The sends and receives happen every 125 usec. Each Ethernet frame is 1502 bytes.

Base on using "restool dpni", "restool dpmac", and ifconfig for each switch port:

• The DPNI stats report no drops and the number of egress frames are what is expected.
• The DPSW reports RX drops.
• No other drops are reported elsewhere.

Base on experiments:

• If I run 1 application with a 3 port switch (2 DPNIs, 1 DPMAC) on each RDB, no drops occur.
  
  • If I run 2 applications with the same 3 port switch on each RDB, drops occur for both applications (each dpni port sends roughly 70-80% of packets).
• If I run 3 applications with a 4 port switch on each RDB, drops occur for all 3 applications (each dpni port sends roughly 51-56% of packets).
• If I run 4 applications with a 5 port switch on each RDB, drops occur for all 4 applications (each dpni port sends roughly 40-46% of packets).

Both RDBs are LX2160A.

The switch:

dpsw_attr.options value is: 0
max VLANs: 16
max FDBs: 1
frame storage memory size: 32768
number of interfaces: 5
current number of VLANs: 1
current number of FDBs: 1

The root container:
dprc.1 contains 103 objects:
object label plugged-state
dpni.1 plugged # 2 total
dpbp.58 plugged # 3 total
dpsw.0 plugged # 1 total
dpmac.18 plugged # 7 total
dpseci.0 plugged # 1 total
dpmcp.44 plugged # 36 total
dprtc.0 plugged # 1 total
dpio.15 plugged # 14 total
dpcon.31 plugged # 32 total
dprc.5 DPDK Container unplugged
dprc.4 DPDK Container unplugged
dprc.3 DPDK Container unplugged
dprc.2 DPDK Container unplugged

Example child container (all are similar)
dprc.2 contains 46 objects:
object label plugged-state
dpni.2 plugged # 1 total
dpbp.15 plugged # 14 total
dpci.1 plugged # 2 total
dpseci.16 plugged # 16 total
dpdmai.1 plugged # 2 total
dpmcp.37 plugged # 1 total
dpio.17 plugged # 2 total
dpcon.39 plugged # 8 total

[phoffmann]

yipingwang‌,

I've attached additional information continuing the thread from my previous post.

I've attached the script that shows how I set up both the dpdmux and dpsw. Usage:

# On RDB 1, one of the following commands is run:
source ./configSw.sh 1 1 # Set up DPDMUX or ...
source ./configSw.sh 1 0 # Set up DPSW.

# On RDB 2, one of the following commands is run:
source ./configSw.sh 2 1 # Set up DPMUX or ...
source ./configSw.sh 2 0 # Set up DPSW.

Also attached:
- ethtoolOutput.txt # ethtool output, identical for dpdmux, dpsw
- mux/dpniPort.txt # dpdmux output from /sys/class/net/evb0p0 (associated with DPNI port)
- sw/dpniPort.txt # dpsw output from /sys/class/net/eth2 (associated with DPNI port)
- mux/macPort.txt # dpdmux output from /sys/class/net/evb0 (associated with DPMAC port)
- sw/macPort.txt # dpsw output from /sys/class/net/eth6 (associated with DPMAC port)
- mux/stats.txt # dpdmux output from restool, ifconfig. Number in parentheses indicates percent of packets expected.
- sw/stats.txt # dpsw output from restool, ifconfig.

[yipingwang]

Please refer to the following configuration.

root@localhost:~# ls-addni --no-link
Created interface: eth0 (object:dpni.1, endpoint: )

# create the DPDK owned DPNI

root@localhost:~# source /usr/local/dpdk/dpaa2/dynamic_dpl.sh dpni
(...)

######################### Configured Interfaces #########################

Interface Name Endpoint Mac Address
============== ======== ==================
dpni.3 UNCONNECTED 00:00:00:00:5:1

# create a DPSW connected to the two DPNIs and a DPMAC

root@localhost:~# ls-addsw dpni.1 dpni.3 dpmac.1
Created ETHSW object dpsw.0 with the following 3 ports: eth2,eth3,eth4

# configure the switch interfaces and add them to a bridge

root@localhost:~# ip link set dev eth2 down
root@localhost:~# ip link set dev eth2 address 00:00:00:00:00:02
root@localhost:~# ip link set dev eth2 up
root@localhost:~# ip link set dev eth3 down
root@localhost:~# ip link set dev eth3 address 00:00:00:00:00:03
root@localhost:~# ip link set dev eth3 up
root@localhost:~# ip link set dev eth4 down
root@localhost:~# ip link set dev eth4 address 00:00:00:00:00:04
root@localhost:~# ip link set dev eth4 up

root@localhost:~# ip link add name br0 type bridge
root@localhost:~# ip link set dev br0 up
root@localhost:~# ip link set dev eth2 master br0
root@localhost:~# ip link set dev eth3 master br0
root@localhost:~# ip link set dev eth4 master br0

# start DPNI.1 (eth0) - the Linux owned DPNI

root@localhost:~# ip link set dev eth0 up
root@localhost:~# ip a a 1.1.1.2/24 dev eth0

Thanks,

Yiping

[phoffmann]

yipingwang‌,

I don't understand what you are trying to say. The snippet you posted has 1 DPNI owned by Linux, 1 owned by a DPDK application. I'm using all DPDK-owned DPNIs because of the setup I need. The set up I have works, but I am seeing unexplained packet drops (50%) from the DPNI to the DPSW.

What are you saying I'm doing wrong with my current script? As far as I can tell I'm doing the same thing as you mentioned, except tailored to my application (in this script, the set up is for 4 DPDK applications).

The key problem is the DPSW is functional with this setup, but drops 50% of the packets. An identical setup but with a DPDMUX drops no packets. Why would a DPSW drop packets but a DPMUX not? I would expect the same performance. I am using an LX2160. Are there known issues with the DPSW that are being worked on? Am I misconfiguring something with the DPSW?

[phoffmann]

yipingwang‌

Based on ethtool, the speeds of the dpsw ports connected to the dpnis are 1 Gbps (full duplex), and the speeds of the dpsw ports connected to the dpmacs are 10 Gbps (full duplex).

What speeds are supported between the GPP and the DPSW? Are these rates configurable, or do certain DPAA2 resources need to be configured correctly or restool commands to support the full speeds mentioned? Does the dpsw's number of interfaces affect the actual speeds possible on these interfaces?

Each application is sending:

8 frames * 1502 bytes * 8000 Hz * 8 bits/byte = 0.770 Mbps

That is 0.770 Mbps being sent from dpni X on RDB 1 to dpni X on RDB 2 and

that is 0.770 Mbps being received by the dpni X on RDB1 from dpni X on RDB 2.

Paul

[phoffmann]

yipingwang‌,

Thanks for the response. If I run commands on the LX2160A, similar to what you mentioned (script attached), It only works the first time the DPDK application is launched. After that, packets are no longer sent.

By using restool, I found that after the application is shut down, the connection to DPMAC.3 is down: "endpoint: dpsw.0.0, link is down". Before I even launched the application, this link was up.

The attached log shows how I ran the test (I am using testpmd to make it easy to reproduce).

Why doesn't the application work after the first time?

Paul

[phoffmann]

yipingwang‌,

I realized the script you sent is assigning the dpsw to dprc.2, not dprc.1 which is what I want.

If I correct the script attached in my previous response from:

   restool dprc assign dprc.1 --object=dpsw.0 --child=dprc.2 --plugged=1

to

   restool dprc assign dprc.1 --object=dpsw.0 --plugged=1

then I do not see packets leaving dpmac.3, even for my first attempt.

[phoffmann]

Hi Yiping,

The results are from a small test case that I believe shows the issue. This test case has only 1 DPDK app running on each LX2160A RDB and I create the DPNI using the ls-addni script. I hope to scale this test up so I can have multiple DPDK apps running, each with their own DPNI in a separate DPRC. I've been able to do this using DPDMUXes, but now I need to be able to use DPSWs.

In the test setup, I have 2 LX2160A RDBs connected to each other via MAC.3. I then have a single DPDK application running on each RDB that is sending packets to the DPDK application running on the other RDB. Based on tests:

• If the DPSW is assigned to DPRC.2, both DPDK applications are able to send and receive packets.
• If the DPSW is assigned to DPRC.1, The DPDK application cannot send packets.

In the script I use (attached) You'll see a comment with "PROBLEM". I've uncommented the line used in the test case that doesn't work. The commented out the line indicates the command that does work.

For both cases, I run the commands shown below. Note: I have not attached testprogram - this is a simple DPDK program that sends packets to the MAC specified on the command line using DPRC.2. The command line argument indicates the MAC that the DPDK application should send the packets to.

# Commands run on RDB 1

udhcpc -i eth1
cd /tmp
export RDB_BOARD=1
./nxp_dpsw.sh $RDB_BOARD
DPRC=dprc.2 ./testprogram --file-prefix=d1 -l 2 -n 1 -- -o d1 -p 0x1 -w 00:00:00:02:00:01

# Commands run on RDB 2

udhcpc -i eth1
cd /tmp
export RDB_BOARD=2
./nxp_dpsw.sh $RDB_BOARD
DPRC=dprc.2 ./testprogram --file-prefix=d1 -l 2 -n 1 -- -o d1 -p 0x1 -w 00:00:00:01:00:01

[yipingwang]

DPDK with DPSW is working properly. Please refer the attached log for commands.

[yipingwang]

Hello Paul Hoffmann,

Dprc.1 means kernel container. So when you are creating DPSW in DPRC.1, how are you create dpnis? They need to be created manually and assigned to dprc.2

Please refer LSDK UM section 10.2.8  DPDK on docker – it has some reference commands about using DPSW with DPDK.

The DPDK development team will look into this problem, will provide more update later.

Thanks,

Yiping