We are using DPDK on the LX2160 and are trying to get insight from those out there using it as well.
We have developed a smartNIC that has 2x 25G interfaces. We have 4 Rx queues per each interface defined and are using DPDK with RSS enabled so that on a given interface we have 4 cores/threads per interface pulling packets from its DPDK rx queue.
We then do some things to the packet/mbuf and pass it on to other cores/threads in the pipeline.
Eventually we have to then forward the packet to the host across PCIe, the smartNIC is a PCIe EP. The question I have is that we have to maintain packet order from an interface perspective. We typically would use a timestamp or sequence number.
I see that there is a sequence number generated in the DPAA2 and it supports timestamps. However I can't see in the PMD for the LX2160 that it brings any of these values out into the mbuf hdr structure.
How did NXP imagine that we could implement packet re-order when using RSS and you need to then re-order them?
WOuld it be possible for use to patch the PMD side of the dpaa2 net driver and possible get the timestamp or seqn into the mbuf hdr? Has anyone done this with the LX2160 yet?
1. Yes it is possible to get timestamp and seq number in the packet.
2. What are your RSS criteria? Typically packet getting distributed across queues using RSS belongs to different flows. Do you need to maintain the ordering across flows as well?
3. If you have to order packet belonging to different flows, there are two options:
a. Write your own software sequencer at the output
b. NXP Hardware does support an order restoration feature, however that is primarily for when the packets are getting exited by WRIOP (wire-rate I/O processor) or other accelerator with QMAN queues. NXP can explore this further (if needed) once we have the answer to question 2.
Thanks for responding to my question. The answer to your question #2 is that we need to reorder the packets from the different flows before we send them off to the host/x86 rings for our host side application to process. With that we won’t be able to use your WRIOP reorder feature and have to do the reorder in SW. This is why we want to get the timestamp and/or sequence number in the mbufs. If it’s a sequence number based on the order it arrived on off the network for a given interface that would be better than a timestamp. I presented the diagram of our use of our DPDK threads that process the packets coming in from each interface of the smartNIC to illustrate that we have 3 RSS cores per interface then they connect into a single reoder core and it feeds forwarder cores that will use QDMA to move the packets to the host side rings.
So if it’s possible to get the sequence number in the mbuf hdr will that require an update from NXP to its PMD?
Figure 1. Block Diagram of the DataPlane
We have not tried it yet, sorry. We will be trying this out by no later than the end of the week. As soon as I have results I will definitely let you know.