LX2160, SPDK, NVMeoF and RDMA

[ACastanie]

Hello,

I am using a LX2160ARDB rev2, with LSDK21.08, and I try to use it to do NVMe over fabrics.

The interface used for NVMeoF is the 40Gbps QSFP interface (dpmac.2), directly connected to a network storage node.

I successfully had it working in NVMeoF tcp mode, but not in RDMA mode.

So I am wondering if RDMA is supported by the DPAA2 ?

When I try to discover services using RDMA with "nvme discover -a 10.1.0.101 -s 4420 -t rdma", I get an error "reset by peer" and dmesg gives me that log (nvme-rdma module):

"[3344172.236776] nvme nvme0: rdma connection establishment failed (-104)"

-104 errno is reset by peer.

So I ran tcpdump on the 40Gbps interface to see if any packet for RDMA discovery (RDMA IP CM Service) was going out from the LX2160, and no packet is going out. So I assume that the error "reset by peer" is internal to the driver module and the LX2160.

Do you have any information about how to get RDMA working on a LX2160 ?

Is my problem hardware related (i.e. DPAA2 and RDMA incompatibility) ?

Thank you,

[Oswalag]

Hello,

Unfortunately, LX2160A does not support RDMA.

[ACastanie]

In addition to my answer, I would like to know why it is written in the LX2160 reference manual (rev 1, 10/2021) that the LX2160 is compatible RDMA (in chapter 1.5.4 Storage solutions) ?

I quote:
"With additional features like dual-mode (EP/RC) capabilities on the up to 24 lanes of Gen3 PCI Express, dual x8 Gen3 SR-IOV
ports with 128 VFs each, and RDMA over Converged Ethernet (RoCE) support, LX2160A is an ideal chip for current and next
generation storage solutions. The software enablement on the LX2160A ensures seamless integration into OpenStack
environments with support for Cinder and Swift."

Thank you,

[Oswalag]

Hello,

LX2160 RM only described the scenario for storage application, not described RoCE can be offload by DPAA2 HW, The Linux kernel driver "drivers/infiniband/sw" provide software implementation.

The SW hiberarchy is RDMA application -> RDMA API -> IB transport protocol -> UDP/IP -> ethernet link layer. The RDMA is performed by linux kernel driver after received data from specified UDP port. In other words, this kernel driver is a software implementation for RoCE protocol.

This implementation is provided by upstream linux kernel, not provided by LSDK. And we also not provide any RDMA solution in LSDK, we don't guarantee this function is work, because of it is untested.

[ACastanie]

OK thank you for your answer.

So RDMA is not supported by the HW nor by NXP through the SDK. I think the RM should be updated (maybe it's already the case in newer revision of the document ?) to remove the pair of terms RoCE and support for LX2160 chip.

I tried softRoCE with RXE Linux kernel driver and no luck with it. I will give it another try, but durng the first one I was not able to make SPDK running in RDMA mode through RXE kernel module for softRoCE.

I will look again in the menuconfig for the options you're mentioning, maybe I missed something in the kernel configuration for enabling softRoCE.

Regards,

[ACastanie]

Hello,

Thank you for your answer.

And since using RDMA is not possible due to DPAA2 limitation, do you know if it's possible to use softRoCE on a LX2160 with Linux ?

Also, I know that you have some how-to and results available about using SPDK on LS2088 and LS2 (if I remember correctly), but the bench characteristic and results are for NVMe over PCIe. By any chance, do you have some results to share for a NVMeoF setup (over tcp since RDMA doesn't work) on LX2160 ?