SRIO: SDK 1.6 gives RIO: inbound message copy failed, no buffers error

[ajayjoshi]

Hi,

We have two P4080ds boards connected via a SRIO switch. We can see that the connection between two boards is fine. The ping also works fine, but when we run iperf with heavy load, then we see the following messages "RIO: inbound message copy failed, no buffers error". We tried increased the CONFIG_RIONET_TX_SIZE and CONFIG_RIONET_RX_SIZE to 1024. But still see these errors after repeated testing.

-------------------------------------------------------------

fsl-of-rio ffe0c0000.rapidio: bellirq: 57

fsl-of-rio ffe0c0000.rapidio: pwirq: 478

fsl-of-rio ffe0c0000.rapidio: /rapidio@ffe0c0000/port1: LAW start 0x0000000c20000000, size 0x0000000010000000.

fsl-of-rio ffe0c0000.rapidio: RapidIO PHY type: Serial

fsl-of-rio ffe0c0000.rapidio: Hardware port width: 4

fsl-of-rio ffe0c0000.rapidio: Training connection status: Four-lane

fsl-of-rio ffe0c0000.rapidio: RapidIO Common Transport System size: 256

/soc@ffe000000/rmu@d3000/message-unit@0: txirq: 60, rxirq 61

fsl-of-rio ffe0c0000.rapidio: /rapidio@ffe0c0000/port2: LAW start 0x0000000c30000000, size 0x0000000010000000.

fsl-of-rio ffe0c0000.rapidio: RapidIO PHY type: Serial

-------------------------------------------------------------------

iperf -s ==> on one board, while client on another board.

[ 67] local 13.0.0.8 port 5001 connected with 13.0.0.7 port 46385

RIO: inbound message copy failed, no buffers

RIO: inbound message copy failed, no buffers

RIO: inbound message copy failed, no buffers

RIO: inbound message copy failed, no buffers

RIO: inbound message copy failed, no buffers

RIO: inbound message copy failed, no buffers

Any inputs on the same will be appreciated ...

Regards,

Ajay

[yipingwang]

Hello Ajay,

Please try whether increasing the value of RIO_MAX_TX_RING_SIZE and RIO_MAX_RX_RING_SIZE in arch/powerpc/sysdev/fsl_rmu.c would be helpful.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[ajayjoshi]

Hi Yiping,

Thanks. I have done that. I had increased it to 16K and CONFIG_RIONET_TX_SIZE and CONFIG_RIONET_RX_SIZE to 2048 as well. This has solved the issue to a great extent, but if I increase the iperf client threads to 256, then I still the crash sometimes. The problem is that the kernel is stuck in a loop and does not recover and I would need to do a reboot to recover the system. 

Regards,

Ajay

[yipingwang · Accepted solution]

Hello Ajay,

Please enable DEBUG_PW macro in the RMU driver source file and provide the console log, probably there is error interrupt information provided in the log.

I suspect this problem is caused by Rx packets processing is slower than Tx packets sending from the other side. Please try whether using larger Rx buffer size than Tx buffer would be helpful.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[ajayjoshi]

Hi Yiping,

Thanks, I tried that but could not find any error in the interrupts registers.

But as you said, I had tried increasing the Rx buffer size. Currently with 16K size I don't see any error with SRIO speed set to 1.25Gbps. I am not sure, if the SRIO is configured to higher rate what would be deemed enough as Rx buffer size. If you point me to a value which is considered to be good which would not give any error, with any speed, then I can set it accordingly. 

Regards,

Ajay

[yipingwang]

Hello Ajay,

Would you please provide the console log when Linux Kernel is stuck?

Thanks,

Yiping

[ajayjoshi]

Hi Yiping,

Sorry could not get to this for long. The setup is no longer with me to give the logs . But , yes, you were right the Rx packet processing is slower then Tx, hence the issue. We could set it to large enough value and  get it working.

Thanks for your help.

Regards,

Ajay