Multi Processor RPMSG Communication Failure

[tgrubb_sancsoft]

Having an intermittent issue with multiprocessor RPMSG communication between Linux and M4.

DART-MX8M-MINI

MCUXpresso 2.8.0

M4 side:

- FreeRTOS 

- NXP RPMSG Lite

- Creates RPMSG endpoint 35

- RPMSG receive task receives a message, processes the message, sends a response, repeat

A53 (Linux) side:

- Uses imx_rpmsg_tty driver

- RPMSG endpoint connected to ttyRPMSG35

- C program opens ttyRPMSG35 and gets a file descriptor

- C program then writes an RPMSG to the file descriptor and awaits a response

This process of sending RPMSG from Linux to M4 fails occasionally and cannot be recovered without power cycling the device.

Once in the errored state, activity can still be seen through the serial port on the M4, so the M4 is still running. However, the C program on the Linux side will get an error (timeout) when trying to write() to the ttyRPMSG35 file descriptor and it will get a 0 bytes response when trying to read() from the file descriptor.

Once in the errored state, from the Linux side, dmesg shows:

"imx_rpmsg_tty virtio0.rpmsg-virtual-tty-channel-1.-1.35: timeout waiting for a tx buffer"

"imx_rpmsg_tty virtio0.rpmsg-virtual-tty-channel-1.-1.35: rpmsg_send failed: -512"

Let me know what other information I can provide to help debug this issue.

Has anyone else experienced this or can offer any suggestions?

[tgrubb_sancsoft]

Thanks for the question. 

I am writing between 8 and 18 bytes in a single write.

Thanks,

Tyler

[Sanket_Parekh]

Hi @tgrubb_sancsoft 

Thank you for the response.

Once your A-side TX buffers are exhausted, RPMSG will fall asleep for 15 seconds… I guess you see just that, not complete death of RPMSG but this 15 seconds pause.

How to not unexpectedly waste your buffers and so not face this 15 seconds pause. Having a lot of data to send, don’t sent a byte or few bytes at a time. Instead send as much as you can up to 496 bytes at a time in one write() call. 496 comes from max buffer size 512 minus 16 bytes for RPMSG header. Both sides, Cortex-A and -M should follow this rule. One transfer triggers one interrupt. High interrupt rate is performance killer for every CPU. Sending by single byte instead of burst of bytes you multiply interrupt events. Interrupt rate is not so big problem for Cortex-M, but there are delivery to CM interrupts which are sent back to CA.

Thanks & Regards

Sanket Parekh

[Sanket_Parekh]

Hi @tgrubb_sancsoft 

I hope you are doing well.

How much characters you are sending in single write?

Thanks & Regards

Sanket Parekh