Using rxdb interrupts with rpmsg char driver

[BartholomeusDias]

Hello,

We are trying to find the limits of minimum latency for data exchange from M7 to A53 in an IMX8MN. 
The M7 is running FreeRTOS, and the A53 cores are running linux-imx (lf-5.15.y branch).
My level of assumptions in this post is high, so please correct me if I'm wrong.

Data is gathered using peripheral drivers in the the M7 core with DMA.
The M7 must be able to notify a userspace program which is running on an isolated core on the A53 of newly available samples. The A53 must do the data processing (not enough horsepower in the M7 to do it), and deliver the processed samples for transmission.
Our samplerate is ~100kHz, and we are investigating how small we can make the data block sizes, where smaller is better.

Currently, we are using RPMSG. Because the amount of data is too much to put into a message payload, and we would like to avoid unnecessary copying of data, we are sending short messages that have the pointer to the next block of data in DDR in a ping-pong fashion. This is a dynamically allocated sample buffer created by the "rmtcore_shm" driver based on the low power audio examples from NXP (AN12195SW).

At first we tried to use the "imx_rpmsg_tty" driver, but latency was very high.
Migrating to using the "rpmsg_char" driver already decreased the latency significantly. This is using the blocking read function of the "rpmsg0" device, created with the "rpmsg_ctrl0" driver.

Toggling a pin on both sides indicates a typical delay of 350us from sending the message from M7, to receiving it on the A53.

Because we are using shared memory, we could also make use of simple head and tail pointers in it to keep track of the buffer bookkeeping. Accepting that there is no handshake, we could then use a simple interrupt as a new data notification.
The MU (messaging unit) has 4 general purpose interrupts available, but we do not seem to use them yet in this context. I want to investigate if using these interrupts reduces overhead, and therefore latency.

In our dts, the mu mailboxes are configured based on examples as such:

imx8mn-cm7 {
	compatible = "fsl,imx8mn-cm7";
	rsc-da = <0xb8000000>;
	mbox-names = "tx", "rx", "rxdb";
	mboxes = <&mu 0 1
			  &mu 1 1
			  &mu 3 1>;
	memory-region = <&vdevbuffer>, <&vdev0vring0>, <&vdev0vring1>, <&rsc_table>, 	
		<&m_core_reserved>, <&m7_ddr_alias>, <&m7_itcm>, <&m7_dtcm>;
		status = "okay";
};

"&mu 3 1" would suggest the rxdb (or doorbell) interface should be available. When initially checking the available interrupts on the device per "cat /proc/interrupts | grep imx_mu_chan" I got:

38: 3 0 0 0 GICv3 120 Level imx_mu_chan[0-1], imx_mu_chan[1-1]

Indicating that only the RX and TX interrupts are registered. When calling 

MU_TriggerInterrupts(MUB, kMU_GenInt1InterruptTrigger)

for the second time on the M7 side, I could tell by the return value that the interrupt was not accepted and cleared by the A53 side the first time.

After some digging, it found this in imx_rproc.c:

static const struct imx_rproc_dcfg imx_rproc_cfg_imx8mn = {
	.att		= imx_rproc_att_imx8mn,
	.att_size	= ARRAY_SIZE(imx_rproc_att_imx8mn),
	.method		= IMX_RPROC_SMC,
};

And because of these lines in imx_rproc_xtr_mbox_init:

if (dcfg->method != IMX_SCU_API)
	return 0;

I could see that this prevented the registration of the rxdb callback after that.
Commenting out these lines resulted in:

38: 3 0 0 0 GICv3 120 Level imx_mu_chan[0-1], imx_mu_chan[1-1], imx_mu_chan[3-1]

And when calling the Int1 interrupt trigger from the M7 now, it was indeed handled and reset on the A53 side, I'm guessing inside "imx_mu_isr" function in imx-mailbox.c. Unfortunatly, it does not wake up the userspace read function with 0 data, as I was kind of hoping

Now, I'm puzzling how to create a userspace function that waits for the doorbell to ring and then commence processing. Could you please:

A: tell me if my approach makes any sense.

B: give me some pointers on how to proceed.

Thanks in advance!

[Stan88]

I'm sorry, did you manage to implement data exchange using DDR? The audio example looks a bit complicated. I use Toradex IMX8MP with Toradex reference image and it's not entirely clear where to get the driver or its sources - /dev/rmtcore_shm ? I couldn't find it preinstalled.

Do you have progress in this issue?

[AldoG]

Hello,

Yes, your approach seems good, using small messages from M7 to A53 to notify that data is ready to be consumed, similar to what we do in the sai_low_power_audio demo, available in our SDK, using SRTM (Simplified Real Time Messaging) protocol which is used to communicate between A core and M core.

Best regards,
Aldo.