RPMSG pingpong sample causes Linux kernel internal error

[swf]

Hi everyone!

I have a board featuring the i.MX 8M Plus processor and want to use the M7 coprocessor. My board features this SoM: https://embedded.avnet.com/product/msc-sm2s-imx8plus/

I tried to run the pingpong example , and it seems to work. Here is the dmesg output:

[  116.002743] remoteproc remoteproc0: powering up imx-rproc
[  116.010377] remoteproc remoteproc0: Booting fw image rpmsg_lite_pingpong_rtos_linux_remote.elf, size 165728
[  116.541483] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node vdevbuffer@55400000
[  116.551365] virtio_rpmsg_bus virtio0: rpmsg host is online
[  116.557457] rproc-virtio rproc-virtio.2.auto: registered virtio0 (type 7)
[  116.564366] remoteproc remoteproc0: remote processor imx-rproc is now up
[  117.552628] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1e
[  117.561315] imx_rpmsg_pingpong virtio0.rpmsg-openamp-demo-channel.-1.30: new channel: 0x400 -> 0x1e!
[  117.573936] get 1 (src: 0x1e)
[  117.578498] get 3 (src: 0x1e)
.....
[  117.807693] get 101 (src: 0x1e)
[  117.810865] imx_rpmsg_pingpong virtio0.rpmsg-openamp-demo-channel.-1.30: goodbye!
[  117.920808] imx-rproc imx8mp-cm7: imx_rproc_kick: failed (0, err:-62)

However, after the pingpong example has run, one I2C bus is kind of broken. No communication is possible until a reboot. E.g. when I try to read the time of a real-time clock on the bus this happens:

[  128.772966] Internal error: synchronous external abort: 0000000096000210 [#1] PREEMPT SMP
[  128.781179] Modules linked in: imx_rpmsg_pingpong fsl_jr_uio caam_jr caamkeyblob_desc caamhash_desc caamalg_desc crypto_engine authenc libdes crct10dif_ce polyval_ce polyval_generic flexcan rtc_pcf2131 can_dev at24 caam secvio error g_ether fuse
[  128.803011] CPU: 0 PID: 453 Comm: hwclock Not tainted 6.1.36-6.1.36-2.1.0-6.1.36-2.1.0+gd453b3ad570d #1
[  128.812418] Hardware name: MSC SM2S-IMX8MPLUS (DT)
[  128.817216] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  128.824188] pc : i2c_imx_bus_busy+0x54/0x1a4
[  128.828473] lr : i2c_imx_xfer+0x64/0x350
[  128.832404] sp : ffff80000af83970
[  128.835721] x29: ffff80000af83970 x28: 0000000000000000 x27: 0000000000000037
[  128.842872] x26: 0000000000000001 x25: ffff000005634880 x24: 0000000000068dbc
[  128.850025] x23: 0000000000000000 x22: ffff800009f26000 x21: 0000000000000003
[  128.857177] x20: 00000000ffff5947 x19: 0000000000000000 x18: 0000000000000000
[  128.864331] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[  128.871480] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[  128.878633] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[  128.885785] x8 : 0000000000000000 x7 : 0000000000000000 x6 : ffff000004097590
[  128.892937] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000000000c
[  128.900086] x2 : ffff80000bbb000c x1 : 0000000000000000 x0 : ffff8000095b8840
[  128.907240] Call trace:
[  128.909691]  i2c_imx_bus_busy+0x54/0x1a4
[  128.913626]  i2c_imx_xfer+0x64/0x350
[  128.917212]  __i2c_transfer+0x15c/0x4c0
[  128.921060]  i2c_transfer+0x60/0x124
[  128.924646]  i2c_transfer_buffer_flags+0x60/0x9c
[  128.929274]  pcf2131_i2c_read+0x34/0xa0 [rtc_pcf2131]
[  128.934350]  _regmap_raw_read+0xdc/0x160
[  128.938283]  regmap_raw_read+0x18c/0x264
[  128.942215]  regmap_bulk_read+0x1b8/0x240
[  128.946235]  pcf2131_rtc_read_time+0x54/0x230 [rtc_pcf2131]
[  128.951825]  __rtc_read_time+0x48/0x12c
[  128.955673]  rtc_read_time+0x3c/0x70
[  128.959258]  rtc_dev_ioctl+0x50c/0x8b0
[  128.963015]  __arm64_sys_ioctl+0xa8/0xf0
[  128.966950]  invoke_syscall+0x48/0x114
[  128.970711]  el0_svc_common.constprop.0+0xd4/0xfc
[  128.975428]  do_el0_svc+0x30/0xd0
[  128.978754]  el0_svc+0x2c/0x84
[  128.981816]  el0t_64_sync_handler+0xbc/0x140
[  128.986094]  el0t_64_sync+0x18c/0x190
[  128.989771] Code: f9421f22 b9400403 1ac322a3 8b030042 (39400042)
[  128.995873] ---[ end trace 0000000000000000 ]---

When reading the real-time clock or another device on the bus before the pingpong sample has run, everything works normally.

I experience the same behaviour on kernel 5.15 and 6.1, whereas in kernel 5.10 this problem does not occur.

It should be noted that the linux-imx kernel used has a few commits added by AVNET, for the most part adding devicetree files, see ssh://gitolite@msc-git02.msc-ge.com:9418/thirdparty/linux-imx.git .

I suspect I have set something bad in my devicetree file, but I don't know what. Most of it is copy-pasted from AVNET devicetree files; see the attachment for my dts.

Any help or input is much appreciated!

Kind regards

[Chavira]

Hi @swf!

Thank you for contacting NXP Support!

I can´t access to the device trees that you are referring.

I see you rdevice tree and I noticed some changes compared with our device tree.

You can see our device tree in the link below:

https://github.com/nxp-imx/linux-imx/blob/b586a521770e508d1d440ccb085c7696b9d6d387/arch/arm64/boot/d...

Note: For Linux release version L5.15.71-2.2.0 and later, the run prepare_mcore command must
run before the bootaux command. 

Best Regards!

Chavira

[swf]

Hi Chavira,

thank you for your answer!

Sorry for the inappropriate devicetree with unknown references. I rewrote it with minimal changes needed for my setup so that its references are easily traceable:

/dts-v1/;

#include "../../freescale/imx8mp-evk-rpmsg.dts"

// For some reason, imx8mp-evk-rpmsg.dts deletes the i2c3 node, but we need that i2c bus

// redefine i2c3 as done in imx8mp.dtsi
&aips3 {
	i2c3: i2c@30a40000 {
		compatible = "fsl,imx8mp-i2c", "fsl,imx21-i2c";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x30a40000 0x10000>;
		interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>;
		clocks = <&clk IMX8MP_CLK_I2C3_ROOT>;
		
		// additional properties as added in imx8mp-evk.dts
		clock-frequency = <400000>;
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_i2c3>;
		status = "okay";

		// my custom realtime clock on the i2c3 bus
		myrtc: rtc@53 {
		compatible = "nxp,pcf2131";
		reg = <0x53>;
		reset-source;
		};
	};
};

As you might have noticed, the i2c3 node is deleted in the devicetree you posted (https://github.com/nxp-imx/linux-imx/blob/b586a521770e508d1d440ccb085c7696b9d6d387/arch/arm64/boot/d...).

I don't know the reason for the deletion, but I need that i2c bus because devices are connected to it, so I simply readded it again.

Unfortunately the behaviour is the same:

myboard:~$ hwclock -f /dev/rtc1
2024-08-01 19:04:54.563406+02:00
myboard:~$ sudo sh -c "echo imx8mp_m7_TCM_rpmsg_lite_pingpong_rtos_linux_remote.elf > /sys/class/remoteproc/remoteproc0/firmware && echo start > /sys/class/remoteproc/remoteproc0/state"
myboard:~$ hwclock -f /dev/rtc1

and this shell session is unresponsive; in a new shell session the dmesg output gives the i2c error as written in my original post.

Given that it is exactly this i2c3 bus that is causing problems, maybe readding is a bad idea? But how else can I use the devices connected to it?

Thank you for your help!

[Chavira]

Hi @swf!

The i2c3 is used by M4 core, we delete the node to change the driver.

The driver used in imx8mp-evk.dts is " "fsl,imx8mp-i2c", "fsl,imx21-i2c"; " and for imx8mp-evk-rpmsg.dts is " "fsl,i2c-rpbus" ".

The i2c-rpbus node should define its bus id (which is the node communicating
with M4) in alias

"fsl,i2c-rpbus" for I2C bus over RPMSG 

Another thing can I noticed is the driver user for " imx8mp-cm7 " node in your device tree the driver used is "fsl,imx8mp-cm7" but should be "fsl,imx8mn-cm7" like in our device tree.

In you have the node m4_reserved: m4@0x70000000and should be m4_reserved: m4@0x80000000.

In general I noticed minimal errors that you need to modify maybe those minimal errors is the root cause.

if doesn't work please contact to AVNET support, they should have those examples working in their board.

Best Regards!

Chavira

[swf]

Hi Chavira,

honestly nothing of the propsed changes helped, but anyway thank you for the help!

I contacted AVNET support now.

Kind regards

[Chavira]

Hi @swf!

Unfortunately we don't have the board and we don't have access to the code of that board.

Normally the boards of our partners like avnet take our bsp and they port our software to their boards to provide you new features.

We only provie software for our evk boards and is the reason that is dificult to provide support is you are not using our boards.