iMX8ULP - rpmsg communication errors

[j_k]

Hello,

we are using an iMX8ULP board with a rpmsg service between A35 and CM33 core.

We noticed some bit errors, when the CM33 sends data to the A35.

For CM33 firmware, we used the SDK 25.06.00 and adjusted the "power_mode_switch" demo from NXP. A 'loopback command' was added, so we can send a block of data from the A35 Linux to the CM33, the CM33 copies this data and sends it back to the Linux. We noticed, that sometimes bits flip from 1 to 0. The CM33 receives the data from Linux correctly, but the response to the Linux side has these bit errors.

For example: A35 sends '0xFF', CM33 receives '0xFF', CM33 generates a copy of the data and send '0xFF' back, but A35 reads '0xFD' via rpmsg.

To further track down the issue, can you give us some guidance:

- Is it correct, that rpmsg exchanges the data between A35 and CM33 via the external DRAM?

- How does rpmsg synchronize the data access between both cores?

- Is there any RAM test tool to check the DRAM settings from CM33? (we used NXP Tool to generate uboot timings, both this only works on the A35 side, I think.)

Thanks in advance,

Jan.

[Manuel_Salas]

Hello @j_k 

I hope you are doing very well.

I have a few questions/requests.

Could you please share your device tree related to the reserved memory for vring?

You said you modified the example of "power_mode_switch", which modifications did you do?

Answering your questions.

1- Yes, take a look to the reference Device tree, vdev* and vdevbuffer are in DDR. RPMsg itself doesn’t require DDR, any shared memory reachable by both cores works.

2- Virtio vrings in shared memory + MU interrupts

3- No, Unfortunately there is not a sample test of DRAM on M33 side.

Best regards,

Salas.

[j_k]

Hi Salas,

thanks for the reply.

Here is the part of the device tree:

reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;

    linux,cma {
      compatible = "shared-dma-pool";
      reusable;
      size = <0 0x28000000>;
      linux,cma-default;
    };

    rsc_table: rsc-table@1fff8000{
      reg = <0 0x1fff8000 0 0x1000>;
      no-map;
    };

    dsp_reserved: dsp@8e000000 {
      reg = <0 0x8e000000 0 0x2000000>;
      no-map;
    };

    vdev0vring0: vdev0vring0@aff00000 {
      reg = <0 0xaff00000 0 0x8000>;
       no-map;

    };

    vdev0vring1: vdev0vring1@aff08000 {
      reg = <0 0xaff08000 0 0x8000>;
      no-map;
    };

    vdev1vring0: vdev1vring0@aff10000 {
      reg = <0 0xaff10000 0 0x8000>;
      no-map;
    };

    vdev1vring1: vdev1vring1@aff18000 {
      reg = <0 0xaff18000 0 0x8000>;
      no-map;
    };

    vdevbuffer: vdevbuffer@a8400000 {
      compatible = "shared-dma-pool";
      reg = <0 0xa8400000 0 0x100000>;
      no-map;
    };

    audio_reserved: audio@a8500000 {
      compatible = "shared-dma-pool";
      no-map;
      reg = <0 0xa8500000 0 0x100000>;
    };
};

Compared to the example you linked, we don't have "m33_reserved" section, but it's also not listed in the linkerfile of M33 and not all dsp sections, but we also don't use that.

You said you modified the example of "power_mode_switch", which modifications did you do?

The M33 is connected to some SPI slaves, we made a srtm service, so we can access those from the Linux side. That means we extended the demo and added a 'srtm_spi_service', that works similar to the 'srtm_i2c_service' provided by NXP.

We already checked, that the bit flipping is not a problem of the SPI communication, this looks clean, M33 receives correct data from the slaves, but passing the correct data through rpmsg results in those bit errors on A35 side.

I'd like to add, that we don't see this behavior all the time, it only happens randomly every few commands.

[Manuel_Salas]

Hello,

As this happens randomly, it is very difficult to find the root cause of the issue.

Is there a way to reproduce the issue by my side?

Best regards,

Salas.

[j_k]

Hi Salas,

yes, that's our problem too.

We will try to narrow the error down a bit more. Knowing the communication is done via DRAM, we will recheck with NXP DDR Tools if our settings are OK.

Best greetings, Jan.