imx8mn remoteproc failure 11.0.0_2.0.0_ANDROID_SOURCE

[tylernol]

I am trying to boot m7 firmware via the remoteproc sysfs framework and am encountering issues with .elf files. When an elf is compiled for TCM (build_debug.sh), I get the following panic:

[  214.653635] remoteproc remoteproc0: powering up imx-rproc
[  214.663253] remoteproc remoteproc0: Booting fw image rpmsg_lite_str_echo_rtos_imxcm7.elf, size 372772
[  214.672757] Unable to handle kernel paging request at virtual address ffff800021a0691c

when I try with ddr_debug I get the following failure:

[ 2480.749633] remoteproc remoteproc0: powering up imx-rproc
[ 2480.755622] remoteproc remoteproc0: Booting fw image rpmsg_lite_str_echo_rtos_imxcm7_ddr_debug.elf, size 372760
[ 2480.765900] remoteproc remoteproc0: bad phdr da 0x80000000 mem 0x240
[ 2480.772272] remoteproc remoteproc0: Failed to load program segments: -22
[ 2480.779090] remoteproc remoteproc0: Boot failed: -22

looking at the imx_rproc.c source and imx_rproc_att_imx8mn, I see  

```

/* DDR (Data) */

        { 0x40000000, 0x40000000, 0x80000000, 0 },

```

which seems wrong, The linker files and the imx8mn prog doc have that as 0x8000_0000, as do several of the other SOC  imx_rproc_att definitions. 

[razriel]

I am having similar problem when trying to run the program using remoteproc

when I run the sample code of echo str with linker script for internal ram. it loads and run fine.

when trying to run sample code of echo str with linker script for ddr. i get the following message from kernel

bad phdr da 0x80000000 mem 0x240

the dts file is imx8mn-ddr4-evk-rpmsg.dts defintions are:

{

reserved-memory {

#address-cells = <2>;

#size-cells = <2>;

ranges;

m_core_reserved: m_core@80000000 {

no-map;

reg = <0 0x80000000 0 0x1000000>;

};

vdev0vring0: vdev0vring0@b8000000 {

reg = <0 0xb8000000 0 0x8000>;

no-map;

};

vdev0vring1: vdev0vring1@b8008000 {

reg = <0 0xb8008000 0 0x8000>;

no-map;

};

rsc_table: rsc-table@b80ff000 {

reg = <0 0xb80ff000 0 0x1000>;

no-map;

};

vdevbuffer: vdevbuffer@b8400000 {

compatible = "shared-dma-pool";

reg = <0 0xb8400000 0 0x100000>;

no-map;

};

};

I have modified imx_rpmsg.c to align with the linker file and the dts:

static const struct imx_rproc_att imx_rproc_att_imx8mn[] = {

/* dev addr , sys addr  , size     , flags */

/* ITCM   */

{ 0x00000000, 0x007E0000, 0x00020000, ATT_OWN | ATT_IOMEM },

/* OCRAM_S */

{ 0x00180000, 0x00180000, 0x00009000, 0 },

/* OCRAM */

{ 0x00900000, 0x00900000, 0x00020000, 0 },

/* OCRAM */

{ 0x00920000, 0x00920000, 0x00020000, 0 },

/* OCRAM */

{ 0x00940000, 0x00940000, 0x00050000, 0 },

/* QSPI Code - alias */

{ 0x08000000, 0x08000000, 0x08000000, 0 },

/* DDR (Code) - alias */

{ 0x80000000, 0x80000000, 0x1000000, 0 },

/* DTCM */

{ 0x20000000, 0x00800000, 0x00020000, ATT_OWN | ATT_IOMEM },

/* OCRAM_S - alias */

{ 0x20180000, 0x00180000, 0x00008000, ATT_OWN },

/* OCRAM */

{ 0x20200000, 0x00900000, 0x00020000, ATT_OWN },

/* OCRAM */

{ 0x20220000, 0x00920000, 0x00020000, ATT_OWN },

/* OCRAM */

{ 0x20240000, 0x00940000, 0x00040000, ATT_OWN },

/* DDR (Data) */

{ 0x80200000, 0x80200000, 0x00200000, 0 },

I still get the same error. when trying to run the sample with ddr linker script

MEMORY
{
m_interrupts (RX) : ORIGIN = 0x80000000, LENGTH = 0x00000240
m_text (RX) : ORIGIN = 0x80000240, LENGTH = 0x001FFDC0
m_data (RW) : ORIGIN = 0x80200000, LENGTH = 0x00200000
m_data2 (RW) : ORIGIN = 0x80400000, LENGTH = 0x00C00000
}

[LihuaLiu123]

Any solutions? The same issue i was encounted on IMX.8MP platform

[AldoG]

Hello,

I don’t think this may be caused because of the definition, actually the definition looks correct you may refer to the Reference Manual Memory map section.

I would like to know if you're using the rpmsg device tree?

Best regards,
Aldo.

[tylernol]

yes , I am using the 

imx8mn-evk-rpmsg.dts as is , as well as modified for with the carve outs to let the remoteproc boot/kick the m7 from a53/linux: https://www.nxp.com/docs/en/application-note/AN5317.pdf

I also found this change to imx_rproc.c on linux master, but it is not on the latest android linux branch:android-11.0.0_2.4.0:

https://www.spinics.net/lists/linux-remoteproc/msg10209.html

doing some log comparisons of master /all to the 11.0.0_2.4.0 branch, you can see how far behind 11.0.0_2.4.0 appears to be:

[tylernol]

is there an android branch that has been tagged and released yet that contains these fixes? In the meantime, I will try applying the patches to my local branch and test. 

[tylernol]

you also have tag 

android-11.0.0_2.6.0, but imx_rproc.c is still at

```

| *   c8b68e8f8c31 Merge remote-tracking branch 'lf-lts/lf-5.10.y' into lf-5.10.y_android-11

```

[AldoG]

Hello,

Thank you for sharing your results, could you try with Android 11.0.0_2.6.0, I see that the commit you have shared it is the same as in our latest Linux BSP release.

Could you please share the procedure that you're following for your test?

Also, please share the SDK version that you're using.

Best regards,
Aldo.

[tylernol]

it looks like you changed up CLANG requirements in 

device/nxp/common/build/kernel.mk 

the CLANG required in that make file is 

clang-r416183b but it is not in the prebuilts:

```

jenkins@9eb81884cfae:/aosp/android_build/prebuilts/clang/host/linux-x86$ ls

Android.bp  CleanSpec.mk            OWNERS     RELEASE_NOTES.md  clang-r353983c   clang-r365631c  clang-r370808b  clang-r377782c  clang-r383902   clang-r383902b1  llvm-binutils-stable  soong

Android.mk  GCC_4_9_DEPRECATION.md  README.md  clang-3289846     clang-r353983c1  clang-r370808   clang-r377782b  clang-r377782d  clang-r383902b  clang-stable     profiles

```

so I will have to build it. 

[tylernol]

I have tried with patches from the 11.0.0_2.6.0 tag for imx_rproc, I did not see a difference in results, but I will try a full 11.0.0_2.6.0 build on Wed morning.

I am using 2.11.0 for the SDK .

For the test, I am loading the .elf file, currently the rpmsg_lite_str_echo_rtos sample with the remoteproc sysfs commands. 

[tylernol]

I have tried with 11.0.0_2.6.0, same results. 

imx8mn-evk-rpmsg.dts  crashes, a version of it with the carveouts for the dtm fails to load. 

[tylernol]

procedure for testing:

 - add /data to the firmware search path. `

echo -n "/data" > /sys/module/firmware_class/parameters/path

- copy over elf to /data

adb push rpmsg_lite_str_echo_rtos_imxcm7_ddr_debug.elf /data

- set firmware name

echo -n "rpmsg_lite_str_echo_rtos_imxcm7_ddr_debug.elf" > /sys/class/remoteproc/remoteproc0/firmware

- launch

echo stop  >/sys/class/remoteproc/remoteproc0/state

echo start  >/sys/class/remoteproc/remoteproc0/state

results are a panic

[AldoG]

Hi,

Thank you for testing, sharing your procedure and results.

I have done some test on my side and even if I don't see a crash, I do see a failure from the imx_rproc when following the steps you have shared I get the following output:
imx-rproc imx8mn-cm7: imx_rproc_kick: failed (0, err:-62)

If I start Cortex-M7 at uboot, I can successfully start/stop  using remoteproc with no issues. Looking at the App Note (AN5317) I see this small note:
 
For i.MX 8M platforms, the root clock for M7/M4 must be kept always enabled by Linux to load the firmware code and start Cortex M7/M4. By default, NXP Linux BSP keeps the root clock enabled for the M core when it is started from U-Boot. Otherwise, if you need to firstly start the M core from the Linux booting phase, the clock driver (drivers/clk/imx/clk-composite-8m.c) must be updated to always skip the gate registration to keep the root clock always enabled for the M core.

Please try it and let me know if the issue persist.

Best regards,
Aldo.
 

[tylernol]

what dts file and elf file are you using as it never gets past the elf load for me?

[tylernol]

for our application , we need to be able to start and stop from linux /remoteproc. 

[AldoG]

Hello,

I have used the rpmsg_lite_str_echo_rtos SDK 2.11, builded for DDR debug, for dts I have modified standard RPMSG dts and included the reserved memory nodes:

m7_ddr_alias: m4@0x10000000 {
no-map;
reg = <0 0x10000000 0 0x1000000>;
};
m7_itcm: m4@0x7E0000 {
no-map;
reg = <0 0x7E0000 0 0x20000>;
};
m7_dtcm: m4@0x800000 {
no-map;
reg = <0 0x800000 0 0x20000>;
};

And updated rproc node with this memory sections:

imx8mn-cm7 {
                compatible = "fsl,imx8mn-cm7";
                rsc-da = <0xb8000000>;
                mbox-names = "tx", "rx", "rxdb";
                mboxes = <&mu 0 1
                          &mu 1 1
                          &mu 3 1>;
                memory-region = <&vdevbuffer>, <&vdev0vring0>, <&vdev0vring1>, <&rsc_table>, <&m_core_reserved>, <&m7_ddr_alias>, <&m7_itcm>, <&m7_dtcm>;
                status = "okay";
        };

Best regards,
Aldo.

[tylernol]

I believe I have everything identical , excepts perhaps the compiler being used for generating the M7 elf file.

This is the elf dump(

readelf -S) 

I get for the one I am trying:

There are 25 section headers, starting at offset 0x5cb64:

Section Headers:

  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al

  [ 0]                   NULL            00000000 000000 000000 00      0   0  0

  [ 1] .interrupts       PROGBITS        00000000 010000 000240 00   A  0   0  4

  [ 2] .resource_table   PROGBITS        00000240 010240 000058 00   A  0   0  4

  [ 3] .text             PROGBITS        000002a0 0102a0 008224 00  AX  0   0 16

  [ 4] .ARM              ARM_EXIDX       000084c4 0184c4 000008 00  AL  3   0  4

  [ 5] .init_array       INIT_ARRAY      000084cc 0184cc 000004 04  WA  0   0  4

  [ 6] .fini_array       FINI_ARRAY      000084d0 0184d0 000004 04  WA  0   0  4

  [ 7] .data             PROGBITS        20000000 020000 00006c 00  WA  0   0  4

  [ 8] .ncache.init      PROGBITS        80000000 02006c 000000 00   W  0   0  1

  [ 9] .ncache           PROGBITS        80000000 02006c 000000 00   W  0   0  1

  [10] .bss              NOBITS          2000006c 02006c 00a4a8 00  WA  0   0  4

  [11] .heap             NOBITS          2000a514 02006c 000404 00  WA  0   0  1

  [12] .stack            NOBITS          2000a918 02006c 000400 00  WA  0   0  1

  [13] .ARM.attributes   ARM_ATTRIBUTES  00000000 02006c 000030 00      0   0  1

  [14] .debug_info       PROGBITS        00000000 02009c 016458 00      0   0  1

  [15] .debug_abbrev     PROGBITS        00000000 0364f4 0037c0 00      0   0  1

  [16] .debug_aranges    PROGBITS        00000000 039cb8 001190 00      0   0  8

  [17] .debug_ranges     PROGBITS        00000000 03ae48 000fd0 00      0   0  1

  [18] .debug_line       PROGBITS        00000000 03be18 00b68d 00      0   0  1

  [19] .debug_str        PROGBITS        00000000 0474a5 009cd7 01  MS  0   0  1

  [20] .comment          PROGBITS        00000000 05117c 000049 01  MS  0   0  1

  [21] .debug_frame      PROGBITS        00000000 0511c8 00466c 00      0   0  4

  [22] .symtab           SYMTAB          00000000 055834 004840 10     23 781  4

  [23] .strtab           STRTAB          00000000 05a074 0029f2 00      0   0  1

  [24] .shstrtab         STRTAB          00000000 05ca66 0000fb 00      0   0  1

Key to Flags:

  W (write), A (alloc), X (execute), M (merge), S (strings)

  I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)

  O (extra OS processing required) o (OS specific), p (processor specific)

[tylernol]

here is the panic dump (with debugging enabled in the remoteproc driver

127|evk_8mn:/ $ [  152.624769] remoteproc remoteproc0: powering up imx-rproc

[  152.635015] remoteproc remoteproc0: Booting fw image orro_mcu.elf, size 380732

[  152.642302] remoteproc remoteproc0: da = 0xb8000000 len = 0x400 va = 0x00000000010c5d69

[  152.650418] imx-rproc imx8mn-cm7: map memory: 0000000038b790c7+8000

[  152.656729] imx-rproc imx8mn-cm7: map memory: 0000000026d190bb+8000

[  152.663021] imx-rproc imx8mn-cm7: map memory: 000000007892349b+1000000

[  152.669566] imx-rproc imx8mn-cm7: map memory: 00000000f523127c+1000000

[  152.676113] imx-rproc imx8mn-cm7: map memory: 000000009ecf97dc+20000

[  152.682484] imx-rproc imx8mn-cm7: map memory: 00000000ef749c8f+20000

[  152.688976] remoteproc remoteproc0: da = 0x80000000 len = 0x240 va = 0x000000001646f359

[  152.697029] remoteproc remoteproc0: da = 0x80000240 len = 0x8294 va = 0x00000000926c3e08

[  152.705203] remoteproc remoteproc0: da = 0x800084d4 len = 0xad18 va = 0x0000000018fe661c

[  152.713335] Unable to handle kernel paging request at virtual address ffff80001d0084d4

[  152.721412] Mem abort info:

[  152.724230]   ESR = 0x96000061

[  152.727299]   EC = 0x25: DABT (current EL), IL = 32 bits

[  152.732637]   SET = 0, FnV = 0

[  152.735843]   EA = 0, S1PTW = 0

[  152.739016] Data abort info:

[  152.741919]   ISV = 0, ISS = 0x00000061

[  152.745779]   CM = 0, WnR = 1

[  152.748809] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000427fc000

[  152.755539] [ffff80001d0084d4] pgd=00000000bffff003, p4d=00000000bffff003, pud=00000000bfffe003, pmd=0068000080000715

[  152.766188] Internal error: Oops: 96000061 [#1] PREEMPT SMP

[tylernol]

the armgcc I am using is a recent one:

https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu-rm/10.3-2021.10/gcc-arm-none-eab...

[tylernol · Accepted solution]

trying gcc-arm-none-eabi-10-2020-q4-major and I see some different results in the elf. Hopefully that explains it.  Will try later tonight or Tuesday morning. 

[AldoG]

Hi, 

As a reference I used:
GNU Arm Embedded Toolchain 10-2020-q4-major

Please do let me know of your results!

Best regards,
Aldo.