i.MX93 M33 Can't Use System TCM RAM for Allocation

[jcolebaker]

We're evaluating the i.MX9352 for an IoT device. I've created an application for the M33 core for time-critical IO operations which include collecting a large number of samples from peripherals. For development purposes, I am loading and starting the M33 code from Linux with remoteproc. Code is written in C and using MPUXpresso 26.06.00 SDK.

I got the code working well, but now I need a large buffer for samples (~24 kB). I have tried adding this as either a static array or heap allocated with `malloc`. In either case, I seem to tun out of RAM even though the compile output indicates there is plenty.

Working Version:

Here's the memory information for a build with a small buffer, which **works OK** (but the buffer is too small for our requirements).

Memory region         Used Size  Region Size  %age Used
    m_interrupts:        1140 B       1144 B     99.65%
          m_text:       78300 B     129928 B     60.26%
m_m33_suspend_ram:           0 B         8 KB      0.00%
m_a55_suspend_ram:           0 B         4 KB      0.00%
          m_data:       48016 B       108 KB     43.42%
       m_rsc_tbl:           0 B         4 KB      0.00%
build finished successfully.

Here is some info from the ELF file:

readelf -l imx_m33.elf

Elf file type is EXEC (Executable file)
Entry point 0xffe0595
There are 4 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x001000 0x0ffe0000 0x0ffe0000 0x00474 0x00474 R   0x1000
  LOAD           0x001478 0x0ffe0478 0x0ffe0478 0x131dc 0x131dc RWE 0x1000
  LOAD           0x015000 0x20003000 0x0fff3654 0x00170 0x00170 RW  0x1000
  LOAD           0x000180 0x20003180 0x0fff37e0 0x00000 0x0ba10 RW  0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .interrupts
   01     .resource_table .text .ARM .init_array .fini_array
   02     .data
   03     .bss .heap .stack

Large Static Allocation:

Here's the memory and ELF file info for a build with a **24 kB static allocated buffer**.

I.e.:

static uint32_t m_sample_queue[SAMPLE_QUEUE_LENGTH]; // SAMPLE_QUEUE_LENGTH = 6000

Memory region         Used Size  Region Size  %age Used
    m_interrupts:        1140 B       1144 B     99.65%
          m_text:       78240 B     129928 B     60.22%
m_m33_suspend_ram:           0 B         8 KB      0.00%
m_a55_suspend_ram:           0 B         4 KB      0.00%
          m_data:       72016 B       108 KB     65.12%
       m_rsc_tbl:           0 B         4 KB      0.00%
build finished successfully.

### (As expected, the `m_data` section has increased in size.) ###

readelf -l imx_m33.elf

Elf file type is EXEC (Executable file)
Entry point 0xffe0595
There are 4 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x001000 0x0ffe0000 0x0ffe0000 0x00474 0x00474 R   0x1000
  LOAD           0x001478 0x0ffe0478 0x0ffe0478 0x131a0 0x131a0 RWE 0x1000
  LOAD           0x015000 0x20003000 0x0fff3618 0x00170 0x00170 RW  0x1000
  LOAD           0x000180 0x20003180 0x0fff37a0 0x00000 0x117d0 RW  0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .interrupts
   01     .resource_table .text .ARM .init_array .fini_array
   02     .data
   03     .bss .heap .stack

When I try to start this version in Linux with remoteproc, it fails to start and dmesg shows the following errors:

[ +0.001258] imx-rproc remoteproc-cm33: Translation failed: da = 0xfff37a0 len = 0x117d0
[ +0.000021] remoteproc remoteproc0: bad phdr da 0xfff37a0 mem 0x117d0
[ +0.000006] remoteproc remoteproc0: Failed to load program segments: -22
[ +0.008868] remoteproc remoteproc0: Boot failed: -22

Claude tells me this is a problem with the .bss .heap .stack section, because the PhysAddr is `0x0fff37a0` and the size is now `0x117d0`. `0x0fff37a0 + 0x117d0 = 0x10004f70` which exceeds the M33 Code TCM address range 0x0ffe0000 .. 0x10000000. The explaination was confusing but my interpretation is that the static initialisation has to go into the "code" section, causing it to overflow even though there is plenty of space in the "system" TCM range (the other 128 kB). So maybe this makes sense.

Dynamic (Heap) Allocation:

E.g.:

uint32_t *p_sample_queue = malloc(SAMPLE_QUEUE_LENGTH, sizeof(uint32_t));

The default heap size available to C is only 1 kB, so malloc fails with our large buffer.

I modified the CMake for the project to allocate a larger heap (32 kB) via __heap_size__ which feeds into the linker script:

mcux_add_linker_symbol(
    SYMBOLS "__stack_size__=0x400 \
             __heap_size__=0x8000 \   <---- Added
             __use_shmem__=1 \
             __multicore__=1 \
            "
)

Build output and ELF file info:

Memory region         Used Size  Region Size  %age Used
    m_interrupts:        1140 B       1144 B     99.65%
          m_text:       78240 B     129928 B     60.22%
m_m33_suspend_ram:           0 B         8 KB      0.00%
m_a55_suspend_ram:           0 B         4 KB      0.00%
          m_data:      103760 B       108 KB     93.82%
       m_rsc_tbl:           0 B         4 KB      0.00%
build finished successfully.

#### ELF file info: ####
readelf -l imx_m33.elf

Elf file type is EXEC (Executable file)
Entry point 0xffe0595
There are 4 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x001000 0x0ffe0000 0x0ffe0000 0x00474 0x00474 R   0x1000
  LOAD           0x001478 0x0ffe0478 0x0ffe0478 0x131a0 0x131a0 RWE 0x1000
  LOAD           0x015000 0x20003000 0x0fff3618 0x00170 0x00170 RW  0x1000
  LOAD           0x000180 0x20003180 0x0fff37a0 0x00000 0x193d0 RW  0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .interrupts
   01     .resource_table .text .ARM .init_array .fini_array
   02     .data
   03     .bss .heap .stack

This seems to make the problem WORSE, not better (.bss/.heap/.stack at PhysAddr 0x0fff37a0, size 0x193d0).

[ +0.001320] imx-rproc remoteproc-cm33: Translation failed: da = 0xfff37a0 len = 0x193d0
[ +0.000019] remoteproc remoteproc0: bad phdr da 0xfff37a0 mem 0x193d0
[ +0.000006] remoteproc remoteproc0: Failed to load program segments: -22
[ +0.002908] remoteproc remoteproc0: Boot failed: -22

I thought using heap allocation should allow the code section to be smaller and allocate the memory from the data section. The "m_data" section shown in the build output above is indeed bigger.

I don't really understand the "PhysAddr", which matches the "Code TCM" range from the ref manual, even for things which should be in the "System TCM" region (I think?). The addresses under "VirtAddr" seem correct.

Why does the ELF file still try to place this .bss/.heap/.stack data at PhysAddr 0x0fff37a0, why is it so big when using runtime heap allocation, and is there a way to allocate my large buffer in the "System TCM" region?

[Zhiming_Liu · Accepted solution]

Hi @jcolebaker 

You can choose to change the LMA for data/bss/heap/stack to System TCM. In the MCUX linker script, change the load address (AT) for the data segment from code TCM to System TCM, so that PhysAddr also falls at 0x2000_0000:

.data : { ... } > m_data AT> m_data /* Do not use AT> m_text */
.bss : { ... } > m_data

When LMA == VMA and both are in System TCM, PhysAddr becomes 0x2000_xxxx, which matches an entry in the {0x20000000, …, 0x00040000} (256 KB) range in remoteproc driver, allowing remoteproc to translate correctly.

Best Regards,
Zhiming

[jcolebaker]

Thanks, that did it!

Note, the main segment I needed to move in order to use a much larger heap was the "heap" segment: 

.heap : { ... } > m_data AT> m_data