Hi Peder Alm
Please refer Table 2-2. CM4 memory map in IMX6SXRM.pdf. In that you can see MMDC x32DDR Controller which shows the maximum RAM for cortex M-4.
Let us know in case of any concerns.
DDR Controller supports 16/32-bit LPDDR2-800, DDR3-800 and DDR3L-800. The maximum RAM that the MX6SX can support is given in Table 2-1. System memory map i.MX6SX Reference Manual
2GB for Cortex-A9 and 1.5GB for Cortex-M4