Compiling large models fails.

[lars1]

Hi,

We are developing an application for the iMXRT1060-EVK.

The TF lite model we have developed is now 12.7 MBytes (hex format) as header file. GCC will not compile that but generates "cc1plus.exe: out of memory allocating 67112959 bytes".

It is the compiler that runs out of memory. It is still possible to compile smaller models, approximately 6-7 MBytes without problems.

Is there a workaround for this or are we approaching it wrong trying to put the large flat model in a header file?

Have we hit the roof on what size model that can be used?

Appreciate any help on this.

Regards,

Lars Wallden

[terry_lv]

Have you checked the linker script?

Memory segments and sizes are defined in linker scripts.

If size of a memory segment defined in linker scripts is not enough to allocate enough memory for your array, linking will failed.

[lars1]

Hi Terry, I think the linker script is OK. The model is placed in the external HyperFlash memory that is 64MB so there should be plenty of room. I have modified the board so that the Hyperflash is enabled and tested it with the example in the SDK. Also, I use the external SRAM for the data and it has 32MB available so there are no issues there. The memory map from the compiler shows that the code, constant data and variables are placed in the existing memory without issues.

The problem lies with the GCC compiler itself that it can't find enough memory when building. I hope to solve it by adding RAM to the workstation.

[lars1]

Hi David,

Thank you for answering. The file has been converted to a header file with UTF-8 encoding.

I have seen different discussions regarding other people getting similar errors when using GCC. All my files are encoded with UTF-8 as this is the standard setting for MCUXpresso. I have found several bug reports issued about GCC/GNU and large initialized arrays: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

I would be happy to share the project with NXP support but as you understand I cant post it on the forum.

Regards,

Lars Wallden

Reducing the size of the 

[david_piskula]

I understand. I will contact you about options on how to share it soon. In the meantime, can you try compiling your project with the attached model? It's an mnist model from one of my projects. It has 40 MB and MCUXpresso is able to compile it just fine for me.

Also, are you compiling on a 32-bit or 64-bit system?

Regards,

David

[david_piskula]

I'm also attaching the whole project but since the upload limit is 9MB per file here, I had to separate the model from the project, so you will have to drop the model into "evkmimxrt1060_tensorflow_lite_mnist\source\" 

For more info about the attached project: https://community.nxp.com/t5/eIQ-Machine-Learning-Software/Using-a-TensorFlow-Lite-Model-to-Perform-...

Regards,

David

[lars1]

I have tried to compile the project you shared. It also fails even though the model is approximately half the size of my project model.

This leads me to believe that the problem is that my PC is out of RAM. It seems to have free memory and I suppose the swap also should work but possibly the compiler cannot find enough large free memory that is needed for building the project with the model.

I did discuss this in a thread in the MCUXpresso forum but the answers there was that it was not a RAM memory issue.

Under all circumstances, I will install an additional 8GB today so I have 16GB in total on the PC to see if that might be the solution to the problem.

I will let you know if it works with the additional RAM. If I can get the project you shared to work then most likely my project will compile as well.

Regards,

Lars

[david_piskula]

Okay, good luck and keep me posted. If adding RAM doesn't help, I can contact you over email so that you can share your project with me, if that's ok, and I can try to compile it on my PC.

Regards,

David

[lars1]

Hi David, I have been doing some additional testing.

I removed the latest version of MCUXpresso (v11.2.0) and installed the previous version ( v11.1.1). With the older version, I could compile the project that you sent "tensorflow_lite_minst" but when trying with my project, it not compile. I get the same out of memory message from the compiler:
cc1plus.exe: out of memory allocating 134221823 bytes
make: *** [source/model/subdir.mk:23: source/model/tflite.o] Error 1
"make -r -j4 all" terminated with exit code 2. Build might be incomplete.

If I just truncate the model so it is half-size (comment away the last half) it compiles without problems.

To clarify what I earlier wrote, the TF lite model (12.7 MBytes) is when it is converted to binary, the actual file with the model, hex code, is 75Mbytes. I converted the hex code with srec_cat to binary format. To compile the model and the code together I need to do it in MCUXpresso.

If you can send me your email I would be happy to share my project with you as long as it is not published anywhere.

Regards,

Lars

[david_piskula]

Hello Lars,

indeed I was trying to post an answer yesterday that when I use v11.2.0 I get the same error as you with my own project. Unfortunately, the community site crashed at the time, so the answer didn't get posted.

My email is david.piskula@nxp.com. However, since I could reproduce the issue in v11.2.0 and you were able to compile my project on v11.1.0, I'm pretty sure I will not be able to compile your project on v11.1.0 either.

I contacted our MCUXpresso engineers and will open a ticket regarding this issue. They think it's a gcc, not an MCUXpresso issue but it's pretty weird that v11.2.0 is worse than v11.1.0. I will get back to you as soon as I get more information on this.

Regards,

David

[lars1]

Hi David, the additional memory did not fix the problem. Obviously there is some difference between my workstation/settings and the one that you use. I use the following:

Processor: Intel Core i7-7700 CPU @ 3.60 GHz
Memory: 16.0GB
Disk: SSD with 52 GB free space.
Windows: Win 10 Pro ver 2004 OS Build 19041.450

Installed SDK: SDK_2.8.2_EVK_MIMXRT1060

MCUXpresso IDE v11.2.0 [Build 4120] [2020-07-09].
(This is the latest version of MCUXpresso, are you using the same?)

Since I can't compile the example that you sent I suggest that we look at what could be the problem with that project, since it is well-tested and should work for me also.

This is the console output I get trying to compile "evkmimxrt1060_tensorflow_lite_mnist" project in debug mode:

-------------

19:21:13 **** Incremental Build of configuration Debug for project evkmimxrt1060_tensorflow_lite_mnist ****
make -r -j8 all
Building file: ../source/mnist.cpp
Invoking: MCU C++ Compiler
arm-none-eabi-c++ -std=gnu++11 -DCPU_MIMXRT1062DVL6A -DCPU_MIMXRT1062DVL6A_cm7 -DSDK_DEBUGCONSOLE=1 -DXIP_EXTERNAL_FLASH=1 -DXIP_BOOT_HEADER_ENABLE=1 -DXIP_BOOT_HEADER_DCD_ENABLE=1 -DSKIP_SYSCLK_INIT -DSDK_DEBUGCONSOLE_UART -DFLATBUFFERS_LITTLEENDIAN -DTFLITE_MCU -DARM_MATH_CM7=0 -D__FPU_PRESENT=1 -DEW_USE_PIXEL_FORMAT_SCREEN -DEW_MAX_SURFACE_CACHE_SIZE=0x200000 -DEW_MAX_GLYPH_SURFACE_WIDTH=256 -DEW_MAX_GLYPH_SURFACE_HEIGHT=256 -DEW_USE_DOUBLE_BUFFER=1 -DEW_USE_FREE_RTOS=0 -DEW_SURFACE_ROTATION=0 -DEW_FRAME_BUFFER_COLOR_FORMAT=EW_FRAME_BUFFER_COLOR_FORMAT_RGBA8888 -D__MCUXPRESSO -D__USE_CMSIS -DDEBUG -D__NEWLIB__ -I../board -I../embeddedwizard/PlatformPackage/RGBA8888 -I../embeddedwizard/PlatformPackage/RTE -I../embeddedwizard/TargetSpecific/ -I../embeddedwizard/TargetSpecific/Drivers/ -I../embeddedwizard/TLSF/ -I../embeddedwizard/ -I../source -I../ -I../tensorflow-lite/tensorflow/contrib/lite -I../tensorflow-lite/tensorflow/contrib/lite/kernels -I../tensorflow-lite/tensorflow/contrib/lite/profiling -I../tensorflow-lite/tensorflow/contrib/lite/schema -I../tensorflow-lite/third_party/flatbuffers/include/flatbuffers -I../component/lists -I../device -I../CMSIS -I../drivers -I../component/uart -I../component/serial_manager -I../xip -I../utilities -I../tensorflow-lite -I../tensorflow-lite/third_party/flatbuffers/include -O0 -fno-common -g3 -Wall -Wno-sign-compare -c -ffunction-sections -fdata-sections -ffreestanding -fno-builtin -fno-rtti -fno-exceptions -fmerge-constants -fmacro-prefix-map="../source/"=. -mcpu=cortex-m7 -mfpu=fpv5-d16 -mfloat-abi=hard -mthumb -D__NEWLIB__ -fstack-usage -specs=nano.specs -MMD -MP -MF"source/mnist.d" -MT"source/mnist.o" -MT"source/mnist.d" -o "source/mnist.o" "../source/mnist.cpp"

cc1plus.exe: out of memory allocating 65536 bytes
make: *** [source/subdir.mk:49: source/mnist.o] Error 1
"make -r -j8 all" terminated with exit code 2. Build might be incomplete.

19:21:31 Build Failed. 1 errors, 0 warnings. (took 17s.849ms)

----------------

Looking at available RAM during compilation it shows around 7.5GB.

My suspicion now is that there is some difference in my settings for the compiler/MCUXpresso or that we use different versions.

Regards,

Lars

[david_piskula]

Hello Lars,

I assume you used xxd on the model to convert it into a header file? Did you make sure the encoding is correct? According to what I've managed to find online, it seems one of the causes for this error is trying to compile a UTF-16 encoded file.

Can you share your project with the large model included, so that I can try to reproduce your issue?

Regards,

David Piškula