Out of TCM memory in imx95 Neutron delegate

[David41]

I'm working on imx95 version 6.6.23, trying to run a large model using neutron delegate.

I'm getting the following errors (several times) when loading the model:

WARNING: De-extracting subgraph 'subgraph_002' because of a failure!
WARNING: The failure occurred with message 'Out of TCM memory!' at line 293 in file '/home/osc/ci/npi-imx95/neutron-software/neutron-converter/src/NeutronTCMAllocator.cpp' in domain 'Allocator'.

The library is not provided in source code so there is no way to track the error.

How do I know what is the problem? What is the limit of what item in the model?

Thanks for the help

[Bio_TICFSL]

Hello,

That;s an issue with larger models at the moment of load the i.MX95 get the out of memory issue, but that is not a simple error its a warning that you may have to reduce the model. The i.mx95 development still on beta phase and will be investigating issue like this. Thanks for the catch.

Regards

[aekam_acldigital]

Hello,

I am evaluating i.MX95 EVK for the ML performance. I already have experience with i.MX8M Plus and have developed a couple of applications already.

As a starting point, I considered running sample application under eiq-example.

I followed the steps, however I ran into a problem with 'Out of TCM memory'.

Here is the log for your reference.

root@imx95evk:~/eiq-example/face_recognition# uname -a
Linux imx95evk 6.6.36-lts-next-gd23d64eea511 #1 SMP PREEMPT Wed Sep 4 08:22:45 UTC 2024 aarch64 GNU/Linux

root@imx95evk:~/eiq-example# ls -la models/
total 24024
drwxr-xr-x 2 root root 4096 Feb 14 11:44 .
drwxr-xr-x 8 root root 4096 Feb 14 11:45 ..
-rw-r--r-- 1 root root 24538272 Feb 14 11:44 facenet_512_int_quantized.tflite
-rw-r--r-- 1 root root 49472 Feb 14 11:44 yoloface_int8.tflite

root@imx95evk:~/eiq-example/face_recognition# python3 main.py --input=/dev/video13 --delegate=/usr/lib/libneutron_delegate.so
INFO: NeutronDelegate delegate: 45 nodes delegated out of 54 nodes with 4 partitions.

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
WARNING: De-extracting subgraph 'subgraph_159' because of a failure!
WARNING: The failure occurred with message 'Out of TCM memory!' at line 412 in file '/opt/bamboo/agent-home/xml-data/build-dir/MLTEC-NEUTNPIIMX950-BDEG/neutron-software/neutron-converter/src/NeutronTCMAllocator.cpp' in domain 'Allocator'.

Traceback (most recent call last):
File "/root/eiq-example/face_recognition/main.py", line 30, in <module>
recognizer = Facenet("../models/facenet_512_int_quantized.tflite", args.delegate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/eiq-example/face_recognition/face_recognition.py", line 20, in __init__
self.interpreter.allocate_tensors()
File "/usr/lib/python3.12/site-packages/tflite_runtime/interpreter.py", line 531, in allocate_tensors
return self._interpreter.AllocateTensors()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to build Neutron graph.
Delegate kernel was not initializedNode number 198 (NeutronDelegate) failed to prepare.Failed to apply the default TensorFlow Lite delegate indexed at 0.
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!
Error: invalid model handle!

Let me know how this can be resolved.

Regards,

Aekam Parmar