IMX 95 NPU not activated

[OlivierG]

HI

I use a IMX 95 Verdin Toradex Card a1 (not b0)

I would like to evaluate the NPU

if i launch :
/usr/bin/tensorflow-lite-2.18.0/examples# python3 ./label_image.py --ext_delegate /usr/lib/libneutron_delegate.so

i get the following messages

Error in cpuinfo: prctl(PR_SVE_GET_VL) failed
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Is there a connection between these two messages?
Should I conclude that the NPU is not enabled?
What should I do to resolve this issue?

Greetings

Olivier

[Zhiming_Liu]

Hi,

Please download eIQ to convert .tflite with Neutron support.

The eIQ1.16 is for L6.12.20 Neutron support.

For Neutron Converter CLI, please refer eIQ User Guide.

Best Regards,
Zhiming

[OlivierG]

hi

on IMX 95 i get

python3 label_image.py --ext_delegate /usr/lib/libneutron_delegate.so

Loading external delegate from libneutron_delegate.so with args: {}
ext_delegate [<tflite_runtime.interpreter.Delegate object at 0xffffa6d52660>]
INFO: NeutronDelegate delegate: 0 nodes delegated out of 31 nodes with 0 partitions.

Error in cpuinfo: prctl(PR_SVE_GET_VL) failed
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Warm-up time: 122.3 ms

Inference time: 118.2 ms

0.870588: military uniform
0.031373: Windsor tie
0.011765: mortarboard
0.007843: bow tie
0.007843: bulletproof vest

in comparison, on IMX 93 i get

python3 label_image.py --ext_delegate /usr/lib/libethosu_delegate.so
Loading external delegate from /usr/lib/libethosu_delegate.so with args: {}
INFO: Ethosu delegate: device_name set to /dev/ethosu0.
INFO: Ethosu delegate: cache_file_path set to .
INFO: Ethosu delegate: timeout set to 60000000000.
INFO: Ethosu delegate: enable_cycle_counter set to 0.
INFO: Ethosu delegate: enable_profiling set to 0.
INFO: Ethosu delegate: profiling_buffer_size set to 2048.
INFO: Ethosu delegate: pmu_event0 set to 0.
INFO: Ethosu delegate: pmu_event1 set to 0.
INFO: Ethosu delegate: pmu_event2 set to 0.
INFO: Ethosu delegate: pmu_event3 set to 0.
INFO: EthosuDelegate: 31 nodes delegated out of 31 nodes with 1 partitions.
Warm-up time: 4.1 ms

Inference time: 3.9 ms

0.874510: military uniform
0.031373: Windsor tie
0.015686: mortarboard
0.011765: bulletproof vest
0.007843: bow tie

on  imx 93 there is no message:

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

The inference time is much shorter  118.2 ms -> 3.9 ms

Olivier

[Zhiming_Liu]

Hi @OlivierG 

Please use older verison with tensorflow 2.16.2, like L6.6.X.

root@imx95frdm:/usr/bin/tensorflow-lite-2.16.2/examples# python3 label_image.py --ext_delegate /usr/lib/libneutron_delegate.so
Loading external delegate from /usr/lib/libneutron_delegate.so with args: {}

INFO: NeutronDelegate delegate: 29 nodes delegated out of 31 nodes with 1 partitions.

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Warm-up time: 3.9 ms

Inference time: 1.8 ms

0.850980: military uniform
0.058824: Windsor tie
0.011765: bulletproof vest
0.007843: bow tie

There are some issue about NPU delegate even in latest BSP L6.12.34


Best Regards,
Zhiming

[min-eta]

Hi Zhiming, your reply to this message would seem to indicate that in the older Linux version 6.6.x, to use the NeutronDelegate, you DO NOT need to first use the neutron_converter to convert some of the TFLite Ops in the Model to the custom NeutronGraph.

But for the newer Linux 6.12.x, you DO have to first use neutron_converter to convert the Model first?

@min-eta

[Zhiming_Liu]

Hi @min-eta 

Yes, the software framework has changed since L6.12.y

Best Regards,
Zhiming

[OlivierG]

HI

i try the folowing python code

import tensorflow as tf

modelpath ='model_yolo_11s_int8_fp32.tflite'

with open(modelpath, 'rb') as fid:
tflite_model = fid.read()

interpreter = tf.lite.Interpreter(model_content=tflite_model,experimental_delegates=[tf.lite.experimental.load_delegate('libneutron_delegate.so')])
interpreter.allocate_tensors()

i get the following error message

/usr/lib/python3.13/site-packages/tensorflow/lite/python/interpreter.py:457: UserWarning: Warning: tf.lite.Interpreter is deprecated and is scheduled for deletion in
TF 2.20. Please use the LiteRT interpreter from the ai_edge_litert package.
See the [migration guide](https://ai.google.dev/edge/litert/migration)
for details.

warnings.warn(_INTERPRETER_DELETION_WARNING)
INFO: NeutronDelegate delegate: 0 nodes delegated out of 339 nodes with 0 partitions.

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Exception ignored in: <function Delegate.__del__ at 0xffff5171bf60>
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/tensorflow/lite/python/interpreter.py", line 122, in __del__
AttributeError: 'NoneType' object has no attribute 'c_void_p'

Does the python code match your answer?
i precise that libethosu_delegate.so is not on my Card

Greetings

Olivier

[Zhiming_Liu]

Hi,

Yes, the sw team is investigating this issue.

Best Regards,
Zhiming

[Zhiming_Liu]

Hi,

Is there a connection between these two messages?

-->There is no connection about them.

Should I conclude that the NPU is not enabled?

-->If you can see below log, that means NPU delegate is working. 

INFO: NeutronDelegate delegate: .......

What should I do to resolve this issue?

-->Can you share full log?

Best Regards,
Zhiming

[OlivierG]

Hi

now, i have a imx95-19x19-lpddr5-evk

The Yocto is
i.MX Linux Yocto Project BSP 6.12.20_2.0.0 Release

When i launch
root@imx95evk:/usr/bin/tensorflow-lite-2.18.0/examples# ./benchmark_model --graph=mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libneutron_delegate.so
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Graph: [mobilenet_v1_1.0_224_quant.tflite]
INFO: Signature to run: []
INFO: External delegate path: [/usr/lib/libneutron_delegate.so]
INFO: Loaded model mobilenet_v1_1.0_224_quant.tflite
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 0 nodes delegated out of 31 nodes with 0 partitions.

INFO: Though EXTERNAL delegate is explicitly applied, the model graph will not be executed by the delegate.
Error in cpuinfo: prctl(PR_SVE_GET_VL) failed
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: The input model file size (MB): 4.27635
INFO: Initialized session in 56.257ms.
INFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
INFO: count=5 first=122578 curr=118671 min=118400 max=122578 avg=119396 std=1595

INFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
INFO: count=50 first=118459 curr=117971 min=117954 max=118782 avg=118259 std=173

INFO: Inference timings in us: Init: 56257, First inference: 122578, Warmup (avg): 119396, Inference (avg): 118259
INFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
INFO: Memory footprint delta from the start of the tool (MB): init=12.0352 overall=15.1602

when i launch
cat /sys/class/remoteproc/remoteproc0/state
i get : offline

Greetings

Olivier

[Zhiming_Liu]

Hi @OlivierG 

Please refer previous reply here.

To use NPU in i.MX95, customer need to convert common .tflite to neutron .tflite support with eIQ or neutron converter CLI.

Here you can try this model which i converted with eIQ.

Best Regards,
Zhiming

[OlivierG]

Hi

When i launch the python sample with the converted file, i get

root@imx95evk:/usr/bin/tensorflow-lite-2.18.0/examples# python3 label_image.py -m mobilenet_v1_1.0_224_quant_converted.tflite
Error in cpuinfo: prctl(PR_SVE_GET_VL) failed
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Traceback (most recent call last):
File "/usr/bin/tensorflow-lite-2.18.0/examples/label_image.py", line 92, in <module>
interpreter.allocate_tensors()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/tflite_runtime/interpreter.py", line 534, in allocate_tensors
return self._interpreter.AllocateTensors()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: Encountered unresolved custom op: NeutronGraph.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom Node number 1 (NeutronGraph) failed to prepare.Encountered unresolved custom op: NeutronGraph.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom Node number 1 (NeutronGraph) failed to prepare.

With the executable version i get
root@imx95evk:/usr/bin/tensorflow-lite-2.18.0/examples# ./label_image -m mobilenet_v1_1.0_224_quant_converted.tflite
INFO: Loaded model mobilenet_v1_1.0_224_quant_converted.tflite
INFO: resolved reporter
Error in cpuinfo: prctl(PR_SVE_GET_VL) failed
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: Encountered unresolved custom op: NeutronGraph.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
ERROR: Node number 1 (NeutronGraph) failed to prepare.
ERROR: Encountered unresolved custom op: NeutronGraph.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
ERROR: Node number 1 (NeutronGraph) failed to prepare.
ERROR: Failed to allocate tensors!

Greetings 

Olivier

[Zhiming_Liu]

Hi @OlivierG 

Can you try to test with neutron delegate /usr/lib/libneutron_delegate.so ?



Best Regards,
Zhiming

[OlivierG]

Hi

I build the last version of the yocto kernel imx-6.12.34-2.1.0

when i launch
python3 label_image.py -e /usr/lib/libneutron_delegate.so
i get
Loading external delegate from /usr/lib/libneutron_delegate.so with args: {}
/usr/lib/python3.13/site-packages/tflite_runtime/interpreter.py:457: UserWarning: Warning: tf.lite.Interpreter is deprecated and is scheduled for deletion in
TF 2.20. Please use the LiteRT interpreter from the ai_edge_litert package.
See the [migration guide](https://ai.google.dev/edge/litert/migration)
for details.

warnings.warn(_INTERPRETER_DELETION_WARNING)
INFO: NeutronDelegate delegate: 0 nodes delegated out of 31 nodes with 0 partitions.

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Warm-up time: 94.5 ms

Inference time: 60.2 ms

0.878431: military uniform
0.027451: Windsor tie
0.011765: mortarboard
0.011765: bulletproof vest
0.007843: sax

i modify the python code to correct the warning

python3 label_image2.py -e /usr/lib/libneutron_delegate.so
i get
Loading external delegate from /usr/lib/libneutron_delegate.so with args: {}
INFO: NeutronDelegate delegate: 0 nodes delegated out of 31 nodes with 0 partitions.

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Warm-up time: 58.0 ms

Inference time: 55.5 ms

0.878431: military uniform
0.027451: Windsor tie
0.011765: mortarboard
0.011765: bulletproof vest
0.007843: sax

i always get
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

i must precise that on IMX95
cat /sys/class/remoteproc/remoteproc0/state
return : offline

The same command on the IMX93 board return : running

On IMX95
dmesg | grep remoteproc

return
remoteproc remoteproc0: imx-rproc is available

On IMX93
dmesg | grep remoteproc
return
remoteproc remoteproc0: imx-rproc is available
remoteproc remoteproc0: powering up imx-rproc
remoteproc remoteproc0: Booting fw image ethosu_firmware, size 244744
remoteproc remoteproc0: remote processor imx-rproc is now up

i think that the NPU is not activated

Greetings 

Olivier 

[OlivierG]

On IMX95
dmesg | grep remoteproc

return
remoteproc remoteproc0: neutron-rproc is available
remoteproc remoteproc1: imx-rproc is available
remoteproc remoteproc1: attaching to imx-rproc
remoteproc remoteproc1: remote processor imx-rproc is now attached

remoteproc remoteproc0: powering up neutron-rproc
remoteproc remoteproc0: Booting fw image NeutronFirmware.elf, size 66896
remoteproc remoteproc0: remote processor neutron-rproc is now up
remoteproc remoteproc0: stopped remote processor neutron-rproc

remoteproc remoteproc0: powering up neutron-rproc
remoteproc remoteproc0: Booting fw image NeutronFirmware.elf, size 66896
remoteproc remoteproc0: remote processor neutron-rproc is now up
remoteproc remoteproc0: stopped remote processor neutron-rproc

[Zhiming_Liu]

Hi @OlivierG 

Please try to use this model in attachment, this model is converted from L6.12.34 neutron converter.

root@imx95evk:/usr/bin/tensorflow-lite-2.19.0/examples# ./label_image -m mobilenet_v1_1.0_224_int8_converted.tflite --external_delegate_path=/lib/libneutron_delegate.so -i grace_hopper.bmp -l labels.txt
INFO: Loaded model mobilenet_v1_1.0_224_int8_converted.tflite
INFO: resolved reporter
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.

INFO: Neutron delegate version: v1.0.0-be8bf399, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: invoked
INFO: average time: 1.388 ms
INFO: 0.756863: 653 military uniform
INFO: 0.145098: 907 Windsor tie
INFO: 0.0156863: 458 bow tie
INFO: 0.0117647: 466 bulletproof vest
INFO: 0.00784314: 835 suit

Best Regards,
Zhiming