NPU compatibility

[AB22]

I am trying to get a model (SSD Mobilenet V2 fpnlite) to run exclusively on the NPU of the i.MX8M Plus. I am getting the following output: 

INFO: Vx delegate: allowed_cache_mode set to 0.
INFO: Vx delegate: device num set to 0.
INFO: Vx delegate: allowed_builtin_code set to 0.
INFO: Vx delegate: error_during_init set to 0.
INFO: Vx delegate: error_during_prepare set to 0.
INFO: Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

I am not sure what part of my model isn't compatible as it is fully quantized. I tried the ssd_mobilenet_v2_coco_quant_postprocess.tflite model from the following page: 

https://github.com/nxp-imx-support/imx-lane-detection?tab=readme-ov-file

According to that page, it runs on the NPU: 

i.MX 8M Plus

NPU

11.03 ms

./benchmark_model --graph=mobilenet_ssd_v2_coco_quant_postprocess.tflite --external_delegate_path=/usr/lib/libvx_delegate.so

Unfortunately, I get the same output with this model as I do with the model I trained. Any suggestions on what might be causing the need to use the CPU?

[Zhiming_Liu]

Hi @AB22 

For some unsupported operators , these operators will be mapped into CPU. If you want to make sure all operators  are running NPU, please replace the operators in your model referring the 14 OVXLIB Operation Support with NPU in this document.

https://www.nxp.com/docs/en/user-guide/UG10166.pdf


Best Regards,
Zhiming

[AB22]

Hi @Zhiming_Liu 

  Does that mean the NPU data provided for the ssd_mobilenet_v2_coco_quant_postprocess.tflite model from this page (https://github.com/nxp-imx-support/imx-lane-detection?tab=readme-ov-file) is just for the time to process the NPU compatible operations and some steps were done in the CPU? Both my model and the example have a TFLite_Detection_PostProcess operation which is likely not compatible with the NPU. Do I remove this operation and parse the output myself or can the TFLite_Detection_PostProcess operation fallback to the XNNPACK delegate. My initial attempts to run it with fallback have been unsuccessful: 

INFO: Vx delegate: allowed_cache_mode set to 0.
INFO: Vx delegate: device num set to 0.
INFO: Vx delegate: allowed_builtin_code set to 0.
INFO: Vx delegate: error_during_init set to 0.
INFO: Vx delegate: error_during_prepare set to 0.
INFO: Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
ERROR: Failed to verify graph
ERROR: Node number 168 (Vx Delegate) failed to invoke.

[Zhiming_Liu]

Hi @AB22 

If TFLite_Detection_PostProcess operation is not supported, it will fallback to the XNNPACK delegate. Here is test log based on L6.12.34.

INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Graph: [ssd_mobilenet_v2_coco_quant_postprocess.tflite]
INFO: Signature to run: []
INFO: External delegate path: [/usr/lib/libvx_delegate.so]
INFO: Loaded model ssd_mobilenet_v2_coco_quant_postprocess.tflite
INFO: Vx delegate: allowed_cache_mode set to 0.
INFO: Vx delegate: device num set to 0.
INFO: Vx delegate: allowed_builtin_code set to 0.
INFO: Vx delegate: error_during_init set to 0.
INFO: Vx delegate: error_during_prepare set to 0.
INFO: Vx delegate: error_during_invoke set to 0.
INFO: EXTERNAL delegate created.
WARNING: Fallback unsupported op 32 to TfLite
INFO: Explicitly applied EXTERNAL delegate, and the model graph will be partially exe                                         cuted by the delegate w/ 1 delegate kernels.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: The input model file size (MB): 6.17533
INFO: Initialized session in 53.627ms.
INFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but termin                                         ate if exceeding 150 seconds.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
INFO: count=1 curr=7823404 p5=7823404 median=7823404 p95=7823404

INFO: Running benchmark for at least 50 iterations and at least 1 seconds but termina                                         te if exceeding 150 seconds.
INFO: count=73 first=13235 curr=13066 min=12935 max=14602 avg=13650.3 std=582 p5=1295                                         3 median=13501 p95=14432

INFO: Inference timings in us: Init: 53627, First inference: 7823404, Warmup (avg): 7                                         .8234e+06, Inference (avg): 13650.3
INFO: Note: as the benchmark tool itself affects memory footprint, the following is o                                         nly APPROXIMATE to the actual memory footprint of the model at runtime. Take the info                                         rmation at your discretion.
INFO: Memory footprint delta from the start of the tool (MB): init=9.19922 overall=20                                         7.281

Best Regards,
Zhiming

[AB22]

Hi @Zhiming_Liu 

 Your example runs successfully between the CPU and NPU. Mine fails. I am using: 
Linux verdin-imx8mp-15335644 6.6.23-7.0.0-devel-ga65ddf143dee #1 SMP PREEMPT Wed Sep 25 11:25:16 UTC 2024 aarch64 GNU/Linux

 Any reasons you can think of as to why mine fails? I notice that mine lacks this output during warmup: 

INFO: Explicitly applied EXTERNAL delegate, and the model graph will be partially executed by the delegate w/ 1 delegate kernels.

I have been attempting to remove the NPU incompatible operation but what is left is not quantized after export to tflite. Is the standard solution to simply let it run in both the CPU and NPU? Hopefully, I can get that working.

[Zhiming_Liu]

Hi @AB22 

Is that able to upgrade the kernel version to 6.12.y? You can try  to download 6.12.y bsp from toradex website.

Best Regards,
Zhiming

[AB22]

Hi @Zhiming_Liu 

  I was never successful getting my Tensorflow 2.10 trained SSD Mobilenet V2 fpnlite model to run on the NPU of the i.MX8M Plus. It seems to have multiple operations that are not supported which I cannot prevent from being included. I have turned to trying to train with eIQ 1.17. I tested a model trained with eIQ (base model fpn_ssd_mobelinet_v2) and it did run on the NPU without issuing any errors. But, I am not getting quality training in the eIQ portal. I have imported my dataset which trained effectively in Tensorflow 2.10. In eIQ, I can only get my  evaluation mean average precision to about 0.045. Any suggestions as to what I might be doing incorrectly?

[Zhiming_Liu]

Hi @AB22 

Can you share your trained model from Tensorflow 2.10? I want to run it on L6.6.23.

Best Regards,
Zhiming

[AB22]

Hi @Zhiming_Liu 

  I have attached a compressed tflite. Here is the output from Qt when I run it:

INFO: Vx delegate: allowed_cache_mode set to 0.
INFO: Vx delegate: device num set to 0.
INFO: Vx delegate: allowed_builtin_code set to 0.
INFO: Vx delegate: error_during_init set to 0.
INFO: Vx delegate: error_during_prepare set to 0.
INFO: Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Loaded image 320 x 320
after input created
after memcpy
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
W [HandleLayoutInfer:332]Op 162: default layout inference pass.
ERROR: Failed to verify graph
ERROR: Node number 168 (Vx Delegate) failed to invoke.
Error during inference

I haven't tried the newer kernel because my mipi camera has issues with more recent BSPs from Toradex. 

Thank you

[Zhiming_Liu]

Hi @AB22 

There is no issue on L6.6.23 BSP. Please make sure that this model can support 320x320 image input. After you check there is no issue in your program, please find support from Toradex, maybe this relates to their BSP.

Best Regards,
Zhiming

[AB22]

Hi @Zhiming_Liu 

  I ran your two command line tests and they resulted in similar output with my build. Perhaps the code I am using to run the model within Qt is not correct.

Thank you 

[AB22]

Hi @Zhiming_Liu 

  Thank you for helping me identify that the model itself was not my problem. I can run inference on a test image within my Qt application without error but only if I limit the framerate of update of my display to 8 fps (1920x1080). Might you know what it is that may be the limiting resource that is causing my required 30 fps framerate to interfere with the inference?

[Zhiming_Liu]

Hi @AB22 

The best reference to handle video is using NNStreamer. Please refer 8 Vision Pipeline with NNStreamer in this guide: https://www.nxp.com/docs/en/user-guide/UG10166.pdf

The NNStreamer example pipeline is here: https://github.com/nxp-imx/nxp-nnstreamer-examples

Best Regards,
Zhiming

[AB22]

It turns out the conflict between the display update and inference only occurs on the first inference during the warmup period for the NPU.