How to tell when inference is ran by GPU or NPU on iMX8MPlus

[BrunoSenzio]

Hello,

From the iMX Machine Learning Guide:

https://www.nxp.com/docs/en/user-guide/IMX-MACHINE-LEARNING-UG.pdf

When I am running benchmarks for TFLite models, there is an option --nnapi=true

How do I know when the inference is being done by the GPU and the NPU?

Thanks

[joanxie · Accepted solution]

To run the inference using the GPU/NPU hardware accelerator, add the --use_nnapi=true command line argument:

For NPU: adb shell /data/local/tmp/benchmark_model --graph=/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite --use_nnapi=true --nnapi_accelerator_name=vsi-npu

For GPU: adb root adb shell setprop vendor.USE_GPU_INFERENCE 1 adb shell /data/local/tmp/benchmark_model --graph=/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite --use_nnapi=true

more detailed information, you can refer to the link as below,

"https://www.nxp.com/docs/en/user-guide/IMX_ANDROID_TENSORFLOWLITE_USERS_GUIDE.pdf"

[jalane]

I would also like to know how to verify that I am running inference on the NPU vs. the GPU. 

I have my own scripts and have attempted to run inference on a set of models using the USE_GPU_INFERENCE environment variable.  I am using TensorFlow Lite and the VX delegate on the i.MX 8M Plus evaluation kit with the Linux v5.15.32-2.0.0 image.  The results that I am getting are so remarkably similar (warmup times within 2-3 ms of each other and inference times within 0.01 ms of each other on average) that I am skeptical that they are running on different accelerators.

How can I make sure that I'm actually using both devices?  Thank you.

[BrunoSenzio]

Thank you joanxie for clarification.

I have one more question.

I am trying to access this link:

https://community.nxp.com/community/eiq/blog/2019/07/15/eiq-sample-apps-tflite-quantization

Do you know if there is a similar link?

Thanks in advance

[joanxie]

I got "Page not found", I don't know what this is

[BrunoSenzio]

Hello,

It is the quantization guide for the eIQ Sample Apps from here

https://community.nxp.com/t5/eIQ-Machine-Learning-Software/eIQ-Sample-Apps-Overview/ta-p/1122925

Is there a similar link to the broken one?

Thanks and regards.

[joanxie]

try this:

"https://community.nxp.com/t5/Software-Community-Articles/eIQ-Sample-Apps-TFLite-Quantization/ba-p/11..."