Unsupported Node in onnxruntime with vsi npu

[phillin702]

I am using IMX 8M Plus,and deploying the model in NPU with onnxruntime. My code is download from https://source.codeaurora.org/external/imx/onnxruntime-imx/tree/csharp/test/Microsoft.ML.OnnxRuntime... .

When I set execution_provider to CPU,it works fine, as shown below.

when I set execution_provider to vsi_npu, it works very slowly, and flag some unsupported nodes, as shown below.

Is this because nodes which are not supported switch from NPU to CPU ??

Or is there something wrong with our operation??

[Zhiming_Liu]

PRelu is not in VSI NPU node support list , that log shows that this operation node has been switched to CPU. 

[phillin702]

Thank you for your reply ， But  Appendix C Neural network API reference in 《i.MX Machine Learning User's Guide》，it show that the prelu operation is supported，Didn't it mean NPU support?

[Zhiming_Liu]

Hi @phillin702 

The npu driver and onnxruntime driver is also updating in each BSP release,the hardware supports it but the driver hasn't. 

[phillin702]

Thank you for your reply. Does this mean I have an incorrect driver version and need to update the driver?

[Zhiming_Liu]

I will test the lastest version and reply here.

[phillin702]

Thank you for your reply. In addition, are other operations such as BN and Flatten also not supported? Do you have a list of operators supported by ONNX NPU?

[Zhiming_Liu]

Have you include the npu provider when you compile this source code?

Test on lasteset BSP[L5.15.52] with vsi_npu:

root@imx8mpevk:~# ./onnxruntime_sample
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
Using Onnxruntime C API
Number of inputs = 1
Input 0 : name=data_0
Input 0 : type=1
Input 0 : num_dims=4
Input 0 : dim 0=1
Input 0 : dim 1=3
Input 0 : dim 2=224
Input 0 : dim 3=224
Score for class [0] = 0.000045
Score for class [1] = 0.003846
Score for class [2] = 0.000125
Score for class [3] = 0.001180
Score for class [4] = 0.001317
Done!

[phillin702]

Hello, do you have any new progress?

[Zhiming_Liu]

Previous reply contains my test log under VSI NPU with L5.15.52

The NPU test can pass with Prelu.

[phillin702]

Thank you for your reply. Yes, we had include the npu provider when we compile this source code.