ONNXRuntime i.MX 8M-Plus vsi_npu execution provider create vx tensor fail error

[jimmyc2]

I’ve been having issues getting it to work with onnxruntime as documented in the NXP IMX Machine Learning User Guide.

I can confirm that the libvx_delegate works as expected and we see the hardware acceleration using tensorflow/tflite example, but when I try to run in onnxruntime using the vsi npu execution provider I get this error:

2022-06-28 20:43:12.921710951 [I:onnxruntime:, inference_session.cc:1329 Initialize] Session successfully initialized.
2022-06-28 20:43:12.979434216 [I:onnxruntime:, sequential_executor.cc:157 Execute] Begin execution
E [_init_tensor:461]Create vx tensor fail.
E [_create_tensor:540]Create vx tensor fail.
python3: ovxlib_delegate.cpp:839: int nnrt::OvxlibDelegate::addTensor(vsi_nn_graph_t*, vsi_nn_tensor_attr_t*, size_t, const void*, bool): Assertion `false' failed.

For information, I am using the 5.10.72_2.2.0 image and building ORT from source from https://source.codeaurora.org/external/imx/onnxruntime-imx/tree/?h=lf-5.10.72_2.2.0

The onnx graph being tested is the INT8 model from https://github.com/onnx/models/blob/main/vision/classification/mobilenet/model/mobilenetv2-12-int8.o...

Does ONNXRuntime actually work with INT8 and the VSI_NPU? The model in the guide (https://github.com/onnx/models/blob/main/vision/classification/mobilenet/model/mobilenetv2-7.onnx) is a FP32 model, which runs on CPU and will complete but defer back to the CPU if vsi_npu is selected.

[Bio_TICFSL · Accepted solution]

Hi,

According reply from R&D, VSI NPU execution provider in ONNX  support for quantized models is very limited (I would say none). And the Modelrunner only uses ONNX APIs too.

There is currently an ongoing migration from NNRT and nn-imx modules to TIM-VX and so as a part of this migration we expect the support for quantized ops. No estimates or dates for the delivery, it is a long-term plan roughly targeting the end of 2022.So, I advise ctms not to use onnxruntime.

Regards

[mattcrossmarc]

Can you please provide an update on support for NPU execution of onnx models?

Thanks,
Matt