very slow inference with pyEIQ with ONNX parser

korabelnikov — Mon, 07 Dec 2020 18:54:13 GMT

Hi,

Previously I used armnn 20.x built myself. I converted ONNX to armmn first. Then I measured time and it was 80ms-300ms in depending on model and input size. Backend was 'CpuAcc'

Now I'm trying to use pyEIQ from BSP and get time ~3 seconds with 'VsiNpu' and ~1.5 second with 'CpuAcc' on middle model.

I found it strange. Any ideas why is it and how to solve it?

(I'm not sure but I remember that I run my models with 'VsiNpu' and there was a significant gain in inference speed. I wrote down time but forget way I got it)

I suppose issue may be with ONNXParser, but I can't check .armnn model directly cause libarmnnSerializer.so hasn't built.

so few more questions:

1) how load .armnn model?

2) what armnn version and patches you uses. I'm think it's not 19.08, cause pyarmmn started from 20.x version

Thank in advance

Re: very slow inference with pyEIQ with ONNX parser

korabelnikov — Tue, 08 Dec 2020 11:45:43 GMT

upd.

The maximum speed was when I used quantized tflite models.

As instance, resnet 18 takes ~0.5 seconds on tflite and VsiNpu but on armnn and VsiNpu take several seconds.

So, can such huge performance difference be because of armnn engine?

topic very slow inference with pyEIQ with ONNX parser in eIQ Machine Learning Software

very slow inference with pyEIQ with ONNX parser

Re: very slow inference with pyEIQ with ONNX parser