2200021_en-US

[#1]

Inference Degradation on i.MX95 Neutron NPU

Dear NXP Technical Support Team,

We are currently developing a pose estimation model using ResNet as the backbone, and are considering deploying it on a product based on the i.MX95 platfo

However, we are encountering significant degradation in inference accuracy when running quantized models on the Neutron NPU. Our investigation revealed that even with standard classification models, quantized models that perform correctly on the CPU show degraded results after conversion for NPU execution. This issue also affects ResNet, which we use as our backbone.

Through partial NPU assignment testing using neutron-converter, we confirmed that the degradation occurs when layers other than specific Conv2D layers are assigned to the NPU.

We would appreciate your insights regarding the following:

1. What could be the cause of the inference degradation?
   For example, quantization or conversion parameter issues, hardware limitations of the NPU, or bugs in the neutron-converter.

2. Are there any known issues related to this problem?
   For example, bugs in BSP 6.12.20 or eIQ Toolkit 1.16.0, known limitations when running ResNet models on the NPU, or any other documented issues with the i.MX95 NPU.

3. What countermeasures can we take to mitigate this issue?

Development Environment

• Hardware: i.MX95 EVK
• BSP Version: 6.12.20_2.0.0(NXP)
• eIQ Toolkit: 1.16.0 (Windows)

Model Conversion Pipeline

• PyTorch → ONNX: torch.onnx (opset version 15)
• ONNX → TensorFlow (.pb): PINTO0309/onnx2tf
• INT8 Quantization: tensorflow.lite.TFLiteConverter (version 2.12.0)
• NPU Conversion: neutron-converter (version 2.0.2+0X0cebb80a)

Inference Results

When executing on the NPU, the inference accuracy drops drastically.
We tested with quantized models of MobileNetV2 and ResNet18 (from torchvision) using ImageNetV2 (500 samples):

model	float32 CPU Accuracy (Top1 / Top5)	int8 CPU Accuracy (Top1 / Top5)	int8 NPU Accuracy (Top1 / Top5)
MobileNetV2	70.2% / 89.4%	68.4% / 88.6%	0.2% / 0.8%
ResNet18	64.0% / 86.6%	63.6% / 87.4%	0.0% / 0.6%

Additional Test

We performed partial NPU assignment tests using the --include-operators option in neutron-converter on the quantized ResNet18 model.
We observed that assigning layers other than specific Conv2D layers to the NPU leads to inference degradation.

case	NPU assignment	result	Accuracy Top1 / Top5
Baseline	none (all CPU)	ー(baseline)	63.6 / 87.4 %
Case 1	all	bad	0.0 / 0.6 %
Case 2	Conv2D layers which in/out shape is [1×56×56×64]	good	64.0 / 87.4 %
Case 3	Case 2 + Add layers	bad	0.0 / 0.6 %
Case ４	Case 2 + other Conv2D layers	bad	0.0 / 0.8 %

Thank you so much for your support.

Best regards,
Ryosuke

Re: Inference Degradation on i.MX95 Neutron NPU

Hello,

Please note that the i.MX95 is still in pre-production stage so we cannot offer support to it, please contact your local NXP Sales/NXP FAE for further assistance.

Best regards/Saludos,
Aldo.