Compiling and running AI models for the i.MX 95 Neutron NPU (Japanese blog)
Introduction
The eIQ Neutron NPU integrated into the i.MX95 allows you to offload AI model inference processing to a dedicated accelerator. However, it's not always possible to run the model directly; several preparations are necessary, including quantization, operator compatibility, and version compatibility between the compiler and target runtime.
This article outlines the entire process from converting a model created with ONNX or TensorFlow to TFLite, to INT8 quantization, compilation for Neutron NPUs, deployment to i.MX95, and performance evaluation using benchmark_model . While focusing on the standard and flexible Flow A , we also introduce Flow B , which leverages existing ONNX quantization environments.
The command examples are for Neutron-S. When actually using them, be sure to check the documentation and release notes that come with your installed Neutron SDK , Yocto BSP , and runtime on your target.
table of contents
overview
Host-side tool setup
Conversion procedure (Flow A details)
Alternative flow: Flow B (ONNX level quantization)
i. Deployment to MX95 target and performance evaluation
(Optional) Bit-exact verification: neutron-runner
Supported Operators and Quantization Constraints
Troubleshooting / Common Pitfalls
In conclusion
1. Overview
This article outlines the conversion procedure for running AI models trained with ONNX or TensorFlow on the eIQ Neutron NPU installed in the i.MX95. It is based on the official documentation included in the docs folder after installing the Neutron SDK: "Neutron SDK User Guide," "Neutron Compiler User Guide," "TFLite Profiler/Quantizer User Guide," "Neutron Runner User Guide," "Supported Operators (Neutron-S / Neutron-C)," and the i.MX Machine Learning User Guide (UG10166) .
ℹ️ Note: The Neutron NPU in the i.MX95 is classified as the "Neutron-S" architecture (subsystem type). It differs from "Neutron-C" for MCUs such as the i.MXRT700 in terms of conversion flow and some tool options. This article focuses on the i.MX95 (Neutron-S).
1.1 Overall Flow (Flow A: Recommended)
The standard flow provided by the Neutron SDK involves converting the model to TFLite format, profiling it on a representative dataset, then quantizing it using INT8, and finally compiling it for the NPU with neutron-compiler .
ℹ️Supplementary information for first-time users
The i.MX95's Neutron NPU actually runs on INT8 quantized TFLite models compiled for the Neutron NPU . This procedure will ultimately convert your ONNX/TensorFlow model to this format.
The starting step will vary depending on the format of the pre-trained model you have prepared. Please use the following as a guide and start from the step that applies to you.
INT8 quantized TFLite model: Skip steps 1-3 and start from step 4.
FP32 TFLite model: Skip step 1 and start from step 2 onwards.
ONNX/TensorFlow FP32 model: Perform steps 1 through 1.
procedure
Implementation details
Main deliverables/execution environment
1
Convert from ONNX to TFLite FP32
onnx2tflite /Host PC
2
Profiling with representative datasets
tflite-profiler , profile.csv / Host PC
3
INT8 quantization based on profile
tflite-quantizer , quantized.tflite / Host PC
4
Compiled for Neutron NPU
neutron-compiler --target imx95 /Host PC
5
Bit-exact verification as needed
neutron-runner /Host PC
6
Deploy the converted model to the target.
.tflite Model/i.MX95 board
7
Inference execution and performance evaluation
benchmark_model + libneutron_delegate.so / i.MX95 board
1.2 Two Quantization Flows (Flow A / Flow B)
The Neutron SDK User Guide defines two flows for obtaining a quantized TFLite model.
flow
pipeline
Quantization tools
Features
Flow A (Recommended)
ONNX FP32 → TFLite FP32 → TFLite INT8 → Neutron conversion
tflite-profiler + tflite-quantizer (included with Neutron SDK)
It offers a high degree of quantization flexibility. Specific operators can be excluded from quantization, allowing mapping to the Neutron Interpolator while maintaining FP precision.
Flow B
ONNX FP32 → ONNX INT8 → TFLite INT8 → Neutron conversion
onnx2quant (eIQ ONNX2TFLite)
Useful when an existing ONNX-based quantization pipeline is in place. Fine-grained operator exclusion control is limited.
💡 Hint: If you're unsure, we recommend Flow A (tflite-profiler + tflite-quantizer). Flow A is essential if you want to map certain operators (such as nonlinear activation functions) to the Neutron Interpolator without quantizing them.
2. Setting up the host-side tools
Model conversion will be performed on the development host (Linux/Windows PC). Install the following tools from the NXP eIQ repository.
# ONNX → TFLite Conversion Tool
pip install eiq-onnx2tflite --index-url https://eiq.nxp.com/repository/
# eIQ Neutron SDK (including tflite-profiler, tflite-quantizer, neutron-compiler, neutron-runner, etc.)
pip install eiq-neutron-sdk --index-url https://eiq.nxp.com/repository/
ℹ️ Note: Recommended minimum configuration for development host (neutron-compiler): CPU 2GHz or higher x 8 cores or higher, RAM 4GB or higher. Neutron-S (i.MX95, etc.) uses a constraint programming solver, so conversion time may take several minutes to several tens of minutes depending on the model.
ℹ️If pip install fails
If pip install fails to complete in your internal proxy environment, please consider the following options.
Using the zip version of eIQ Toolkit — After obtaining and extracting the zip file from the eIQ Toolkit product page , you can run the Neutron tools under bin/ . The zip version uses hyphens, as in tflite-profiler , tflite-quantizer , neutron-compiler , and neutron-runner . On the other hand, the pip/uv version may use underscores, as in tflite_profiler , tflite_quantizer , and neutron_compiler . However, onnx2tflite and onnx2quant are not included in the zip version and must be installed using pip or uv. The following command examples use notation equivalent to the zip version.
Install using uv — this may be more successful than pip install depending on your environment.
pip install uv
uv pip install eiq-onnx2tflite \
--index-url https://eiq.nxp.com/repository/ \
--extra-index-url https://pypi.org/simple/
If a proxy is required, you may need to configure http_proxy / https_proxy (including uppercase environment variables). The configuration method depends on your company's environment.
2.1 Neutron SDK Version and Yocto BSP Compatibility
The Neutron Driver, Firmware, and Compiler must all be the same Neutron SDK version. If they are not compatible, the following warning will be output at runtime.
Microcode version mismatch! Neutron Driver is 3.1.0-0X359f358d but model
converted with 3.0.1-0Xd5d7918e. Runtime (driver, firmware) should be aligned with compiler
Compatibility between Yocto BSP versions and Neutron SDK versions (excerpt, based on the latest BSP as of Q3 2026):
Yocto BSP
Support Release Period
Default Neutron SDK
LF_6.12.3_1.0.0
Q1/2025
1.2.0
LF_6.12.20_2.0.0
Q2/2025
2.0.2
LF_6.12.34_2.1.0
Q3/2025
2.1.3
LF_6.12.49_2.2.0
Q4/2025
2.2.2
LF_6.18.2_1.0.0
Q1/2026
2.2.3
LF_6.18.20_2.0.0
Q2/2026
3.1.2
LF_6.18.37_2.1.0
Q3/2026
3.2.1
Other Neutron SDK versions can also be used as "Upgradable," but in that case, you will need to update the target runtime (Driver/Firmware/Delegate) to the corresponding version and re-compile the model with the same version of neutron-compiler. For details, please refer to the version compatibility table in the "Neutron SDK User Guide."
3. Conversion Procedure (Flow A Details)
Step 1: Convert the ONNX model to TFLite FP32 format.
If your model is already in TFLite FP32 format, this step is unnecessary. Use the onnx2tflite tool to convert your ONNX model to TFLite format.
onnx2tflite -o .tflite .onnx
ℹ️ Note regarding SDK version
-i (Input specification) and -target (FP32/INT8 switching) are deprecated in some SDK versions. The input ONNX file is specified using positional arguments, and check the latest options with onnx2tflite --help before execution. If you change the input/output layout, also check the presence and default values of --keep-io-format / --no-keep-io-format in the SDK you are using.
Step 2: Profiling the FP32 model
Using tflite-profiler, we measure the dynamic range (min/max/histogram) of each tensor using a representative input dataset (calibration data).
./tflite-profiler \
--input < my_model_fp32 > .tflite \
--dataset < input_data_dir > \
--output < my_model_fp32_profile > .csv
ℹ️Notes regarding input data
This contains a .bin file in float32 format that matches the input dimensions of the model. For image models, you can use scripts/serialize_image.py included with the SDK to resize, normalize, and convert to float32 .bin . --use_random_data , used when there is no representative dataset, is for shape verification and communication verification, but is not suitable for accuracy verification.
./tflite-profiler \
--input < my_model_fp32 > .tflite \
--use_random_data \
--random-data-samples 10 \
--output < my_model_fp32_profile > .csv
A sample size of 200 to 1000 samples is usually considered sufficient for a representative dataset. For accuracy validation of quantization results, it is recommended to use a separate validation dataset that was not used for calibration.
Step 3: INT8 quantization based on profile information
We will use tflite-quantizer to perform INT8 quantization based on the profile (CSV) generated in step 2.
./tflite-quantizer \
--input < my_model_fp32 > .tflite \
--profile < my_model_fp32_profile > .csv \
--output < my_model_quantized > .tflite
The main quantization parameters (default values) are as follows:
option
explanation
Default value
--quantization-constant-type
Weight quantization data type
INT8
--quantization-constant-type-bias
Quantized data types with bias (Conv2D/FC, etc.)
INT32
--quantization-variable-type
Quantized data type of activation
INT8
--quantization-constant-granularity
Granularity of weight quantization (PTQ/PCQ)
PCQ (per channel)
--quantization-constant-schema
Weight quantization schema
Symmetric
--quantization-variable-schema
Activation Quantization Schema
Asymmetric
--quantization-calibration-method
Calibration method (MinMax/Percentile)
MinMax
To exclude a specific operator from quantization, use --quantize-operator-types-except (e.g., ADD,AVERAGE_POOL_2D ). Quantization-sensitive compound operators like LayerNorm can also be excluded using layernorm .
💡 Hint: After quantization, you can verify the operation with a dummy input using the --run-after-quantize option (only valid for standard INT8 quantization configurations).
ℹ️Supplementary information regarding SDK versions
Quantization options may vary depending on the SDK. In our verification environment, SDK 3.2.3 has been confirmed to work with the options shown in the table and --quantization-constant-granularity PCQ , but please be sure to verify with tflite-quantizer --help of your target SDK before use.
Step 4: Compiling for Neutron NPU
Compile the quantized TFLite model using neutron-compiler for the i.MX95 Neutron NPU.
./neutron-compiler \
--input < my_model_quantized > .tflite \
--output < my_model_converted > .tflite \
--target imx95
neutron-compiler performs the conversion in the following three stages:
step
Content
① Optimize
Transform the graph into a mathematically equivalent form that conforms to Neutron hardware constraints (graph path).
② Extract
Hierarchically mapping corresponding operators from NeutronOperator to custom operators in NeutronGraph.
③ Generate
Tile partitioning, scheduling, memory allocation, and binary generation of microcode/weights/kernels.
You can check the list of supported targets and operator types using the following command.
neutron-compiler --show-targets
neutron-compiler --show-operator-types
neutron-compiler --target imx95 --show-kernel-kinds
3.1 Major Compiler Options
option
explanation
--target
The target Neutron object to be converted (e.g., imx95). If omitted, it defaults to mcxn94x.
--output
Output file path. If omitted, _compiled will be appended to the input name.
--dump-statistics / --dump-statistics-file
Output cycle estimation statistics (per NeutronGraph) to console/file.
--optimization-level
OFast : Fast heuristic (default) / OOpt : Optimal solution using ILP/SAT solver (conversion time is long)
--include-operator-types / --exclude-operator-types
Specify the operator types to map to/exclude from the NPU.
--min-num-ops-per-graph
The minimum number of operators to extract as a NeutronGraph. Smaller graphs are optimized for CPU execution.
--use-profiling / --use-profiling-tick-trace
Enable runtime profiling on actual hardware (kernel level/tick level)
--keep-graphs
Preserve the intermediate graph (necessary for bit-exact verification in neutron-runner)
--force-determinism
This suppresses multithreaded contention in the solver and obtains definitive conversion results (conversion time increases by approximately 2-3 times).
ℹ️ Note: Because Neutron-S (i.MX95, etc.) compilers use a constraint programming solver, the generated microcode (especially the TCM address) may differ slightly when the same model is converted multiple times. This is because one of several optimal solutions is selected due to multithreaded contention, and all are correct results. Use --force-determinism when determinism is required.
4. Alternative Flow: Flow B (ONNX Level Quantization)
If your existing pipeline relies on ONNX-level quantization ( onnx2quant ), you can also use the following flow. Fine-grained operator exclusion control is more limited than in Flow A.
procedure
Implementation details
Tools used / Main deliverables
1
Quantizing the ONNX FP32 model to INT8
onnx2quant , model_int8.onnx
2
Converting from ONNX INT8 to TFLite INT8
onnx2tflite , model_int8.tflite
3
Compiled for Neutron NPU
neutron-compiler --target imx95
# ① ONNXレベルでINT8量子化
# -cには入力テンソル名と.npyキャリブレーションデータのディレクトリを指定
onnx2quant .onnx -o .onnx -c " ; "
# ② ONNX INT8 → TFLite INT8変換（QDQ形式の場合）
onnx2tflite --qdq-aware-conversion -o .tflite .onnx
# ③ Neutron NPU用コンパイル
neutron-compiler --input .tflite --output .tflite --target imx95
ℹ️Flow B Notes
onnx2quant Then, either -c (calibration data specification) or -u (random calibration data) is required. For multiple input models, specify -c for each input. Use --qdq-aware-conversion when converting ONNX in QDQ format. Before execution, check the options of the SDK being used with onnx2quant --help / onnx2tflite --help .
5. Deployment to the i.MX95 target and performance evaluation
5.1 Neutron Runtime Configuration
The Neutron runtime for i.MX95 (Neutron-S) consists of the following three components:
component
file
Location
Neutron Firmware
NeutronFirmware.elf
/lib/firmware/
Neutron Driver
libNeutronDriver.so
/lib/
TensorFlow Neutron Delegate
libneutron_delegate.so
/lib/
⚠️ Note: Always back up existing files before updating the runtime. If the driver, firmware, and compiler versions do not match, a "Microcode version mismatch" warning will be displayed, and the system may become unstable.
5.2 Model Deployment
Copy the converted TFLite model ( .tflite ) to the i.MX95 target board (using scp, etc.).
5.3 Performance Evaluation (benchmark_model)
We will use the TensorFlow Lite benchmark_model application included in the NXP Yocto BSP and perform performance evaluation via Neutron Delegate.
/usr/bin/tensorflow-lite-2.19.0/examples/benchmark_model \
--graph =< my_model_converted > .tflite \
--external_delegate_path = /usr/lib/libneutron_delegate.so \
--enable_op_profiling = true \
--num_threads = 5
Example output (summary by node type):
Number of nodes executed: 26
============================== Summary by node type ==============================
[Node type] [count] [avg ms] [avg %] [cdf %] [mem KB] [times called]
NeutronDelegate 1 2.882 87.919% 87.919% 0.000 1
Slice (ND, X8) 10 0.175 5.339% 93.258% 0.000 10
CONCATENATION 2 0.096 2.929% 96.187% 0.000 2
Unary Elementwise (NC) 2 0.078 2.379% 98.566% 0.000 2
Copy (NC, X8) 11 0.047 1.434% 100.000% 0.000 11
Timings (microseconds): count=270 first=3372 curr=3303 min=3015 max=3537 avg=3290.54 std=76
26 nodes observed
💡 Hint: A higher avg % in the "NeutronDelegate" row indicates that the majority of the computation is offloaded to the NPU. Other nodes (Slice, Concatenation, etc.) are executed on the CPU. If this ratio is low, unsupported operators may be causing bottlenecks, so check the constraints of each operator in the "Supported Operators" documentation.
6. (Optional) Bit-exact verification: neutron-runner
neutron-runner is a tool that verifies whether a model converted by neutron-compiler produces results that perfectly match the Neutron Golden Kernel (reference implementation). It is not used for production deployment, but is useful for verifying the correctness of models during migration.
item
Neutron Mode
Vanilla Mode
Target models
Model converted with --keep-graphs using neutron-compiler
Standard TFLite models (FP32/FP16/INT8)
Target
IMX95, IMXRT700
N/A
the purpose
bit-exact verification using Golden Kernel
Reference run (baseline for comparison)
Typical validation workflow:
# ① Compile with --keep-graphs enabled
neutron-compiler --input model.tflite --output model_neutron.tflite --target imx95 --keep-graphs
# ② Execute using the Neutron runtime (Golden Kernel)
neutron-runner --input model_neutron.tflite --dataset inputs/ --use-neutron-runtime = true --target imx95
# ③ Run with Vanilla runtime (reference)
neutron-runner --input model.tflite --dataset inputs/ --use-neutron-runtime = false
# ④ Compare results
diff model_neutron_results_0.bin model_results_0.bin
ℹ️ Note: Operators not supported by the Neutron Golden Kernel will automatically fall back to the TFLite Vanilla implementation. After execution is complete, a list of the operators to which it fell back will be displayed in the console.
neutron-runner execution path in a pip install environment
In the pip/uv version, a Python wrapper for neutron-runner is not generated, and you may need to directly specify the ELF binary within the site package.
/eiq_neutron_sdk/bin/neutron-runner \
--input model_neutron.tflite \
--dataset inputs/ \
--use-neutron-runtime=true \
--target imx95
In the zip version of eIQ Toolkit, execute bin/neutron-runner as usual.
7. Supported Operators and Quantization Constraints
There are common constraints on the operators that neutron-compiler can map to the NPU. Operators that do not satisfy these constraints will fall back to CPU execution.
7.1 Common constraints (common to Neutron-S / Neutron-C)
The shape of the tensor must be static (fixed) (batch size N is fixed at 1; Neutron-C partially supports variable batches).
The tensor type is one of UINT8 / INT8 / INT16 / INT32 / INT64.
The tensor layout is NHWC (TFLite standard).
The quantization scale must be a positive FLOAT32 value.
The weights are symmetrically quantized (offset = 0) using PTQ or PCQ.
The bias is symmetrically quantized using PTQ or PCQ, and its scale is input_scale × weights_scale
Layers that do not meet the above constraints will not be mapped to the NPU.
7.2 i. Main operators supported by MX95 (Neutron-S)
CONV_2D , DEPTHWISE_CONV_2D , FULLY_CONNECTED , ADD , SUB , MUL , AVERAGE_POOL_2D , MAX_POOL_2D , CONCATENATION , PAD , PADV2 , RESHAPE , RESIZE_BILINEAR , RESIZE_NEAREST_NEIGHBOR , MEAN , SUM , LOGISTIC , TANH , HARD_SWISH , PRELU , LEAKY_RELU , RELU systems, TRANSPOSE_CONV , UNIDIRECTIONAL_SEQUENCE_LSTM etc (see SupportedOperatorsS.md for detailed constraints).
⚠️ Note: The list of supported operators is updated with each version of the Neutron Compiler (e.g., Group Conv2D support in 3.2.1, CAST/SELECT/UNIDIRECTIONAL_SEQUENCE_LSTM added in 3.1.1). Be sure to check the release notes for the SDK version you are using.
8. Troubleshooting / Common Pitfalls
Symptoms
cause
handle
Microcode version mismatch warning
Driver/Firmware/Compiler version mismatch
All components are reconverted and rearranged to match the same Neutron SDK version.
The average percentage of NeutronDelegate is low (low NPU utilization).
The model contains many unsupported operators (such as Resize/Reduce).
Check the constraints in Supported Operators, replace them with alternative operators, or manually map the operators to the Interpolator while keeping them as FP.
Accuracy decreases significantly after quantization.
Calibration data lacks representativeness, or specific operators are sensitive to quantization.
Increase the number of calibration samples (200-1000). Exclude LayerNorm and similar values using --quantize-operator-types-except
The neutron-compiler conversion is extremely slow.
Use OOpt optimization level, or large model
First, check with OFast (default). Use OOpt only when maximum performance is required (uses up to 16 cores, increases conversion time).
In neutron-runner, some operators fall back to vanilla.
There are operators that are not supported by the Golden Kernel.
No issues found (expected operation). Check the fallback list to ensure it's not a performance-critical area.
Command not found ( command not found )
The command name and location vary depending on the installation method.
For pip/uv versions, check for underscore notation and actual paths within the site package; for zip versions, check for hyphen notation and what is under bin/
onnx2tflite Option error with onnx2quant
Differences between the included documentation and the SDK version
Check the current option name with --help before execution.
In conclusion
This article introduced a standard conversion flow for running AI models on the i.MX95 Neutron-S NPU. The following three points are particularly important for practical use:
Ensure that the Neutron SDK and the target runtime versions are matched. Inconsistencies between the Driver, Firmware, Delegate, and Compiler will result in Microcode version mismatch .
First, use Flow A to check the correspondence between quantization and operators. Profiling with a representative dataset will make it easier to adjust accuracy and mapping to the NPU.
Check the results of benchmark_model on an operator-by-operator basis. If there are many operations other than NeutronDelegate , unsupported operators or data conversions may be causing performance bottlenecks.
Model accuracy, NPU offload rate, and inference latency vary depending on the model structure, quantization settings, input data, and SDK version. It is recommended to start with a small model, go through the conversion and evaluation on a real device, and then gradually apply it to the target model if there are no problems.
The command examples and corresponding information in this article may vary depending on the version of the Neutron SDK you are using. For the latest official specifications, please refer to the documentation and release notes included with your SDK.
=========================
We are currently unable to respond to comments left in the "Comment" section of this post. We apologize for the inconvenience, but please refer to " Technical Questions to NXP - How to Contact Us( Japanese Blog) " when making an inquiry. (If you are already an NXP distributor or have a relationship with NXP, you may ask your representative directly.)
This is a practical guide for running AI models on the i.MX95's eIQ Neutron NPU.
This document explains the process from converting an ONNX/TensorFlow model to TFLite, quantizing it to INT8, compiling it for Neutron NPUs, deploying it to the target, and evaluating its performance using benchmark_model. Japanese Blog i.MX Processors Technology Focus
View full article
