2410859_en-US

[#1]

i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLM

SUMMARY

We run one quantized ONNX on i.MX95, once under CPUExecutionProvider and once under
NeutronExecutionProvider. The two arms do not give the same answers: on 2,466 paired
multiple-choice items, 25.7 % receive a different answer (43.6 % on MMLU). In free
generation the same thing happens - on MMLU prompts, 13 of 19 generations produce a
different answer letter.

This is not a difference in wording. It is a difference in what the model answers. Meanwhile aggregate benchmark accuracy moves by only -1.8 points.

We would like to understand what the Neutron runtime does numerically that produces
this, and whether this magnitude is expected.

SETUP

Reproducible on your side: the model is from UG10166 Table 2 and the quantization is
your own recipe, unmodified.

- Board: i.MX95 19x19 EVK (IMX95LPD5EVK-19), LPDDR5
- BSP: LF6.18.2_1.0.0, device tree imx95-19x19-evk-neutron.dtb
- Runtime: ONNX Runtime 1.22.0 from the BSP + NeutronExecutionProvider
- Neutron Converter: 3.1.3
- Model: meta-llama/Llama-3.2-1B-Instruct
- Quantization: NXP/eiq-olive rev aae820e,
examples/Llama3/llama3_2-1B_Spinquant_RTN_ONNX_4bits.json
- Conversion: convert_ort_models_to_neutron.py, applied to the CPU arm's model.onnx
- Environment: NEUTRON_CMA_512SLOTS=6

Both arms come from one quantized model. The NPU arm is that model passed through
convert_ort_models_to_neutron.py. We record a SHA-256 of the source - graph and
external weights - at conversion time and re-verify it before every measurement, so
there is no second quantization run and no second export.

WHAT WE OBSERVE

1. One item in four gets a different answer.

2,466 paired items from MMLU, ARC-Challenge and PIQA, 0-shot, scored by log-likelihood
of the candidate continuations - one forward per candidate, no generation, no sampling.
For each benchmark, the share of items whose ANSWER changed, then the share whose
CORRECTNESS changed:

- MMLU (4 choices): 43.6 % answer changed, 27.1 % correctness changed
- ARC-Challenge (4 choices): 21.3 % answer changed, 13.3 % correctness changed
- PIQA (2 choices): 9.4 % answer changed, 9.4 % correctness changed
- Aggregate: 25.7 % answer changed, 17.3 % correctness changed

The two figures differ because a third of the changes (208 of 634) move from one wrong
answer to another wrong answer - a real behavioural change that leaves accuracy
untouched. PIQA, being binary, has no such blind spot, and its two figures coincide
exactly.

2. The same happens in free generation - the answer itself changes.

50 prompts, greedy decoding, 64 tokens. For the MMLU prompts the expected output starts
with the answer letter, so the answer can be read directly from the generation
(n = 19):

- Different answer letter between the two arms: 13 of 19
- Correct on CPU: 7 of 19
- Correct on Neutron: 6 of 19
- Disagreements where both arms are wrong, with different wrong answers: 6 of 13

Two thirds of the answers change, and the score is essentially the same (7 versus 6).
The sample is small - we report it as an illustration; the 2,466-item measurement above
is the quantitative one.

An example, verbatim. The prompt lists four options and asks for a letter, and the
expected answer is

CPU: " A\nExplanation: The expression 9(9m + 3t) is equivalent to 81m + 27t.
The best answer is A."
NPU: " C\nThe best answer is C."

Both are wrong, they are wrong differently, and no benchmark score records it.

3. The divergence is present in the very first forward, not accumulated.

In this letter format the first generated token IS the answer, and 60 % of generations
already differ there - a token produced by the prefill pass alone, before any decoding
step. Across all 50 prompts the divergence curve rises steeply then flattens: 84 % by
token 4, 98 % by token 64. That is what propagation of an initial difference looks like
under greedy decoding, not error accumulating through the KV cache.

4. Aggregate accuracy hides all of it.

50.28 % on CPU against 48.50 % on Neutron, a -1.78 point difference, and none of the
three benchmarks is individually significant. Restricting to the 634 items where the
arms disagree, CPU is correct 37.1 % of the time against Neutron's 30.1 % - 235 versus
191 among decided items, i.e. 55/45 where symmetric noise would give 50/50.

WHAT WE RULED OUT

- Silent CPU fallback: ORT profiling reports 80 of 80 MatMulNBits on
NeutronExecutionProvider, 0 on CPU. No partial placement.
- Two different models: same source file, SHA-256 of graph and external weights
recorded at conversion and re-verified before scoring.
- Sampling: greedy decoding throughout; no temperature, no top-k, no top-p.
- Different inputs: both arms consume the same items file, in the same order, from the
same fixed seed.
- Scoring artefacts: the board only captures per-token log-probabilities; all decision
logic runs offline on the host, identically for both arms.
- Run-to-run noise on the NPU: replaying the same items in the same Neutron session
yields bit-identical log-probabilities. The NPU arm is reproducible; the discrepancy
is against CPU, not against itself.

OUR QUESTION

Is this expected behaviour for the INT4 path on Neutron-S - and if it is, how should we
validate an LLM deployment on the NPU, given that aggregate benchmark accuracy clearly
does not surface it?

We are not assuming a defect. Some difference between a floating-point CPU kernel and
an integer NPU path is normal, and we would like to know what magnitude you consider
normal, and what criterion you use yourselves to accept an LLM port to Neutron. If a
quarter of answers changing is within expectations, that is a useful thing for us to
know and to design around. 

Re: i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLMThanks for your quick answer !

D.SRe: i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLM

Hi @DamienSCHNEBELEN 

IMX95 NPU can only run matmul, and LLM performance on NPU is actually only average. Therefore, the phenomenon you observed is within the predictable range, and we do not recommend running LLM models on an NPU.

B.R