i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLM SUMMARY We run one quantized ONNX on i.MX95, once under CPUExecutionProvider and once under NeutronExecutionProvider. The two arms do not give the same answers: on 2,466 paired multiple-choice items, 25.7 % receive a different answer (43.6 % on MMLU). In free generation the same thing happens - on MMLU prompts, 13 of 19 generations produce a different answer letter. This is not a difference in wording. It is a difference in what the model answers. Meanwhile aggregate benchmark accuracy moves by only -1.8 points. We would like to understand what the Neutron runtime does numerically that produces this, and whether this magnitude is expected. SETUP Reproducible on your side: the model is from UG10166 Table 2 and the quantization is your own recipe, unmodified. - Board: i.MX95 19x19 EVK (IMX95LPD5EVK-19), LPDDR5 - BSP: LF6.18.2_1.0.0, device tree imx95-19x19-evk-neutron.dtb - Runtime: ONNX Runtime 1.22.0 from the BSP + NeutronExecutionProvider - Neutron Converter: 3.1.3 - Model: meta-llama/Llama-3.2-1B-Instruct - Quantization: NXP/eiq-olive rev aae820e, examples/Llama3/llama3_2-1B_Spinquant_RTN_ONNX_4bits.json - Conversion: convert_ort_models_to_neutron.py, applied to the CPU arm's model.onnx - Environment: NEUTRON_CMA_512SLOTS=6 Both arms come from one quantized model. The NPU arm is that model passed through convert_ort_models_to_neutron.py. We record a SHA-256 of the source - graph and external weights - at conversion time and re-verify it before every measurement, so there is no second quantization run and no second export. WHAT WE OBSERVE 1. One item in four gets a different answer. 2,466 paired items from MMLU, ARC-Challenge and PIQA, 0-shot, scored by log-likelihood of the candidate continuations - one forward per candidate, no generation, no sampling. For each benchmark, the share of items whose ANSWER changed, then the share whose CORRECTNESS changed: - MMLU (4 choices): 43.6 % answer changed, 27.1 % correctness changed - ARC-Challenge (4 choices): 21.3 % answer changed, 13.3 % correctness changed - PIQA (2 choices): 9.4 % answer changed, 9.4 % correctness changed - Aggregate: 25.7 % answer changed, 17.3 % correctness changed The two figures differ because a third of the changes (208 of 634) move from one wrong answer to another wrong answer - a real behavioural change that leaves accuracy untouched. PIQA, being binary, has no such blind spot, and its two figures coincide exactly. 2. The same happens in free generation - the answer itself changes. 50 prompts, greedy decoding, 64 tokens. For the MMLU prompts the expected output starts with the answer letter, so the answer can be read directly from the generation (n = 19): - Different answer letter between the two arms: 13 of 19 - Correct on CPU: 7 of 19 - Correct on Neutron: 6 of 19 - Disagreements where both arms are wrong, with different wrong answers: 6 of 13 Two thirds of the answers change, and the score is essentially the same (7 versus 6). The sample is small - we report it as an illustration; the 2,466-item measurement above is the quantitative one. An example, verbatim. The prompt lists four options and asks for a letter, and the expected answer is 😧 CPU: " A\nExplanation: The expression 9(9m + 3t) is equivalent to 81m + 27t. The best answer is A." NPU: " C\nThe best answer is C." Both are wrong, they are wrong differently, and no benchmark score records it. 3. The divergence is present in the very first forward, not accumulated. In this letter format the first generated token IS the answer, and 60 % of generations already differ there - a token produced by the prefill pass alone, before any decoding step. Across all 50 prompts the divergence curve rises steeply then flattens: 84 % by token 4, 98 % by token 64. That is what propagation of an initial difference looks like under greedy decoding, not error accumulating through the KV cache. 4. Aggregate accuracy hides all of it. 50.28 % on CPU against 48.50 % on Neutron, a -1.78 point difference, and none of the three benchmarks is individually significant. Restricting to the 634 items where the arms disagree, CPU is correct 37.1 % of the time against Neutron's 30.1 % - 235 versus 191 among decided items, i.e. 55/45 where symmetric noise would give 50/50. WHAT WE RULED OUT - Silent CPU fallback: ORT profiling reports 80 of 80 MatMulNBits on NeutronExecutionProvider, 0 on CPU. No partial placement. - Two different models: same source file, SHA-256 of graph and external weights recorded at conversion and re-verified before scoring. - Sampling: greedy decoding throughout; no temperature, no top-k, no top-p. - Different inputs: both arms consume the same items file, in the same order, from the same fixed seed. - Scoring artefacts: the board only captures per-token log-probabilities; all decision logic runs offline on the host, identically for both arms. - Run-to-run noise on the NPU: replaying the same items in the same Neutron session yields bit-identical log-probabilities. The NPU arm is reproducible; the discrepancy is against CPU, not against itself. OUR QUESTION Is this expected behaviour for the INT4 path on Neutron-S - and if it is, how should we validate an LLM deployment on the NPU, given that aggregate benchmark accuracy clearly does not surface it? We are not assuming a defect. Some difference between a floating-point CPU kernel and an integer NPU path is normal, and we would like to know what magnitude you consider normal, and what criterion you use yourselves to accept an LLM port to Neutron. If a quarter of answers changing is within expectations, that is a useful thing for us to know and to design around. Re: i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLM Thanks for your quick answer ! D.S Re: i.MX95 Neutron: NPU/CPU discrepancy on one answer in four with the same INT4 LLM Hi @DamienSCHNEBELEN
IMX95 NPU can only run matmul, and LLM performance on NPU is actually only average. Therefore, the phenomenon you observed is within the predictable range, and we do not recommend running LLM models on an NPU.
B.R
記事全体を表示
