i.MX95: neutron_perf.sh shows false 100% NPU Load (perf_counters line is read byte by byte) Hello, On i.MX95, neutron_perf.sh (eiq-examples-npu-utils, UG10166 section 8.4.5.1) sometimes shows 100% NPU Load while the actual load is about 60%. Environment - Board: IMX95LPD5EVK-19 - BSP: LF_6.18.20_2.0.0 (neutron_perf.sh is the same in lf-6.18.37_2.1.0) - eIQ Neutron SDK 3.2.3 Steps to reproduce 1. Convert the MobileNet model in the BSP (mobilenet_v1_1.0_224_quant.tflite in the TensorFlow Lite examples folder) with neutron-compiler for imx95. 2. On the board, run benchmark_model with the converted model and the Neutron delegate in the background, and run neutron_perf.sh at the same time. 3. The actual load is about 60%, but 100% was shown 11 times in 60 seconds. Cause 1. The perf_counters file in debugfs is not seekable, so the bash "read" command reads it one byte at a time. The driver function perf_counters_read() rebuilds the line on every read call, so one line mixes values from different moments, and some values break. In our test, 52 of 300 reads with "read" were broken, and 0 of 300 with "cat". 2. When a 32-bit counter reaches its maximum value, the driver resets it to 0. The script takes the absolute value of the negative difference, which gives 100%. Proposed fix (tested: no 100% in the same test) 1. Read the whole line at once (for example with cat), and then split it into the variables. 2. When the NACT difference is negative, add 2 to the power of 32 instead of taking the absolute value. We can share the patch if needed. On the driver side, using seq_file for perf_counters would probably also prevent cause 1 (not verified). Note: even with the fix, the NPU Load drops for one interval each time NACT reaches its maximum value (about every 7 seconds at 60% load), because NACT stops there until the next read resets it. Questions 1. Is this a known issue? 2. Could the fix be included in a future BSP release? Thank you. Re: i.MX95: neutron_perf.sh shows false 100% NPU Load (perf_counters line is read byte by byte) Hello, This is a gentle follow-up, as there has been no reply for two weeks. To make the report easier to review, I have attached the proposed fix as a patch (neutron_perf_fix.patch.txt). It changes only two lines of neutron_perf.sh, as described in my first post: 1. Read the perf_counters line all at once with cat, and then split it into the variables. 2. When the NACT difference is negative, add 2 to the power of 32 instead of taking the absolute value. The patch is against npu_utils/neutron_perf.sh in the nxp-imx/eiq-example repository on GitHub (branch lf-6.18.20_2.0.0). This is the script described in UG10166 (i.MX Machine Learning User's Guide), section 8.4.5.1: https://www.nxp.com/docs/en/user-guide/UG10166.pdf Could someone from NXP check the two questions in my first post? 1. Is this a known issue? 2. Could this issue be fixed in a future BSP release? The attached patch is only a suggestion. A different fix, for example on the driver side, is also fine. Thank you.
View full article