2202942_en-US

[#1]

i.MX RT700 eIQ Neutron NPU Lab Guides

These lab guides provide step-by-step instructions on how to take a quantized TensorFlow Lite model and use the Neutron Compiler Tool found in eIQ Neutron SDK to convert the model to run on the eIQ Neutron NPU found on i.MX RT700 devices. 

The eIQ Neutron NPU for i.MX RT700 Lab Guide documents focus on using the Neutron Compiler tool found inside eIQ Neutron SDK  to convert a model and then import that converted model into an eIQ MCUXpresso SDK example. There are labs for VSCode, GCC, and MCUXpresso IDE. 

The labs were designed to run on the i.MX RT700 EVK, but the same concepts can be applied to MCX N boards as well and are similar to the MCX N eIQ Neutron NPU labs. You can also explore the TFLM Getting Started Guide for information on how to use your own model and data for inference. 

Also be sure to check out AN14700 - i.MX RT700 eIQ Neutron NPU Enablement and Performance which goes into more details on the eIQ Neutron N3-64 NPU found on i.MX RT700. 

The VS Code is copied below but is also included as an attached PDF below as well as labs for using ARM GCC and MCUXpresso IDE. 

1                 Lab Overview

This document will cover how to convert models using NXP’s eIQ Neutron SDK and will also highlight the performance improvements that can be achieved with the eIQ Neutron NPU.

This version of the lab will use VSCode. If using command line GCC or MCUXpresso IDE see those versions of the lab.

2               Software and Hardware Installation

This section will cover the hardware and software needed for this lab.

2.1 Hardware

The i.MX RT700 EVK is used in this lab.

2.2 NXP Software Installation

1. Download the latest eIQ Neutron SDK
2. Install VSCode.
3. Install the latest MCUXpresso for VSCode plugin.
4. Run the MCUXpresso Installer tool and install three key components:
   1. MCUXpresso SDK Developer
   2. Arm GNU Toolchain
   3. LinkServer
   4. anthony_huereca_0-1786510346209.pnganthony_huereca_0-1786510346209.pnganthony_huereca_0-1787639704899.pnganthony_huereca_0-1787639704899.png
5. Download a quantized Mobilenet TFLite model that can be found here and rename it to mobilenet_quant.tflite on your hard drive.
6. Install the latest LinkServer CMSIS-DAP debug firmware on your board by putting a jumper on JP20, unplugging and plugging in the micro-USB cable on J54, and running the C:\NXP\LinkServer_\MCU-LINK_installer\scripts\program_CMSIS.cmd script. Then remove the jumper and do a power on reset.

3               Label Image Example

This section will use the eIQ Label Image example found in the MCUXpresso SDK to showcase how the eIQ Neutron NPU can significantly decrease inference times for quantized models.

3.1 Convert Models

Use eIQ Neutron SDK to convert a pre-existing Mobilenet model into a Neutron optimized model.

1. Download a quantized Mobilenet TFLite model that can be found here and rename it to mobilenet_quant.tflite on your hard drive.
2. Unzip the eIQ Neutron SDK package in a directory of your choosing.
3. Optionally add \eIQ_NeutronSDK_\bin to your executable path so that the neutron-compiler utility can be directly called from the command line.
4.        Then use the following command to do the conversion (all one line):

neutron-compiler --dump-header-file-output --dump-header-file-input  --target imxrt700 --use-sequencer --input mobilenet_quant.tflite --output mobilenet_npu.tflite

anthony_huereca_1-1786510372263.pnganthony_huereca_1-1786510372263.png

These options will generate a C array of both the converted model and the input model, which we’ll use to compare the performance of them. In typical situations you would only need the dump-header-file-output option though. This command also directed the convertor to use sequencer mode which can result in faster inference times with the tradeoff of a larger model. Also note that starting in eIQ Neutron SDK 3.2.1, the neutron-converter tool was renamed to neutron-compiler.

3.2 View Models

After conversion, you can explore the models using a tool like Netron. Take a moment to look at the original model compared to the new converted model.

1. The original TFLite file: mobilenet_quant.tflite
2. anthony_huereca_2-1786510380913.pnganthony_huereca_2-1786510380913.png
   
3. The Neutron converted file: mobilenet_npu.tflite
4. anthony_huereca_3-1786510387349.pnganthony_huereca_3-1786510387349.png
   
5. You can see how almost all the operators in the original model were replaced with a NeutronGraph operator. Those NeutronGraph operaters are what will be executed on the eIQ Neutron NPU when this model is ran on the i.MXRT700. Any layers that were not converted to a NeutronGraph operator will instead be ran on the Cortex-M33 core.
6. Take a look at the file size of each of the .tflite files and you can see that, in general, the NPU converted file will take up less flash space. Note that this might be counter-acted by the slightly increased size required for using the eIQ Neutron libraries.
7. During the conversion process the dump-header-file-output argument generated the .h header file for the NPU optimized model that can be used in the eIQ MCUXpresso SDK projects.
8. The dump-header-file-input argument generated the .h header file for the original non-converted model. This will be used so the inference time of the original model that only runs on the Cortex-M33 core can be compared to the NPU converted model that makes use of the eIQ Neutron NPU.
9. So let’s run these models to see the performance improvements.

3.3 Modify an eIQ Example to Run Models

Now let’s use the MCUXpresso SDK eIQ Label Images example to run the models and see how long the inference time is.

1. Open VSCode
2. Go to the MCUXpresso for VSCode plugin and click on Import Repository
3. anthony_huereca_4-1786510398165.pnganthony_huereca_4-1786510398165.png
   
4. Go to the Remote Archive tab. It may take a bit for the list of packages to populate. Once it does, type in for RT700 in the Package field to select the MIMXRT700-EVK. Select the latest SDK version and then select a directory to download the MCUXpresso SDK into. Agree to the license and then finally click on Import. It will take several minutes to download and extract the SDK package. This time can be reduced by unchecking the Create Git repository option if not needed. Also ensure MUCXpresso SDK 26.06 or later is used as there are several important VS Code issues fixed in that version.

anthony_huereca_5-1786510404488.pnganthony_huereca_5-1786510404488.png

4. Next import an eIQ example project. In the Quickstart Panel, select Import Example from Repository
5. anthony_huereca_6-1786510411278.pnganthony_huereca_6-1786510411278.png
   
6. Then on the screen that pops up, select the RT700 repository that was downloaded in the previous step. In the Template field type in label_image to search for the eiq_examples/tflm_label_image_cm33_core0 project. Then change the App type to Freestanding application. Then select a directory location to import the project into. And then select the Arm GNU Toolchain that was installed as part of the MCUXpresso Installer. Finally click on Import.
7. anthony_huereca_7-1786510418890.pnganthony_huereca_7-1786510418890.png
   
8. A pop-up will come up that asks if you trust the authors of the files in this folder. Click on Yes.
9. anthony_huereca_8-1786510425633.pnganthony_huereca_8-1786510425633.png
   
10. It should look like the following when done. Make sure the Project Files looks similar to the image below which should be the case if the Freestanding option was used when importing the project:
11. anthony_huereca_9-1786510432167.pnganthony_huereca_9-1786510432167.png
    
12. Now we need to import the models that were generated in the last section into this project.
13. Navigate down to the Project Files folder and inside the tflm folder right click on model.cpp and then select Reveal in File Explorer to open the location of that file on your hard drive.
14. anthony_huereca_10-1786510437757.pnganthony_huereca_10-1786510437757.png
    
15. Now copy and paste the two .h header files that were generated in the previous section into this file location. It should look like the following when complete:
16. anthony_huereca_11-1786510444462.pnganthony_huereca_11-1786510444462.png
    
17. Back in VSCode, hover your mouse over the Projects name and then hit the Refresh icon to get the two new header files to appear in the Project Files list.
18. anthony_huereca_12-1786510454038.pnganthony_huereca_12-1786510454038.png
    

Note that simply having the files in the Project Files view does not mean they will automatically be included in the project. As they are header files no further changes are needed, but if C or CPP files are added to a VSCode project then other configuration files would also need to be updated. No change is needed here. This is just for informational purposes only.

12. Now we need to slightly modify those two header files to add some information for the eIQ MCUXpresso SDK project that describe how much memory this model will require and to describe some of the normalization values that this model uses:
    1. In the pcq_npu folder open model_data.h, which contains the default model for this example.

anthony_huereca_13-1786510459397.pnganthony_huereca_13-1786510459397.png

1. In model_data.h and find the following section of code and copy it. Be sure to include the array declaration.
2. anthony_huereca_14-1786510470892.pnganthony_huereca_14-1786510470892.png
   
3. Then in mobilenet_quant.h copy that code above the array, overwriting the default #defines above the array. Make sure not to erase the commented lines at the top of the file as those comments will be used later.

anthony_huereca_15-1786510489494.pnganthony_huereca_15-1786510489494.png

13. Do the same steps to update mobilenet_ npu.h as well
14. After changing both files, now double click on model.cpp to open it.
15. anthony_huereca_16-1786510501059.pnganthony_huereca_16-1786510501059.png
    
16. Go to line 27 and change it to point to the non-NPU accelerated model in mobilenet_quant.h. It should look like the following after changed: 
17. anthony_huereca_17-1786510506933.pnganthony_huereca_17-1786510506933.png
    
18. Next look at line 55 in that same model.cpp file to find where the model is loaded by the TFLM inference engine using the C array name model_data. Because the model array name in the new header file is the same as the original header file we replaced, no change is needed here. This is just for informational purposes only.

anthony_huereca_18-1786510512561.pnganthony_huereca_18-1786510512561.png

16. Likewise, the image data that will be fed into the model can be found in source image_data.h file contains an array of the binary data from the Stopwatch image found in stopwatch.bmp. No change is needed here. This is just for informational purposes only.
17. anthony_huereca_19-1786510521373.pnganthony_huereca_19-1786510521373.png
    

3.4 Compile and Run

1. Compile the project by hovering your mouse over the project name and then clicking on the Build Project icon.  

anthony_huereca_20-1786510525894.pnganthony_huereca_20-1786510525894.png

2. Connect a USB micro B cable from your computer to the USB port on the i.MX RT700 EVK at J54. Also ensure that JP1 and JP3 are shunted on the board and that SW10 has pin 1 OFF and pin 2 ON

anthony_huereca_21-1786510535725.pnganthony_huereca_21-1786510535725.pnganthony_huereca_22-1786510540192.pnganthony_huereca_22-1786510540192.png

3. Open TeraTerm or other serial terminal program, and connect to the virtual COM port that board enumerated as when you plugged in the USB cable (your COM number will likely be different than the screenshot). Use 115200 baud, 1 stop bit, no parity. There is a built-in serial terminal in VSCode that can be used for this. Click on Start Monitoring to connect:     
4. anthony_huereca_23-1786510547583.pnganthony_huereca_23-1786510547583.png
   
5. Then in VSCode hover your mouse over the project name and click on Debug
6. anthony_huereca_24-1786510554219.pnganthony_huereca_24-1786510554219.png
   
7. You should see VSCode connect and download the program to your board in the Console tab.

6. Once complete, it will pause at the start of main(). Hit the Resume icon  to run the program and look at the Serial Monitor tab.
7. anthony_huereca_25-1786510561242.pnganthony_huereca_25-1786510561242.png
   
8. When you run the project, if you look at the terminal output, it looks like there’s an error that we’ll fix in the next step: Didn't find op for builtin opcode 'CONV_2D'
9. anthony_huereca_26-1786510567902.pnganthony_huereca_26-1786510567902.png
   
10. Stop the debugger by clicking on the red square in the debug panel
11. anthony_huereca_27-1786510573274.pnganthony_huereca_27-1786510573274.png
    
12. This error was done on purpose in this lab, to demonstrate that when changing models, the list of operators needs to be updated as well.
13. To fix the error, open the model_mobilenet_ops_npu.cpp file in the pcq_npu folder   

anthony_huereca_28-1786510578901.pnganthony_huereca_28-1786510578901.png

11. Inside the MODEL_GetOpsResolver function is a list of operators. If you open the mobilenet_quant.h header file you’ll also find a list of operators used by the model in the comment block at the top of the file.

anthony_huereca_29-1786510584088.pnganthony_huereca_29-1786510584088.png

12. Copy that list from mobilenet_quant.h into the MODEL_GetOpsResolver function in model_mobilenet_ops_npu.cpp to replace the variable declaration and the original list.
13. anthony_huereca_30-1786510602975.pnganthony_huereca_30-1786510602975.png
    
14. Recompile and reprogram the board using the previous steps.
15. You should now see the following on the serial terminal:
16. anthony_huereca_31-1786510608936.pnganthony_huereca_31-1786510608936.png
    
17. Stop the debugger by clicking on the red square
18. anthony_huereca_32-1786510615586.pnganthony_huereca_32-1786510615586.png
    
19. Now we’ll run the program again, but this time with the Neutron NPU accelerated version of the model.
20. Re-open model.cpp and this time change line 27 to point to the Neutron NPU converted version of the model in the mobilenet_ npu.h file:
21. anthony_huereca_33-1786510622508.pnganthony_huereca_33-1786510622508.png
    
22. Re-open model_mobilenet_ops_npu.cpp and update the MODEL_GetOpsResolver function with the variable declaration and the operators listed in the comment block of the mobilenet_npu.h file. Note that the operators needed here may change depending on the Neutron Compiler version.
23. anthony_huereca_34-1786510631401.pnganthony_huereca_34-1786510631401.png
    
24. Build and program the program as before.
25. However there’s now another error with a “Microcode version mismatch”. The model output also has the wrong answer for the image. This is because the Neutron libraries included by default in MCUXpresso SDK 26.06 are for eIQ Neutron SDK 3.1.1 but we used a newer eIQ Neutron SDK to convert our model.
26. anthony_huereca_35-1786510645979.pnganthony_huereca_35-1786510645979.png
    

The default library version can be confirmed by looking at the default model data in model_data.h where it shows it was converted using an older version. 

anthony_huereca_36-1786510650993.pnganthony_huereca_36-1786510650993.png

But the model that was converted as part of this lab used a newer Neutron Compiler version

anthony_huereca_37-1786510654472.pnganthony_huereca_37-1786510654472.png

21. To fix this error, the Neutron libraries used by this project need to be updated to the newer eIQ Neutron SDK version. Go to where eIQ Neutron SDK was unzipped and navigate to the eIQ_NeutronSDK_\target\imxrt700 folder
22. Then use VSCode to open the directory to copy those files by right clicking on the Project Files->mimxrt700evk_cm33_core0->cm33  folder and then select Reveal in File Explorer to open the location of that file on your hard drive.

anthony_huereca_2-1787639778651.pnganthony_huereca_2-1787639778651.png

22. Then overwrite the files from the eIQ Neutron SDK folder into your project:

File Name	Source Directory in eIQ Neutron SDK	Target Directory in Project
libNeutronDriver.a	target\imxrt700\rt700\cm33	\mimxrt700evk_cm33_core0\cm33
libNeutronFirmware.a	target\imxrt700\rt700\cm33	\mimxrt700evk_cm33_core0\cm33
NeutronDriver.h	target\imxrt700\driver\include\	\mimxrt700evk_cm33_core0\driver_include
NeutronErrors.h	target\imxrt700\common\include\	\mimxrt700evk_cm33_core0\include

Note: The target\imxrt700\driver\include\NeutronEnvConfig.h file and the libraries in target\imxrt700\cmodel are used by the ExecuTorch inference engine and so are not needed for this TFLM example.

25. After the new Neutron libraries are copied over, clean the project to ensure the new libraries will be used
26. 
    
27. Then recompile and debug the project as done before
28. This time you should see the following on the terminal. That’s over a 100x improvement in inference time, with the same confidence percentage on this static image.
29. 
    
30. The decrease in inference time is very model dependent depending on how well that specific model could be optimized for the NPU.
31. Due to the changes made as part of this lab inside the Repository directory, which is shared among SDK projects, other eIQ examples will also use the newer Neutron library now. This means those examples will not properly work as the default eIQ projects use a model converted with that earlier version of eIQ Neutron SDK.

4              Further Optimizations

There are several items that can be further optimized for your particular model

1. The terminal output also shows the TensorArena Size used for this model. The kTensorArenaSize variable that is set in mobilenet_npu.h can be adjusted accordingly to reduce the memory usage. This kTensorArenaSize variable is used to determine the size of the TensorArena memory buffer required by the model for scratch data during calculation. It is also estimated and printed out during the neutron-compiler output in the Total data field. However this estimate is often slightly smaller than the actual amount used, which is printed in the serial terminal during inference:
2. 
   
   
3. The power used while inferencing can be further reduced from the eIQ projects by turning off unnecessary clocks in active mode. See the power_comp_only SDK example.
4. The NPU clock can also be turned off while not actively inferencing with:

• CLKCTL0->PSCCTL5_CLR |= (1UL << CLKCTL0_PSCCTL5_NPU0_SHIFT);

The NPU module clock can be turned on right before the call to MODEL_RunInference() with:

• CLKCTL0->PSCCTL5_SET |= (1UL << CLKCTL0_PSCCTL5_NPU0_SHIFT);

4. Always make sure to use the newest Neutron Compiler and libraries found in the most recently released eIQ Neutron SDK to get the latest performance improvements and model support.

5              Conclusion

This lab demonstrated how the eIQ Neutron NPU can significantly decrease inference time on quantized models. These same steps can be used to benchmark other quantized models to see the performance improvements that the eIQ Neutron NPU can have.  

---

Updated August 2026 for change of neutron-converter to neutron-compiler in eIQ Neutron SDK 3.2.1 release

Hands-On Training