This article will describe how to use the HiFi modules found on certain NXP microcontrollers as an optional method for inferencing a model.
There are several ways of running a TFLite neural network model on NXP microcontrollers:
- Inference a model only using the main core of the device (CM33 or M7)
- MCX N
- i.MX RT1050
- i.MX RT1060
- i.MX RT1170
- i.MX RT1180
- i.MX RT595
- i.MX RT685
- i.MX RT700
- Inference a model directly on the HiFi4 or HiFi1 core (on supported platforms)
- i.MX RT595
- i.MX RT685
- i.MX RT700
- Use the Neutron NPU to accelerate inference of a model - with the CM33 controlling the NPU and acting as a fallback for any non-NPU supported layers
- i.MX RT700
- MCX N
- Use the Neutron NPU to accelerate inference of a model - with the HiFi4 controlling the NPU and acting as a fallback for any non-NPU supported layers
- i.MX RT700
This article will cover options #2 and #4 which make use of the HiFi DSP module. Running a model on the HiFi4 (option #2) will be much faster than running a model just on the CM33/M7 (option #1).
However using the NPU (options #3 and #4) will be significantly faster than only using the DSP due to the hardware optimizations that an NPU provides for neural network calculations. The exact performance gains will be model specific, and also depend on the layer(s) that may not have been converted to use the NPU as NeutronGraph nodes. The accuracy should remain the very similar regardless of method being used. Any type of TFLite neural network model can be ran on the HiFi1/HiFi4 as those DSP modules are just being used accelerate the neural network math that the model uses.
CIFAR10 on i.MX RT700 using default MCUXpresso SDK projects:
- CM33: 105.925ms
- HiFi1*: 148.487ms
- HiFi4: 12.312ms
- NPU w/ CM33 Fallback: 1.048ms
- NPU w/ HiFi4 Fallback: 0.983ms
*HiFi1 runs at 32MHz
Software requirements:
Go to the Cadence i.MX RT700 or Cadence i.MX RT685 pages to download the following software.
- Xtensa Xplorer IDE
- License Key
- HIFI DSP Configuration File (NEWLIB)
If using a HiFi4 example then download the HiFi4 license and DSP configuration files. Likewise, if using a HiFi1, then will need the HiFi1 license and DSP configuration files. Also you will need to download the Windows or Linux version of these files depending on which host OS you are using on your PC.
Finally add the following global system variables which should be set based on the location that Xtensa Explorer was installed (assuming RT700 with HiFi4):
- XCC_DIR=
\XtDevTools\install\tools\RI-2023.11-win32\XtensaTools - XTENSA_CORE= rt700_hifi4_RI23_11_nlib
MCUXpresso SDK HiFi ML Examples:
There are several HiFi related examples in MCUXpresso SDK for i.MX RT700:
- tflm_cifar10 – Uses Neutron NPU to inference the CIFAR10 model and uses the CM33 as the fallback for any non-NPU operators.
- tflm_cifar10_hifi1 – Uses HiFi1 to inference the CIFAR10 model. Does not use the Neutron NPU
- tflm_cifar10_hifi4 – Uses HiFi4 to inference the CIFAR10 model. Does not use the Neutron NPU
- tflm_cifar10_hifi4_neutron – Uses Neutron NPU to inference the CIFAR10 model and uses the HiFi4 as the fallback for any non-NPU operators.
- tflm_label_image – Uses Neutron NPU to inference the CIFAR10 model and uses the CM33 as the fallback for any non-NPU operators.
- tflm_label_image_hifi4 - Uses HiFi4 to inference the Mobilenet model. Does not use the Neutron NPU
When using the HiFi eIQ projects provided in MCUXpresso SDK, ensure that the SDK is:
- Located in a short filename path (ie C:\nxp\RT700), as an excessively long filename path can cause compile issues
- Directory path contains no spaces
- If using VS Code import as a Repository project instead of Free Standing.
- Ensure using at least MCUXPresso SDK 26.03 as there are several important fixes in the 26.03 release for HiFi4 projects
Extending Memory Area in HiFi4 Examples for Larger Models:
Open:
\boards\mimxrt700evk\eiq_examples\tflm_cifar10_hifi4_neutron\linker\hifi4\min-rt\ldscripts\elf32xtensa.x
And make modifications below to extend the memory used for models.
dsp_core_seg : org = 20200000, len = 0x200000
dsp_core_ncache_seg : org = 0x20400000, len = 0x180000
anthony_huereca_9-1783320725255.png
_memmap_mem_dsp_core_start = 0x20200000;
_memmap_mem_dsp_core_end = 0x20580000;
_memmap_seg_dsp_core_start = 0x20200000;
_memmap_seg_dsp_core_max = 0x20580000;
anthony_huereca_10-1783320808490.png
Ensure that the stack heap is at 0x2058_0000:
PROVIDE(__stack = 0x20580000);
_heap_sentry = 0x20580000;
anthony_huereca_11-1783320872686.png
Now larger models will compile successfully in Xtensa.
Then after copying in the compiled HiFi4 binary files into the MCUXpresso IDE project, modify source/dsp_config.h file to update the DSP_SRAM_ADDRESS:
anthony_huereca_12-1783321027565.png
Then clean and build the project.
HiFi Lab:
See the attached lab document for more details on using the HiFi DSP modules to inference models.