This article will describe how to use the HiFi modules found on certain NXP microcontrollers as an optional method for inferencing a model.
There are several ways of running a TFLite neural network model on NXP microcontrollers:
- Inference a model only using the main core of the device (CM33 or M7)
- MCX N
- i.MX RT1050
- i.MX RT1060
- i.MX RT1170
- i.MX RT1180
- i.MX RT595
- i.MX RT685
- i.MX RT700
- Inference a model directly on the HiFi4 or HiFi1 core (on supported platforms)
- i.MX RT595
- i.MX RT685
- i.MX RT700
- Use the Neutron NPU to accelerate inference of a model - with the CM33 controlling the NPU and acting as a fallback for any non-NPU supported layers
- i.MX RT700
- MCX N
- Use the Neutron NPU to accelerate inference of a model - with the HiFi4 controlling the NPU and acting as a fallback for any non-NPU supported layers
- i.MX RT700
This article will cover options #2 and #4 which make use of the HiFi DSP module. Running a model on the HiFi4 (option #2) will be much faster than running a model just on the CM33/M7 (option #1).
However using the NPU (options #3 and #4) will be significantly faster than only using the DSP due to the hardware optimizations that an NPU provides for neural network calculations. The exact performance gains will be model specific, and also depend on the layer(s) that may not have been converted to use the NPU as NeutronGraph nodes. The accuracy should remain the very similar regardless of method being used. Any type of TFLite neural network model can be ran on the HiFi1/HiFi4 as those DSP modules are just being used accelerate the neural network math that the model uses.
CIFAR10 on i.MX RT700 using default MCUXpresso SDK projects:
- CM33: 105.925ms
- HiFi1*: 148.487ms
- HiFi4: 12.312ms
- NPU w/ CM33: 1.048ms
- NPU w/ HiFi4: 0.983ms
*HiFi1 runs at 32MHz
Software requirements:
Go to the Cadence i.MX RT700 or Cadence i.MX RT685 pages to download the following software.
- Xtensa Xplorer IDE
- License Key
- HIFI DSP Configuration File (NEWLIB)
If using a HiFi4 example then download the HiFi4 license and DSP configuration files. Likewise, if using a HiFi1, then will need the HiFi1 license and DSP configuration files. Also you will need to download the Windows or Linux version of these files depending on which host OS you are using on your PC.
Finally add the following global system variables which should be set based on the location that Xtensa Explorer was installed (assuming RT700 with HiFi4):
- XCC_DIR=
\XtDevTools\install\tools\RI-2023.11-win32\XtensaTools - XTENSA_CORE= rt700_hifi4_RI23_11_nlib
MCUXpresso SDK HiFi ML Examples:
There are several HiFi related examples in MCUXpresso SDK for i.MX RT700:
- tflm_cifar10 – Uses Neutron NPU to inference the CIFAR10 model and uses the CM33 as the fallback for any non-NPU operators.
- tflm_cifar10_hifi1 – Uses HiFi1 to inference the CIFAR10 model. Does not use the Neutron NPU
- tflm_cifar10_hifi4 – Uses HiFi4 to inference the CIFAR10 model. Does not use the Neutron NPU
- tflm_cifar10_hifi4_neutron – Uses Neutron NPU to inference the CIFAR10 model and uses the HiFi4 as the fallback for any non-NPU operators.
- tflm_label_image – Uses Neutron NPU to inference the CIFAR10 model and uses the CM33 as the fallback for any non-NPU operators.
- tflm_label_image_hifi4 - Uses HiFi4 to inference the Mobilenet model. Does not use the Neutron NPU
When using the HiFi eIQ projects provided in MCUXpresso SDK, ensure that the SDK is:
- Located in a short filename path (ie C:\nxp\RT700), as an excessively long filename path can cause compile issues
- Directory path contains no spaces
- If using VS Code import as a Repository project instead of Free Standing.
MCUXpresso SDK 26.03 Known Issue:
In MCUXpresso SDK 26.03 an additional compile option named -mlongcalls needs to be used to avoid issues when adding additional operators the ops list.
For VS Code, inside
\mcuxsdk\examples\_boards\mimxrt700evk\eiq_examples\tflm_cifar10_hifi4\hifi4 edit the reconfig.cmake file in the mcux_add_xtensa_configuration options:
mcux_add_xtensa_configuration(
CC "-DXOS_CLOCK_FREQ=237500000 -std=c99 -mlongcalls"
CX "-stdlib=libc++ \
-mlongcalls \
-std=c++17"
)
If using Xtensa Explorer:
Then in the Addl compiler tab, under Additional options, add -mlongcalls and click OK to save.
HiFi4 Neutron Example:
The HiFi4 Neutron example may not execute in MCUXpresso SDK 26.03. Please either upgrade the Neutron libraries to the latest version, or else use the default HiFi4 Neutron example in MCUXpresso SDK 25.12 in the meantime. This will be fixed in MCUXpresso SDK 26.06.
HiFi Lab:
See the attached lab document for more details on using the HiFi DSP modules to inference models.