Introduction
We already have application note AN13336 about running TFLM micro_speech example on Tensilica HiFi4 DSP on i.MX 8M Plus EVK.
In this document, TFLM micro_speech example will be upgraded to support i.MX DSP remote processors driver in newer Linux kernel, patch to support i.MX8QXP/i.MX8QM MEK will be provided, steps about how to download and install toolchain, how to build and run the example will be listed.
Hardware: i.MX8MP EVK, i.MX8QXP MEK, i.MX8QM MEK.
Software: Linux 5.15.52, TensorFlow Lite Micro
Running TFLM micro_speech demo on i.MX8MP EVK
1 Download Xtensa toolchain
1.1 Download license key
Login https://tensilicatools.com/download/i-mx-8-hifi-4-sdk/, download GETTING STARTED GUIDE, then follow the GUIDE to get a license key for i.MX8MP SDK.
1.2 Download CONFIGURATION file for Linux
There are two configuration packages:
DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip
hifi4_mscale_mpxclib_prod_linux.tgz
The target binary built with xclib is smaller than the one built with newlib.
In this example, we will use NEWLIB package DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip
1.3 Follow the GETTING STARTED GUIDE to download and install XTENSA XPLORER IDE FOR LINUX V8.0.13
After installed, Xtensa tool package can be found at installation folder:
xtensa/XtDevTools/downloads/RI-2020.4/tools/XtensaTools_RI_2020_4_linux.tgz
2. Install Xtensa development toolchain on Linux OS
2.1 Create imx-audio-toolchain folder
mkdir -p ./imx-audio-toolchain/Xtensa_Tool/tools
mkdir -p ./imx-audio-toolchain/Xtensa_Tool/builds
2.2 Set up the configuration-independent Xtensa Tool
cd imx-audio-toolchain/Xtensa_Tool
tar zxf XtensaTools_RI_2020_4_linux.tgz -C tools/
2.3 Set up the configuration-specific core files and the Xtensa Tool
Unzip DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip, you will get hifi4_mscale_v1_0_2_prod_linux.tgz
cd imx-audio-toolchain/Xtensa_Tool/
tar zxf hifi4_mscale_v1_0_2_prod_linux.tgz -C builds/
2.4 Install the Xtensa development toolchain
cd imx-audio-toolchain/Xtensa_Tool
./builds/RI-2020.4-linux/hifi4_mscale_v1_0_2_prod/install --xtensa-tools ./tools/RI-2020.4-linux/XtensaTools --registry ./tools/RI-2020.4-linux/XtensaTools/config
2.5 Set the LM_LICENSE_FILE environment variable
Create ~/.flexlmrc, in the file, set LM_LICENSE_FILE environment variable to the absolute path of your license key folder.
LM_LICENSE_FILE=<absolute path of your license key folder>
2.6 In imx-audio-toolchain, create environment variables script setup_env.sh for building
#!/bin/bash
export TOOLCHAIN_FOLDER=RI-2020.4-linux
export TOOLCHAIN_PATH=$PWD
export PATH=$TOOLCHAIN_PATH/Xtensa_Tool/tools/$TOOLCHAIN_FOLDER/XtensaTools/bin:$PATH
export XTENSA_CORE=hifi4_mscale_v1_0_2_prod
export XTENSA_BASE=$PWD/Xtensa_Tool/
export XTENSA_TOOLS_VERSION=RI-2020.4-linux
export XTENSA_SYSTEM=$PWD/Xtensa_Tool/tools/RI-2020.4-linux/XtensaTools/config
export PLATF=imx8m
3. Build TFLM micor speech for HiFi4
3.1 Clone the specific TensorFlow repository and check out the proper branch
git clone https://github.com/pnikam-cad/tensorflow.git
cd tensorflow
git checkout hifi5_hifi4_nnlib_new_versions
3.2 Apply patch
0001-i.MX-8M-Plus-HiFi4-TFLM-Enablement-changes-required-.patch
0002-resolve-download-error-and-upgrade-toolchain-to-2022.patch
0003-upgrade-to-support-remoteproc-in-linux-5.10.72.patch
3.3 Compile Micro Speech example
In imx-audio-toolchain, run setup script setup_env.sh
In tensorflow folder, build micro_speech_mock demo with command
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa_hifi TARGET_ARCH=hifi4 XTENSA_USE_LIBC=true micro_speech_mock
The compiled binary is tensorflow/lite/micro/tools/make/gen/xtensa_hifi_hifi4/bin/micro_speech_mock
3.4 Run Micro Speech example
Boot i.MX8MP with imx8mp-evk-dsp.dtb, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
echo start >/sys/class/remoteproc/remoteproc0/state
On UART4 output(the fourth terminal in J23 DEBUG port), you can get such log
3.5 Sample data
If you want to use new sample data, please refer to AN13336 about how to generate and convert.
Running TFLM micro_speech example on i.MX8QXP MEK
4 Download Xtensa toolchain
4.1 Download license key
Login https://tensilicatools.com/download/i-mx8-quadxplus-hifi-4-sdk/, download GETTING STARTED GUIDE, then follow the GUIDE to get a license key for i.MX8QXP SDK.
4.2 Download CONFIGURATION file for Linux
There are two configuration packages:
hifi4_nxp_v4_3_1_prod_linux.tgz
hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz
In this example, we will use XCLIB package: hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz
4.3 Follow the GETTING STARTED GUIDE to download and install XTENSA XPLORER IDE FOR LINUX V8.0.13
After installed, Xtensa tool package can be found at installation folder:
xtensa/XtDevTools/downloads/RI-2020.4/tools/XtensaTools_RI_2020_4_linux.tgz
5. Install Xtensa development toolchain on Linux OS
5.1 Create imx-audio-toolchain folder
mkdir -p ./imx-audio-toolchain/Xtensa_Tool/tools
mkdir -p ./imx-audio-toolchain/Xtensa_Tool/builds
5.2 Set up the configuration-independent Xtensa Tool
cd imx-audio-toolchain/Xtensa_Tool
tar zxf XtensaTools_RI_2020_4_linux.tgz -C tools/
5.3 Set up the configuration-specific core files and the Xtensa Tool
cd imx-audio-toolchain/Xtensa_Tool/
tar zxf hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz -C builds/
5.4 Install the Xtensa development toolchain
cd imx-audio-toolchain/Xtensa_Tool
./builds/RI-2020.4-linux/hifi4_nxp_v4_3_1_prod_XC/install --xtensa-tools ./tools/RI-2020.4-linux/XtensaTools --registry ./tools/RI-2020.4-linux/XtensaTools/config
5.5 Set the LM_LICENSE_FILE environment variable
Create ~/.flexlmrc, in the file, set LM_LICENSE_FILE environment variable to the absolute path of your license key folder.
LM_LICENSE_FILE=<absolute path of your license key folder>
5.6 In imx-audio-toolchain, create environment variables script setup_env.sh for building
#!/bin/bash
export TOOLCHAIN_FOLDER=RI-2020.4-linux
export TOOLCHAIN_PATH=$PWD
export PATH=$TOOLCHAIN_PATH/Xtensa_Tool/tools/$TOOLCHAIN_FOLDER/XtensaTools/bin:$PATH
export XTENSA_CORE=hifi4_nxp_v4_3_1_prod_XC
export XTENSA_BASE=$PWD/Xtensa_Tool/
export XTENSA_TOOLS_VERSION=RI-2020.4-linux
export XTENSA_SYSTEM=$PWD/Xtensa_Tool/tools/RI-2020.4-linux/XtensaTools/config
export PLATF=imx8
6. Build TFLM micor speech for HiFi4
6.1 Clone the specific TensorFlow repository and check out the proper branch
git clone https://github.com/pnikam-cad/tensorflow.git
cd tensorflow
git checkout hifi5_hifi4_nnlib_new_versions
6.2 Apply patch
0001-i.MX-8M-Plus-HiFi4-TFLM-Enablement-changes-required-.patch
0002-resolve-download-error-and-upgrade-toolchain-to-2022.patch
0003-upgrade-to-support-remoteproc-in-linux-5.10.72.patch
6.3 Compile Micro Speech example
In imx-audio-toolchain, run setup script setup_env.sh
In tensorflow folder, build micro_speech_mock demo with command
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa_hifi TARGET_ARCH=hifi4 micro_speech_mock
The compiled binary is tensorflow/lite/micro/tools/make/gen/xtensa_hifi_hifi4/bin/micro_speech_mock
6.4 Apply Linux Kernel patch to enable UART2 for HiFi4
0001-linux-enable-uart2-for-8qxp-hifi4.patch
6.5 Run Micro Speech example
Boot i.MX8QXP MEK board, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
echo start >/sys/class/remoteproc/remoteproc1/state
On base board RS232 port J37, you can get such log
Running TFLM micro_speech example on i.MX8QM MEK
7. Install toolchain and Build TFLM micor speech for HiFi4
Follow above step 4 ~ step 6.3 to download and setup toolchain, apply TFLM patch and compile Micro Speech example.
8. Linux kernel patch and image
In above i.MX8QXP example, micro speech example print log through UART2.
In i.MX8QM MEK Linux demo image imx-image-full-imx8qmmek.wic, M4 is enabled and UART2 is occupied by M4.
To resolve this conflict, we will not enable M4. The steps to run Linux image without M4 are shown below.
1. Apply patch 0001-linux-enable-uart2-for-8qm-hifi4.patch to enable UART2 for HiFi4
2. Build Linux dtb, we will boot i.MX8QM MEK board with imx8qm-mek.dtb.
3. Assume you boot 8QM from SD card, program uboot image without M4 to SD card
sudo dd if=imx-boot-imx8qmmek-sd.bin-flash of=/dev/sdX
9. Run Micro Speech example
Boot i.MX8QM MEK board with your new imx8qm-mek.dtb, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
echo start >/sys/class/remoteproc/remoteproc2/state
On base board RS232 port J37, you can get such log
10. Troubleshooting
10.1 No remoteproc device in /sys/class/remoteproc/
Run lsmod, check if imx_dsp_rproc module is loaded. If not, rebuild Linux kernel and modular and update to your board.
10.2 No output on HiFi4 UART
- In 8MP, check if you bootup the board with imx8mp-evk-dsp.dtb
- There may be several remoteproc devices in /sys/class/remoteproc, check which one is imx-dsp-rproc by below command, then "echo start" to that device.
cat /sys/class/remoteproc/remoteproc1/name
10.3 In 8QXP MEK, Get "remoteproc remoteproc1: can't start rproc imx-dsp-rproc: -110" error when run the "echo start" command
Check if you have applied Linux patch to enable UART2 for HiFi4, and bootup your board with the dtb file.
11 Reference
https://www.nxp.com/webapp/Download?colCode=AN13336
XPLOR8QM8QP-UG_Rev0.pdf in https://tensilicatools.com/download/i-mx8-quadxplus-hifi-4-sdk/