Blogs

cancel
Showing results for 
Search instead for 
Did you mean: 

Blogs

shaojun_wang
NXP Employee
NXP Employee

Introduction

We already have application note AN13336 about running TFLM micro_speech example on Tensilica HiFi4 DSP on i.MX 8M Plus EVK.
In this document, TFLM micro_speech example will be upgraded to support i.MX DSP remote processors driver in newer Linux kernel, patch to support i.MX8QXP/i.MX8QM MEK will be provided, steps about how to download and install toolchain, how to build and run the example will be listed.

Hardware: i.MX8MP EVK, i.MX8QXP MEK, i.MX8QM MEK.
Software: Linux 5.15.52, TensorFlow Lite Micro

Running TFLM micro_speech demo on i.MX8MP EVK

1 Download Xtensa toolchain

1.1 Download license key  

Login https://tensilicatools.com/download/i-mx-8-hifi-4-sdk/, download GETTING STARTED GUIDE, then follow the GUIDE to get a license key for i.MX8MP SDK.

1.2 Download CONFIGURATION file for Linux

There are two configuration packages:
    DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip
    hifi4_mscale_mpxclib_prod_linux.tgz
The target binary built with xclib is smaller than the one built with newlib.
In this example, we will use NEWLIB package DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip

1.3 Follow the GETTING STARTED GUIDE to download and install XTENSA XPLORER IDE FOR LINUX V8.0.13

After installed, Xtensa tool package can be found at installation folder:
    xtensa/XtDevTools/downloads/RI-2020.4/tools/XtensaTools_RI_2020_4_linux.tgz

2. Install Xtensa development toolchain on Linux OS

2.1 Create imx-audio-toolchain folder

    mkdir -p ./imx-audio-toolchain/Xtensa_Tool/tools
    mkdir -p ./imx-audio-toolchain/Xtensa_Tool/builds

2.2 Set up the configuration-independent Xtensa Tool

    cd imx-audio-toolchain/Xtensa_Tool
    tar zxf XtensaTools_RI_2020_4_linux.tgz -C tools/

2.3 Set up the configuration-specific core files and the Xtensa Tool

Unzip DSP_configuration_for_Linux_(NEWLIB)_i.MX8MP_RI_2020_4.zip, you will get hifi4_mscale_v1_0_2_prod_linux.tgz
    cd imx-audio-toolchain/Xtensa_Tool/
    tar zxf hifi4_mscale_v1_0_2_prod_linux.tgz -C builds/

2.4 Install the Xtensa development toolchain

    cd imx-audio-toolchain/Xtensa_Tool
    ./builds/RI-2020.4-linux/hifi4_mscale_v1_0_2_prod/install --xtensa-tools ./tools/RI-2020.4-linux/XtensaTools --registry ./tools/RI-2020.4-linux/XtensaTools/config

2.5 Set the LM_LICENSE_FILE environment variable

Create ~/.flexlmrc, in the file, set LM_LICENSE_FILE environment variable to the absolute path of your license key folder.
    LM_LICENSE_FILE=<absolute path of your license key folder>

2.6 In imx-audio-toolchain, create environment variables script setup_env.sh for building

#!/bin/bash
export TOOLCHAIN_FOLDER=RI-2020.4-linux
export TOOLCHAIN_PATH=$PWD
export PATH=$TOOLCHAIN_PATH/Xtensa_Tool/tools/$TOOLCHAIN_FOLDER/XtensaTools/bin:$PATH
export XTENSA_CORE=hifi4_mscale_v1_0_2_prod
export XTENSA_BASE=$PWD/Xtensa_Tool/
export XTENSA_TOOLS_VERSION=RI-2020.4-linux
export XTENSA_SYSTEM=$PWD/Xtensa_Tool/tools/RI-2020.4-linux/XtensaTools/config
export PLATF=imx8m

3. Build TFLM micor speech for HiFi4

3.1 Clone the specific TensorFlow repository and check out the proper branch

    git clone https://github.com/pnikam-cad/tensorflow.git
    cd tensorflow
    git checkout hifi5_hifi4_nnlib_new_versions

3.2 Apply patch

    0001-i.MX-8M-Plus-HiFi4-TFLM-Enablement-changes-required-.patch
    0002-resolve-download-error-and-upgrade-toolchain-to-2022.patch
    0003-upgrade-to-support-remoteproc-in-linux-5.10.72.patch

3.3 Compile Micro Speech example

In imx-audio-toolchain, run setup script setup_env.sh
In tensorflow folder, build micro_speech_mock demo with command
    make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa_hifi TARGET_ARCH=hifi4 XTENSA_USE_LIBC=true micro_speech_mock
The compiled binary is tensorflow/lite/micro/tools/make/gen/xtensa_hifi_hifi4/bin/micro_speech_mock

3.4 Run Micro Speech example

Boot i.MX8MP with imx8mp-evk-dsp.dtb, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
    echo start >/sys/class/remoteproc/remoteproc0/state 
    On UART4 output(the fourth terminal in J23 DEBUG port), you can get such log

    shaojun_wang_0-1668152261544.png

3.5 Sample data

If you want to use new sample data, please refer to AN13336 about how to generate and convert.

Running TFLM micro_speech example on i.MX8QXP MEK

4 Download Xtensa toolchain

4.1 Download license key

Login https://tensilicatools.com/download/i-mx8-quadxplus-hifi-4-sdk/, download GETTING STARTED GUIDE, then follow the GUIDE to get a license key for i.MX8QXP SDK.

4.2 Download CONFIGURATION file for Linux

There are two configuration packages:
        hifi4_nxp_v4_3_1_prod_linux.tgz
        hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz
In this example, we will use XCLIB package: hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz

4.3 Follow the GETTING STARTED GUIDE to download and install XTENSA XPLORER IDE FOR LINUX V8.0.13

After installed, Xtensa tool package can be found at installation folder:
    xtensa/XtDevTools/downloads/RI-2020.4/tools/XtensaTools_RI_2020_4_linux.tgz

5. Install Xtensa development toolchain on Linux OS

5.1 Create imx-audio-toolchain folder

    mkdir -p ./imx-audio-toolchain/Xtensa_Tool/tools
    mkdir -p ./imx-audio-toolchain/Xtensa_Tool/builds

5.2 Set up the configuration-independent Xtensa Tool 

    cd imx-audio-toolchain/Xtensa_Tool
    tar zxf XtensaTools_RI_2020_4_linux.tgz -C tools/

5.3 Set up the configuration-specific core files and the Xtensa Tool

    cd imx-audio-toolchain/Xtensa_Tool/
    tar zxf hifi4_nxp_v4_3_1_prod_XC_linux_redist.tgz -C builds/

5.4 Install the Xtensa development toolchain

    cd imx-audio-toolchain/Xtensa_Tool
    ./builds/RI-2020.4-linux/hifi4_nxp_v4_3_1_prod_XC/install --xtensa-tools ./tools/RI-2020.4-linux/XtensaTools --registry ./tools/RI-2020.4-linux/XtensaTools/config

5.5 Set the LM_LICENSE_FILE environment variable

Create ~/.flexlmrc, in the file, set LM_LICENSE_FILE environment variable to the absolute path of your license key folder.
    LM_LICENSE_FILE=<absolute path of your license key folder>

5.6 In imx-audio-toolchain, create environment variables script setup_env.sh for building

#!/bin/bash
export TOOLCHAIN_FOLDER=RI-2020.4-linux
export TOOLCHAIN_PATH=$PWD
export PATH=$TOOLCHAIN_PATH/Xtensa_Tool/tools/$TOOLCHAIN_FOLDER/XtensaTools/bin:$PATH
export XTENSA_CORE=hifi4_nxp_v4_3_1_prod_XC
export XTENSA_BASE=$PWD/Xtensa_Tool/
export XTENSA_TOOLS_VERSION=RI-2020.4-linux
export XTENSA_SYSTEM=$PWD/Xtensa_Tool/tools/RI-2020.4-linux/XtensaTools/config
export PLATF=imx8

6. Build TFLM micor speech for HiFi4

6.1 Clone the specific TensorFlow repository and check out the proper branch

    git clone https://github.com/pnikam-cad/tensorflow.git
    cd tensorflow
    git checkout hifi5_hifi4_nnlib_new_versions

6.2 Apply patch

    0001-i.MX-8M-Plus-HiFi4-TFLM-Enablement-changes-required-.patch
    0002-resolve-download-error-and-upgrade-toolchain-to-2022.patch
    0003-upgrade-to-support-remoteproc-in-linux-5.10.72.patch

6.3 Compile Micro Speech example

In imx-audio-toolchain, run setup script setup_env.sh
In tensorflow folder, build micro_speech_mock demo with command
    make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa_hifi TARGET_ARCH=hifi4 micro_speech_mock
The compiled binary is tensorflow/lite/micro/tools/make/gen/xtensa_hifi_hifi4/bin/micro_speech_mock

6.4 Apply Linux Kernel patch to enable UART2 for HiFi4

    0001-linux-enable-uart2-for-8qxp-hifi4.patch

6.5 Run Micro Speech example

Boot i.MX8QXP MEK board, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
    echo start >/sys/class/remoteproc/remoteproc1/state 
On base board RS232 port J37, you can get such log

    shaojun_wang_1-1668152304575.png

Running TFLM micro_speech example on i.MX8QM MEK

7. Install toolchain and Build TFLM micor speech for HiFi4

Follow above step 4 ~ step 6.3 to download and setup toolchain, apply TFLM patch and compile Micro Speech example.

8. Linux kernel patch and image

In above i.MX8QXP example, micro speech example print log through UART2.

In i.MX8QM MEK Linux demo image imx-image-full-imx8qmmek.wic, M4 is enabled and UART2 is occupied by M4.

To resolve this conflict, we will not enable M4. The steps to run Linux image without M4 are shown below.

1. Apply patch 0001-linux-enable-uart2-for-8qm-hifi4.patch to enable UART2 for HiFi4
2. Build Linux dtb, we will boot i.MX8QM MEK board with imx8qm-mek.dtb.
3. Assume you boot 8QM from SD card, program uboot image without M4 to SD card
    sudo dd if=imx-boot-imx8qmmek-sd.bin-flash of=/dev/sdX

9. Run Micro Speech example

Boot i.MX8QM MEK board with your new imx8qm-mek.dtb, copy micro_speech_mock to /lib/firmware/imx/dsp/, rename it to hifi4.bin, then load and run the example with command
    echo start >/sys/class/remoteproc/remoteproc2/state 
On base board RS232 port J37, you can get such log

shaojun_wang_0-1668763632591.png

10. Troubleshooting

10.1 No remoteproc device in /sys/class/remoteproc/

Run lsmod, check if imx_dsp_rproc module is loaded. If not, rebuild Linux kernel and modular and update to your board.

10.2 No output on HiFi4 UART

  • In 8MP, check if you bootup the board with imx8mp-evk-dsp.dtb
  • There may be several remoteproc devices in /sys/class/remoteproc, check which one is imx-dsp-rproc by below command, then "echo start" to that device.
        cat /sys/class/remoteproc/remoteproc1/name 

10.3 In 8QXP MEK, Get "remoteproc remoteproc1: can't start rproc imx-dsp-rproc: -110" error when run the "echo start" command

Check if you have applied Linux patch to enable UART2 for HiFi4, and bootup your board with the dtb file.

11 Reference

https://www.nxp.com/webapp/Download?colCode=AN13336

XPLOR8QM8QP-UG_Rev0.pdf in https://tensilicatools.com/download/i-mx8-quadxplus-hifi-4-sdk/

More
2 0 183
kenli
NXP Employee
NXP Employee

Security is becoming more and more important in automotive products and related designs. The attachment shows how to perform simple verification of the secure boot process based on LSDK2108 and LX2160ARDB.

Read more...

More
0 0 165