eIQ Heterogenous Demo

[#1]

This demo shows a low power smart door running eIQ heterogeneously on i.MX8MMini:

• Cortex A53 performs face recognition using eIQ OpenCV
• Cortex M4 performs Key Word Spotting using eIQ CMSIS-NN

The demo application is build around the Django framework running on the board. It has two main usage scenarios:

• The first one is to manage users and inspect the access logs through a dashboard. This dashboard is accesses from a web browser on the host PC.
• The second use case is the smart door application itself running on the board. The scenario is the following: Cortex A cores and connected peripherals stay in low power mode. Cortex M is active, waiting for the Key Word ‘GO’. When the word is detected, Cortex M sends an MU interrupt to Cortex A and the system wakes up. Now Cortex A performs face recognition and allows access for registered users.

In addition to face recognition, the MPUs are able to run a Django server to manage the user’s database, a QT5 application for the graphical interface and perform training on the edge.

The algorithm for face recognition running on Cortex A and the one for key word spotting running on Cortex M are both implemented using eIQ. For the MPU eIQ support is integrated in Yocto. For the MCU the support was ported to i.MX8 from the MCU Expresso SDK for RT for the purpose of this demo.

Software Environment

• Ubuntu 16 host PC
• SD card image with Yocto BSP 4.14.98/sumo 2.0.0 GA for i.MX8MMini platform with eIQ OpenCV AND eIQ heterogenus demo. See detailed steps in Build Yocto Image section
• CMSIS-NN
• MCUXpresso SDK version 2.6.0 for i.MX8MMini (SDK_2.6.0_EVK-MIMX8MM-ARMGCC). See detailed build steps in Build Cortex M4 executable section.

HW Environment

• i.MX 8MMini Kit
• Touch screen display (preferred resolution 1920x1080). Tested with HDMI connection to the board.
  NOTE: if the display does not support touch, a mouse can be connected to the board and used instead
• MIPI-CSI Camera module
• Microphone: Synaptics CONEXANT AudioSmart® DS20921
• Ribbon, 4 female-female wires and 60 pins connector to connect mic to board
• Optional: headphones (used to test recording on M4 - everything recorded by the mic will be played to the headphones).
• Host PC for remote access to the demo application (used Chrome browser)

NOTE: The board and the host PC should be in the same network to communicate.

Build Yocto image:

Step 1 – Project initialization:

$: mkdir imx-linux-bsp
$: cd imx-linux-bsp-bsp
$: repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-sumo -m imx-4.14.98-2.0.0_machinelearning.xml
$: repo sync

Step 2 - Setup Project build:

$: MACHINE=imx8mmevk DISTRO=fsl-imx-xwayland source ./fsl-setup-release.sh -b bld-xwayland

Step 3 – Download project layer in ${BSPDIR}/sources/:

$: git clone https://source.codeaurora.org/external/imxsupport/meta-eiq-heterogenous

Step 4 – Add project layer into bblayers:
Add the following line into ${BSPDIR}/sources/base/conf/bblayers.conf:

BBLAYERS += " ${BSPDIR}/sources/meta-eiq-heterogenous "

Step 5 – Enable eIQ and other dependencies. Add the following lines into conf/local.conf:

EXTRA_IMAGE_FEATURES = " dev-pkgs debug-tweaks tools-debug \
 tools-sdk ssh-server-openssh"

IMAGE_INSTALL_append = " net-tools iputils dhcpcd which gzip \
 python3 python3-pip wget cmake gtest \
 git zlib patchelf nano grep vim tmux \
 swig tar unzip parted \
 e2fsprogs e2fsprogs-resize2fs"

IMAGE_INSTALL_append = " python3-pytz python3-django-cors-headers"

IMAGE_INSTALL_append = " opencv python3-opencv"
PACKAGECONFIG_append_pn-opencv_mx8 = " dnn python3 qt5 jasper \
 openmp test neon"

PACKAGECONFIG_remove_pn-opencv_mx8 = "opencl"

TOOLCHAIN_HOST_TASK_append = " nativesdk-cmake nativesdk-make"

PREFERRED_VERSION_opencv = "4.0.1%"
PREFERRED_VERSION_python3-django = "2.1%"

IMAGE_ROOTFS_EXTRA_SPACE = "20971520"

Step 6 – Bake the image:
$: bitbake image-eiq-hetero

Build Cortex M4 executable

• Download MCUXpresso SDK version 2.6.0 for i.MX8MMini (SDK_2.6.0_EVK-MIMX8MM-ARMGCC)
  • OS: Linux, Toolchain: GCC ARM Embedded
  • Components: Amazon-FreeRTOS, CMSIS DSP Library, multicore
  • SDK Version: 2.6.0 (2019-06-14)
  • SDK Tag: REL_2.6.0_REL10_RFP_RC3_4
• Download CMIS NN and copy "CMSIS\NN" folder to "$MCUXpressoSDK_ROOT\CMSIS"
• Got to "$MCUXpressoSDK_ROOT\boards\evkmimx8mm\demo_apps\"
• Get M4 app from CAF:

  git clone https://source.codeaurora.org/external/imxsupport/eiq-heterogenous-cortexm4

• [Win]: Open ARM GCC console and go to "$MCUXpressoSDK_ROOT\boards\evkmimx8mm\demo_apps\eiq-heterogenous-cortexm4\armgcc\"
• [Win]: Call "build_ddr_release.bat" to obtain "eiq-kws.bin".
• Deploy "eiq-kws.bin" to the Yocto image on the boot partition.

Prepare the Demo

1.  Connect 12V power supply to the board, switch SW101 to power on the board
2.  Connect a USB cable between the host PC and the J901 USB port on the target board.
3.  Open two serial terminals for A53 core and M4 core with the following settings:
    - 115200 baud rate
    - 8 data bits
    - No parity
    - One stop bit
    - No flow control

4. Connect display to the board (used 1920x1080 HDMI display connected to the board with an IMX-MIPI-HDMI adapter).

NOTE: depending on the display, you might want to change the config in "/etc/xdg/weston/weston.ini". The demo was tested by uncommenting the following section in this file:

[output] 
name=HDMI-A-1
mode=1920x1080@60
transform=90

5.Connect MIPI-CSI camera to the board.

6. Connect Synaptics microphone to the board using a 60 pins connector with a ribbon. SAI3 is used for record and playback on Cortex M4. The following pins are used:

• Pin 44 (connector) <-> I2S_TX_Data1 (mic board)
• Pin 43 (connector) <-> I2S_TX_LRCLK (mic board)
• Pin 41 (connector) <-> I2S_TX_CLK (mic board)
• Pin 60 (connector) <-> GND (mic board)

7. Using U-Boot command to run the demo.bin file. For details, please refer to "Getting Started with MCUXpresso SDK for i.MX 8M Mini.pdf".
8.  After running the demo.bin, using the "boot" command to boot the kernel on the A core terminal;
9.  After the kernel is boot, using "root" to login.
10.  After login, make sure imx_rpmsg_pingpong kernel module is inserted (lsmod) or insert it (modprobe imx_rpmsg_pingpong).

Run the Demo

Start Key Word Spotting on Cortex M4:
Stop in u-boot and run the eiq-kws.bin executable in DDR:

u-boot=>fatload mmc 0 0x80000000 eiq-kws.bin
u-boot=>dcache flush
u-boot=>bootaux 0x80000000
u-boot=>boot

After the boot process succeeds, the ARM Cortex-M4 terminal displays the following information:

RPMSG Ping-Pong FreeRTOS RTOS API Demo...
RPMSG Share Base Addr is 0xb8000000

During boot the Kernel,the ARM Cortex-M4 terminal displays the following information:

Link is up!
Nameservice announce sent.

Start Face Recognition on Cortex-A:
Insert updated rpmsg driver:

$: modprobe imx_rpmsg_pingpong

After the Linux RPMsg pingpong module was installed, the ARM Cortex-M4 terminal displays the following information:

Looping forever...
Waiting for ping...
Sending pong...
96% go

First time only:

$: cd ~/eiq-heterogenous-cortexa
$: python3 wrap_migrate.py
$: python3 wrap_createsuperuser.py

Start:

$: cd ~/eiq-heterogenous-cortexa
$: python3 manage.py runserver 0.0.0.0:8000 --noreload &
$: /opt/src/bin/src

NOTE: the first instruction will start the django server, the second instruction will show the pin-pad on the display.

Browser access from HOST PC:
- http://$BOARD_IP:8000/dashboard/ Dashboard that facilitates managing users and view access logs
- http://$BOARD_IP:8000/admin/: manage users database