topic Re: Using GPU on i.MX8qmek for DNN infernece in eIQ Machine Learning Software

Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Sun, 14 Jun 2020 08:59:54 GMT

I am currently trying to evaluate different inference engines with TensorFlow and TensorFlow Lite models on i.MX8 QMEK. I follow the eIQ guide form NXP and using L4.14 Release.

I tried with OpenCV DNN module, TFLite Interpreter and Arm NN. I was not able to use GPU with any of them. I know OpenCV does not run on GPU due to OpenCL compatibility issue on i.MX8 but can I not also use GPU with TFLite and Arm NN?

On other hand, the Arm NN examples with e.IQ does not provide an option to use GPU at all.

In this thread, Arm NN support for the i.MX 8 GPUs, vanessamaegima suggested only TFLite engine supports GPU at the moment.

So it is all confusing to me whether GPU is of any use on imx8qmek for running DNN inference.

So if there is a way to use GPU, kindly let me know. This has me bogging my head from quite a while now.

Best Regards

Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Tue, 16 Jun 2020 17:27:33 GMT

Hi ullasbharadwaj‌,

For the L4.14 releases we do not have GPU support for any of the inference engines.

Starting with L4.19 releases eIQ is released as part of the official NXP BSP and we do have GPU implementations for TfLite and ArmNN. I suggest you to try our latest 5.4.3_1.0.0 release which provides our latest TfLite/ArmNN GPU acceleration. Please note that we do not support OpenCV DNN GPU acceleration.

Please refer to the BSP documentation for more details on the eIQ support for each release after L4.19.

Thanks,
Vanessa

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Wed, 17 Jun 2020 12:05:37 GMT

Hi Vanessa,
Thanks for your response. I tried example code for image classification (label_image) in the L4.19 BSP to start with.

1. I was able to run the model on GPU but when I tried my custom SSD mobilenet model (which was exported with post_processing = true during graph generation), I got an error "op code 6 not supoorted with NNAPI).

2. Somehow, tensorflow lite library and Qt5 libraries were not exported into the SDK, i have trouble compiling applications.
Error 1: undefined reference to `tflite::Interpreter::ModifyGraphWithDelegate(....) . I used the library from the sysroot, so I am using the proper tensorflow-lite library version.
Error 2:

CMake Warning at SDKs/yocto/sysroots/aarch64-poky-linux/usr/lib/cmake/Qt5Core/Qt5CoreConfig.cmake
SkippingbecauseOE_QMAKE_PATH_EXTERNAL_HOST_BINSisnotdefined

Local.conf file:

MACHINE ??= 'imx8qmmek'
DISTRO ?= 'fsl-imx-xwayland'
PACKAGE_CLASSES ?= 'package_rpm'
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
IMAGE_INSTALL_append = " ffmpeg"
PACKAGECONFIG_append_pn-opencv_mx8 = " dnn python3 qt5 jasper test"
IMAGE_INSTALL_append = " gputop"
COMMERCIAL_LICENSE ?= "lame gst-fluendo-mp3 libmad mpeg2dec ffmpeg qmmp"
LICENSE_FLAGS_WHITELIST = 'commercial'

BB_DISKMON_DIRS ??= "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
STOPTASKS,/tmp,100M,100K \
ABORT,${TMPDIR},100M,1K \
ABORT,${DL_DIR},100M,1K \
ABORT,${SSTATE_DIR},100M,1K \
ABORT,/tmp,10M,1K"

PACKAGECONFIG_append_pn-qemu-system-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl"
CONF_VERSION = "1"
DL_DIR ?= "${BSPDIR}/downloads/"
ACCEPT_FSL_EULA = "1"

Also, wrt TFLite delgate registration and custom op supported, is there an example code for object detection to showcase how to work with gpu on i.MX8?

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Wed, 17 Jun 2020 13:43:32 GMT

Hi ullasbharadwaj,

1. The label_image TfLite example does not support SSD models. Also GPU models should be quantized (uint8 models). Please take a look at PyeIQ for examples on how to use SSD models.

2. Could you please try the following to build Yocto SDK:

IMAGE_INSTALL_append = "packagegroup-imx-ml"
IMAGE_INSTALL_append = "packagegroup-imx-qt5"

For GPU examples, please take a look at PyeIQ.

Thanks,
Vanessa

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Wed, 17 Jun 2020 17:24:23 GMT

Hi Vanessa,
Thanks and I tried your suggessions.
IMAGE_INSTALL_append = "packagegroup-imx-qt5" gave me error " ERROR: Nothing RPROVIDES 'packagegroup-imx-qt5' ".

IMAGE_INSTALL_append = "packagegroup-imx-ml" is added to local.conf.

Still no libtensorflow-lite.a not included ito the SDK sysroots/usr/lib/. This issue was not there with L4.14.

And the compilation of TFLite app in C++ gives me error, undefined reference to `tflite::Interpreter::ModifyGraphWithDelegate.

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Wed, 17 Jun 2020 18:07:01 GMT

Hi Ullas,

Sorry, please try the following: packagegroup-qt5-imx

For your compilation issue, please try including needed directories in your Makefile/make command with -I instead of using the static library. This issue seems to be fixed in next releases.

Thanks,

Vanessa

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Thu, 18 Jun 2020 13:09:09 GMT

Hi Vanessa,

I tried PyeIQ with 5.4.3 BSP. There are so many dependencies missing, for example "requests" which I installed with whl file manually as there is no internet connection to the board. Now I am facing import errors with PIL.

Do you know if there is a config to include all these or am I missing anything wrt eIQ installation?

And, in the refrence manual for 5.4.3, it is mentions "Arm NN does
not currently support the i.MX 8 GPUs due to Arm NN OpenCL requirements which are
not met by i.MX8 GPUs.". But you mentioned above that it does support GPU. So it is confusing a bit.

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Thu, 18 Jun 2020 15:01:48 GMT

Hi ullasbharadwaj‌,

And, in the refrence manual for 5.4.3, it is mentions "Arm NN does
not currently support the i.MX 8 GPUs due to Arm NN OpenCL requirements which are
not met by i.MX8 GPUs.". But you mentioned above that it does support GPU. So it is confusing a bit.

Please check Arm Compute Library chapter in Linux RM. Arm NN uses ACL delegates for GPU acceleration.

diegodorta‌, could you please check the PyeIQ queries?

Thanks,
Vanessa

Re: Using GPU on i.MX8qmek for DNN infernece

diego_dorta — Thu, 18 Jun 2020 15:26:44 GMT

Hi Ullas,

Which BSP version are you using it? 5.4.3_1.0.0? If so, it will not work properly. You must build an image using the 5.4.3_2.0.0 version which is required by PyeIQ.

About the internet connection, you must connect your board because PyeIQ requires a connection to retrieve what it needs to work.

Thanks,

Diego

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Thu, 18 Jun 2020 15:49:58 GMT

Hi Diego,

Yes it was 5.4.3_1.0.0 !
I will start building 5.4.3_2.0.0 it and come back to you. Thanks for the lead.

Do you know if there is an alternative to get all required python packages on PC and then take it to the board?
Because I am using laptop connected via WiFi and trying to get internet on the board via ethernet which was not possible.

vanessamaegima‌ Thanks for the clarification. Is there example source code for SSD MobileNet object detection in C++ for Arm NN from NXP?

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Thu, 18 Jun 2020 16:00:32 GMT

For ArmNN C++ examples you can refer to the ArmNN repository, for example: armnn/TfLiteMobileNetSsd-Armnn.cpp at branches/armnn_20_05 · ARM-software/armnn · GitHub

Please check the BSP User Guide for more information on the examples.

Re: Using GPU on i.MX8qmek for DNN infernece

diego_dorta — Thu, 18 Jun 2020 16:10:48 GMT

Hi Ullas,

PyeIQ does not retrieve any Python package, it uses exactly what is supported on 5.4.3_2.0.0. What it does is to retrieve Machine Learning models, labels or any related data to run the applications or demos.

Unfortunately, that was how PyeIQ was designed and you will need at least for now (I will check the possibility to include a way to get the models on host machine) connect it to the internet.

The second version will be released soon which plenty of more samples. Check in a few days on the eIQ community :smileyhappy:

Thanks,

Diego

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Thu, 18 Jun 2020 16:20:03 GMT

Thanks for the reply.

Then maybe because of 5.4.3_1.0.0 there were python packages were missing and hence I misunderstood that the PyeIQ will downoad them also.

I will try with 5.4.3_2.0.0 and get back to you :-)

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Fri, 19 Jun 2020 06:23:15 GMT

Hi,
I am trying to build 5.4.3_2.0.0 on Ubuntu 18.04. I am facing this error while building from "optee" package. No way build is getting past this even by removing this package in local.conf. Can you please help?

ERROR: optee-os-3.7.0.imx-r0 do_compile: oe_runmake failed
ERROR: optee-os-3.7.0.imx-r0 do_compile: Execution of '/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/temp/run.do_compile.18914' failed with exit code 1:
make: Entering directory '/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/git'
GEN /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/core/ta_pub_key.c
CHK /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/conf.mk
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/memory_buffer_alloc.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/platform_time.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/camellia.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/config.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/rsa.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/havege.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/nist_kw.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/chachapoly.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/asn1write.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/entropy.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/ssl_cache.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/version.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/sha512.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/blowfish.h
Traceback (most recent call last):
File "scripts/pem_to_pub_c.py", line 61, in <module>
main()
File "scripts/pem_to_pub_c.py", line 24, in main
from Crypto.PublicKey import RSA
File "/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/Crypto/PublicKey/RSA.py", line 585
except ValueError, IndexError:
^
SyntaxError: invalid syntax
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/ecdsa.h
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/debug.h
mk/subdir.mk:159: recipe for target '/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/core/ta_pub_key.c' failed
make: *** [/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/core/ta_pub_key.c] Error 1
make: *** Waiting for unfinished jobs....
INSTALL /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/build.mx8qmmek/export-ta_arm64/include/mbedtls/compat-1.3.h
make: Leaving directory '/home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/git'
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/bharadwaj/Desktop/yocto_imx8/imx-yocto-bsp/build_5_4_3_2_0/tmp/work/imx8qmmek-poky-linux/optee-os/3.7.0.imx-r0/temp/log.do_compile.18914

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Fri, 19 Jun 2020 12:29:09 GMT

Hi Ullas,

When you migrated to 5.4.3_2.0.0 from 5.4.3_1.0.0, did you used the same build directory or did you create a new one for the new build?

Could you please try:

bitbake -c cleanall optee-os
bitbake imx-image-full

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Fri, 19 Jun 2020 12:38:59 GMT

Hi Vanessa,
I created a new directory and I also tried the clean and rebuild as you mentioned but it does not seem to solve.
I have just added these lines below in the conf/local.conf. (I used these in earlier BSP versions)

PACKAGECONFIG_append_pn-opencv_mx8 = " dnn python3 qt5 jasper test"
CORE_IMAGE_EXTRA_INSTALL += " python3 python3-pip"
IMAGE_INSTALL_append = " ffmpeg gputop"
TOOLCHAIN_TARGET_TASK += "tensorflow-lite-dev tensorflow-lite-staticdev"

COMMERCIAL_LICENSE ?= "lame gst-fluendo-mp3 libmad mpeg2dec ffmpeg qmmp"
LICENSE_FLAGS_WHITELIST = 'commercial'

Are some of these causing some conflicts?

Sorry for bothering you too much :-(

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Fri, 19 Jun 2020 17:46:10 GMT

Hi Ullas,

Unfortunately the L5.4.3_2.0.0 release is an Alpha release. Before we try to debug this issue can you please confirm if you had pip3 (python3-pip) installed on your L5.4.3_1.0.0 image when you tried PyeIQ on this release?

We discussed internally and PyeIQ should work with L5.4.3_1.0.0 if you have pip3 installed on the board.

Thanks,
Vanessa

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Fri, 19 Jun 2020 17:53:22 GMT

Hi Vanessa,

Yes pip3 was available. pip3 install eiq.tar.gz gave some import errors.

1. There was "requests" package missing, so I installed it via downloading whl file and installing via pip3 install.

2. Then, it gave errors regarding PIL pacakge when importing it. So I tried to install it also via whl file but then it gave errors regarding zlib libraries missing.

Also, I tried without altering the local.conf as above, some how optee-os is now getting built. This was strange and I didn't expect it.

Best Regards,
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

ullasbharadwaj — Sun, 21 Jun 2020 16:24:07 GMT

Hi Vanessa and Diego,

FInally everything seems to be solved with 5.4.3_2.0.0. I was able to use GPU with TFLite Interpreter using my custom C++ application and also run PyeIQ samples. I would also investigate into ArmNN further.
Thanks for all your leads :-)

However, I am facing problem with OpenCV. Opening mp4 video file for processing is very slow and also imshow() produces improper video output. It seems some problem with decoding I guess. However same file with gplay-1.0 plays perfecty fine at arounf 60 fps also.
I have attached the OpenCV config and also the image output sample.

OpenCV uses gstreamer backend I believe and I get same log messages as in the attached screenshot both with gplay as well as videocapture.open( ). I am not sure what the problem is.

Can you please help?

Also, by default logging is made active for TF lite and opencv... I disable for tflite using VSI_NN_LOG_LEVEL, similarly how can i do it for opencv DNN module?

Best Regards
Ullas Bharadwaj

Re: Using GPU on i.MX8qmek for DNN infernece

vanessa_maegima — Mon, 22 Jun 2020 12:24:57 GMT

Hi ullasbharadwaj‌,
Can you please share the logs you want to disable from OpenCV DNN?

marcofranchi‌, is there anything you can share with Ullas regarding the response above on the video processing?

Thanks,
Vanessa