[Gatesgarth 5.10] TFLite static library issue

[danmartzla]

Hi there,

I've just successfully built the image and the SDK of the new imx-linux version (gatesgarth) and I'm having an issue compiling Tensorflow-Lite c++ apps within the new SDK environment.

Here is the basic Yocto config I used to build the "imx-image-full" image and its SDK in a docker container:

Build Configuration:
BB_VERSION           = "1.48.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "imx8mpevk"
DISTRO               = "fsl-imx-wayland"
DISTRO_VERSION       = "5.10-gatesgarth"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa53 crypto"

Once I try to compile a custom Tensorflow lite application within the SDK, lots of linker errors show up such as this one:

/home/user/gatesgarth_sdk/sysroots/x86_64-pokysdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/10.2.0/real-ld: /home/user/gatesgarth_sdk/sysroots/cortexa53-crypto-poky-linux/usr/lib/libtensorflow-lite.a(transpose_conv.cc.o): in function `ruy::Kernel<(ruy::Path)32, signed char, signed char, int, int>::Run(ruy::PMat<signed char> const&, ruy::PMat<signed char> const&, ruy::MulParams<int, int> const&, int, int, int, int, ruy::Mat<int>*) const':
/usr/src/debug/tensorflow-lite/2.4.0-r0/build/ruy/ruy/kernel_arm.h:123: undefined reference to `ruy::Kernel8bitNeonDotprodInOrder(ruy::KernelParams8bit<8, 8> const&)'

Most of the "undefined reference" errors are related to RUY functions that Tensorflow Lite uses so it seems that somehow these functions are missing in the static library (libtensorflow-lite.a - 179M). The app compilation command is based on the indications described in the i.MX Machine Learning User's Guide document:

$CC label_image.cc bitmap_helpers.cc ../../tools/evaluation/utils.cc \
-I=/usr/include/tensorflow/lite/tools/make/downloads/flatbuffers/include \
-I=/usr/include/tensorflow/lite/tools/make/downloads/absl \ -O1 -DTFLITE_WITHOUT_XNNPACK
-ltensorflow-lite -lstdc++ -lpthread -lm -ldl -lrt

Even using a very simplified code, the errors show up when performing this basic declaration:

tflite::ops::builtin::BuiltinOpResolver resolver;

I've checked the "tensorflow-lite_2.4.0.bb" file used to build Tensorflow Lite for the "gatesgarth" yocto distribution and the following flags are explicitly specified:

EXTRA_OECMAKE = "-DTFLITE_ENABLE_XNNPACK=on -DTFLITE_ENABLE_RUY=on -DTFLITE_ENABLE_NNAPI=on  ${S}/tensorflow/lite/"

Then, seems that RUY is enabled when compiling the static library but it does not work when compiling apps with it.

Any ideas of what it might be happening? I did not have this issue in the older "zeus" SDK.

Thank you very much.

[raluca_popa · Accepted solution]

Hi,

For the L5.10.9_1.0.0 and L5.4.70_2.3.2 releases, the tensorflow-lite library is built as static. This causes object files (and symbols from them) which are dependencies of tensorflow-lite (ruy, absl, flatbuffers...) to be excluded. The solution is to compile tensorflow-lite library as dynamic to force the compiler to resolve the symbols at runtime.

One can compile the dynamic version of libtensorflow-lite following the steps below. Refer to the attached archive containing the yocto recipes and use the version corresponding to the appropriate release.

• In the YOCTO build folder, overwrite the content of the 'sources/meta-imx' folder.
• Clean build tensorflow-lite

bitbake -f -c cleanall tensorflow-lite
bitbake -f -c compile tensorflow-lite

• NOTE: the lib should be successfully generated at this point, check:

         ---> 5.4.70_2.3.2: $YOCTO_BUILD_ROOT/build-xwayland/tmp/work/aarch64-poky-linux/tensorflow-lite/2.4.0-r0/build/libtensorflow-lite.so

          ---> 5.10.9_1.0.0: $YOCTO_BUILD_ROOT/build-xwayland/tmp/work/cortexa53-crypto-poky-linux/tensorflow-lite/2.4.0-r0/build/libtensorflow-lite.so

• Generate the toolchain: 'bitbake -c populate_sdk imx-image-full'
• Install the toolchain using the script at '$YOCTO_BUILD_ROOT/build-xwayland/tmp/deploy/sdk'
  NOTE: The lib should be included in the toolchain at this point, check:

         ---> 5.4.70_2.3.2: $YOCTO_SDK_ROOT/sysroots/aarch64-poky-linux/usr/lib/

         ---> 5.10.9_1.0.0: $YOCTO_SDK_ROOT/sysroots/cortexa53-crypto-poky-linux/usr/lib

• Copy the appropriate ‘libtensorflow-lite.so’ in the Yocto toolchain install:

      ---> 5.4.70_2.3.2: $YOCTO_SDK_ROOT/sysroots/aarch64-poky-linux/usr/lib/

      ---> 5.10.9_1.0.0: $YOCTO_SDK_ROOT/sysroots/cortexa53-crypto-poky-linux/usr/lib
 
• Rebuild the app:
  

  $CC label_image.cc bitmap_helpers.cc -o label_image ../../tools/evaluation/utils.cc -I=/usr/include/tensorflow/lite/tools/make/downloads/flatbuffers/include -I=/usr/include/tensorflow/lite/tools/make/downloads/absl -O1 -DTFLITE_WITHOUT_XNNPACK -ltensorflow-lite -lstdc++ -lpthread -lm -ldl -lrt​

Build will be successful, the binary that is generated is ‘label_image’

• Deploy on the board in ‘/usr/bin/tensorflow-lite-2.4.0/examples':
  
  • The label_image that was just compiled (overwrite or rename the existing one)
  • The ‘libtensorflow-lite.so'. Create symbolic link with version: ‘ln -s libtensorflow-lite.so libtensorflow-lite.so.2.4.0’

• $: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.​

• Run the application

NOTE: this issue will be fixed in the next YOCTO BSP release L5.10.9_2.0.0

[danmartzla]

Hi!

Thank you so much for your answer. This workaround worked well and now I can compile TFLite 2.4.0 apps. Thank you again.

However, now I'm having the following messages when executing the apps with TFLite 2.x models:

INFO: Created TensorFlow Lite delegate for NNAPI.
ERROR: ModifyGraphWithDelegate is disallowed when graph is immutable.
ERROR: Ignoring failed application of the default TensorFlow Lite delegate indexed at 0.

Despite of these "errors", the inference finishes well but I'm not sure whether this is affecting the model performance. Running the already pre-compiled "label_image" (with the TFLite static library) in the Yocto image does not show this error when using the same models. Yet, the average inference time is almost the same in both.

Furthermore, now it does not shows warnings referencing NNAPI non-compatible layers/operations. These layers (RESIZE_BILINEAR, LEAKY_RELU, ...) are still not compatible because they are still segmented according to the "benchmark_model" operation profiling logs.

Thank you again.