failed to build tensorflow on offline machine for imx8mp Scarthgap

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

failed to build tensorflow on offline machine for imx8mp Scarthgap

682 Views
greeran
Contributor IV

Hi

I’m building the fsl-image-qt6 image for i.MX8MP with Scarthgap. I built it on an online machine and copied the downloads directory to the offline machine. I’ve successfully used this workflow many times before with other BSP releases (Kirkstone, Zeus, and others), so this isn’t my first offline BSP build.

However, when I try to build the same image on the offline machine, the build fails at tensorflow-lite-tools-native. It seems to attempt cloning extended third-party dependencies (protobuf, etc.), and the downloaded tar.gz files are not sufficient to satisfy it.

Has anyone come across this issue before? If so, what’s the recommended way to resolve it?

Thanks in advance

0 Kudos
Reply
6 Replies

345 Views
greeran
Contributor IV

i delete all my build, download and sstate and rebuild from scratch. the build was long but successful. i then removed the build_xwayland directory (and kept the downloads and sstate path in the site.conf). i added this flags to the 

BB_NO_NETWORK = "1"
BB_FETCH_PREMIRRORONLY = "1"

i removed the internet cable to simulate my offline computer. when running the bitbake fsl-image-qt6 it fails in the parsing

WARNING: /workdir/var-fslc-yocto/sources/meta-nxp-connectivity/meta-nxp-matter-baseline/recipes-matter/matter/matter.bb: Exception during build_dependencies for fetcher_hashes_dummyfunc | ETA: 0:00:04
WARNING: /workdir/var-fslc-yocto/sources/meta-nxp-connectivity/meta-nxp-matter-baseline/recipes-matter/matter/matter.bb: Error during finalise of /workdir/var-fslc-yocto/sources/meta-nxp-connectivity/meta-nxp-matter-baseline/recipes-matter/matter/matter.bb
ERROR: ExpansionError during parsing /workdir/var-fslc-yocto/sources/meta-nxp-connectivity/meta-nxp-matter-baseline/recipes-matter/matter/matter.bb
Traceback (most recent call last):
File "/workdir/var-fslc-yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1248, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x78232b81e3d0>, d=<bb.data_smart.DataSmart object at 0x78233207c820>, name='default'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev = ud.method.latest_revision(ud, d, name)

File "/workdir/var-fslc-yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1670, in GitSM.latest_revision(ud=<bb.fetch2.FetchData object at 0x78232b81e3d0>, d=<bb.data_smart.DataSmart object at 0x78233207c820>, name='default'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud, d, name)
return rev
File "/workdir/var-fslc-yocto/sources/poky/bitbake/lib/bb/fetch2/git.py", line 850, in GitSM._latest_revision(ud=<bb.fetch2.FetchData object at 0x78232b81e3d0>, d=<bb.data_smart.DataSmart object at 0x78233207c820>, name='default'):

> output = self._lsremote(ud, d, "")
# Tags of the form ^{} may not work, need to fallback to other form
File "/workdir/var-fslc-yocto/sources/poky/bitbake/lib/bb/fetch2/git.py", line 832, in GitSM._lsremote(ud=<bb.fetch2.FetchData object at 0x78232b81e3d0>, d=<bb.data_smart.DataSmart object at 0x78233207c820>, search=''):
if ud.proto.lower() != 'file':
> bb.fetch2.check_network_access(d, cmd, repourl)
output = runfetchcmd(cmd, d, True)
File "/workdir/var-fslc-yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 970, in check_network_access(d=<bb.data_smart.DataSmart object at 0x78233207c820>, info='git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://github.com/NXP/matter.git ', url='https://github.com/NXP/matter.git'):
if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
> raise NetworkAccess(url, info)
elif not trusted_network(d, url):
bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY) but access requested with command git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://github.com/NXP/matter.git (for url https://github.com/NXP/matter.git)
The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

ERROR: Parsing halted due to errors, see error messages above

Summary: There were 2 WARNING messages.
Summary: There were 2 ERROR messages, returning a non-zero exit code.

to fix it i need to connect to the internet and then run bitbake fsl-image-qt6 --runall=fetch and then offline build works. the problem that in the offline machine i will not be able to do the runall=fetch.

might you know what my error is and why i am not able to build offline

thanks

0 Kudos
Reply

605 Views
greeran
Contributor IV

thanks for the response

i am sending the manifests from the repo and the error i get. i tried downloading the download twice and i still get the same error when running in offline (BB_NO_NETWORK = "1" and BB+GENERATE_MIRROR_TARBALLS  = "1" )

 

<manifest>
<include name="imx-6.6.52-2.2.0.xml" />
</manifest>

this is the manifests from the repo

<manifest>
<default revision="scarthgap" sync-j="4"/>
<remote name="yp" fetch="https://git.yoctoproject.org"/>
<remote name="yp-gh" fetch="https://github.com/yoctoproject"/>
<remote name="oe" fetch="https://github.com/openembedded"/>
<remote name="kraj" fetch="https://github.com/kraj"/>
<remote name="community" fetch="https://github.com/Freescale"/>
<remote name="ossystems" fetch="https://github.com/OSSystems"/>
<remote name="qt" fetch="https://github.com/YoeDistro"/>
<remote name="timesys" fetch="https://github.com/TimesysGit"/>
<remote name="imx" fetch="https://github.com/nxp-imx"/>
<remote name="imx-support" fetch="https://github.com/nxp-imx-support"/>
<remote name="sbabic" fetch="https://github.com/sbabic"/>
<remote name="variscite" fetch="https://github.com/varigit"/>
<project name="poky" remote="yp-gh" path="sources/poky" revision="200d12b6a58ad961d60a7774ca0f7a9d29498724" upstream="scarthgap"/>
<project name="meta-openembedded" remote="oe" path="sources/meta-openembedded" revision="72018ca1b1a471226917e8246e8bbf9a374ccf97" upstream="scarthgap"/>
<project name="meta-clang" remote="kraj" path="sources/meta-clang" revision="2b7433611d80f6d0ee1b04156fa91fc73d3c2665" upstream="scarthgap"/>
<project name="meta-freescale" remote="community" path="sources/meta-freescale" revision="0627128b341cfb2bef7a0832ce8cac0ce1127f13" upstream="scarthgap"/>
<project name="meta-freescale-3rdparty" remote="community" path="sources/meta-freescale-3rdparty" revision="6c063450d464eb2f380443c7d9af1b94ce9b9d75" upstream="scarthgap"/>
<project name="meta-freescale-distro" remote="community" path="sources/meta-freescale-distro" revision="b9d6a5d9931922558046d230c1f5f4ef6ee72345" upstream="scarthgap"/>
<project name="meta-arm" remote="yp" path="sources/meta-arm" revision="950a4afce46a359def2958bd9ae33fc08ff9bb0d" upstream="scarthgap"/>
<project name="meta-qt6" remote="qt" path="sources/meta-qt6" revision="586a6cb5aec755803a3be3cec359baafe89d6432" upstream="6.8"/>
<project name="meta-browser" remote="ossystems" path="sources/meta-browser" revision="dc31889c0899971def535dc1c040edf18bc16691"/>
<project name="meta-security" remote="yp" path="sources/meta-security" revision="459d837338ca230254baa2994f870bf6eb9d0139" upstream="scarthgap"/>
<project name="meta-timesys" remote="timesys" path="sources/meta-timesys" revision="4dc3f4bdfde4a86904b6e1a3d58df4696e7a63fa" upstream="scarthgap"/>
<project name="meta-virtualization" remote="yp" path="sources/meta-virtualization" revision="6f3c1d8f90947408a6587be222fec575a1ca5195" upstream="scarthgap"/>
<project name="meta-imx" remote="imx" path="sources/meta-imx" revision="219f6d04a4c339eb6f2dc626f944bbdf9a716ff5" upstream="scarthgap-6.6.52-2.2.0">
<linkfile src="tools/imx-setup-release.sh" dest="imx-setup-release.sh"/>
<linkfile src="README" dest="README-IMXBSP"/>
</project>
<project name="meta-nxp-connectivity" remote="imx" path="sources/meta-nxp-connectivity" revision="f58365ec75a5768ff3e8e8bcd64f237738c93160" upstream="imx_matter_2024_q3-post"/>
<project name="meta-nxp-demo-experience" remote="imx-support" path="sources/meta-nxp-demo-experience" revision="e8e646e22b1f926aac83bb7ad522efa9cd8ccc45" upstream="scarthgap-6.6.52-2.2.0"/>
<project name="meta-swupdate" remote="sbabic" path="sources/meta-swupdate" revision="d598d4e675b541301ea1dfc8f0c8931983b4dcd0" upstream="scarthgap"/>
<project name="meta-variscite-bsp-common" remote="variscite" path="sources/meta-variscite-bsp-common" revision="10fb8ca3956c6359a9f744da47197048cc28a483" upstream="scarthgap_6.6.52-2.2.0_var01"/>
<project name="meta-variscite-bsp-imx" remote="variscite" path="sources/meta-variscite-bsp-imx" revision="19052e7e0b96814b84069fdfd0d03568271c7ec6" upstream="scarthgap_6.6.52-2.2.0_var01"/>
<project name="meta-variscite-hab" remote="variscite" path="sources/meta-variscite-hab" revision="ec4e9f1b2f99ad03077af6b847c38f6e9c2378a7" upstream="scarthgap_6.6.52-2.2.0_var01"/>
<project name="meta-variscite-sdk-common" remote="variscite" path="sources/meta-variscite-sdk-common" revision="ee4ade553e063e5daed4e083c62ebff647d9e7dd" upstream="scarthgap_6.6.52-2.2.0_var01"/>
<project name="meta-variscite-sdk-imx" remote="variscite" path="sources/meta-variscite-sdk-imx" revision="fbd0a9e4a7390f66e63af63ec7338bced09686f1" upstream="scarthgap_6.6.52-2.2.0_var01">
<linkfile src="scripts/var-setup-release.sh" dest="var-setup-release.sh"/>
</project>
<project name="variscite-bsp-base" remote="variscite" path="sources/base" revision="fa940410b6fa12155da5827c06351791147107d0" upstream="scarthgap_6.6.52-2.2.0_var01">
<linkfile src="setup-environment" dest="setup-environment"/>
<linkfile src="README" dest="README"/>
<linkfile src="scripts/var-start-container.sh" dest="var-start-container.sh"/>
</project>
<project name="var-host-docker-containers" remote="variscite" path="sources/var-host-docker-containers" revision="dd7218bb6e1c76db77cf40552b4cad7dfaa795fc" upstream="master"/>
</manifest>
 
the error i get 

Initialising tasks: 100% |##########################################################################################################################################################################| Time: 0:00:12
NOTE: Executing Tasks
ERROR: tensorflow-lite-host-tools-native-2.16.2-r0 do_configure: ExecutionError('/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/temp/run.do_configure.179032', 1, None, None)
ERROR: Logfile of failure stored in: /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/temp/log.do_configure.179032
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/workdir/sources/poky/meta/recipes-devtools/cmake/cmake-native_3.28.3.bb:do_populate_sysroot', '/workdir/sources/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_populate_sysroot', 'virtual:native:/workdir/sources/poky/meta/recipes-devtools/ninja/ninja_1.11.1.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['gettext-minimal-native', 'cmake-native', 'libtool-native', 'quilt-native', 'texinfo-dummy-native', 'openssl-native', 'ncurses-native', 'zlib-native', 'make-native', 'ninja-native', 'perl-native', 'bzip2-native', 'xz-native', 'zstd-native', 'gdbm-native', 're2c-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_configure
| + export HTTP_PROXY=
| + export HTTPS_PROXY=
| + export http_proxy=
| + export https_proxy=
| + cmake_do_configure
| + [ ]
| + [ /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git = /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/build ]
| + [ -e /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/site-file.cmake ]
| + oecmake_sitefile=
| + cmake -G Ninja -DCMAKE_MAKE_PROGRAM=ninja /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/lite/tools/cmake/native_tools -DCMAKE_INSTALL_PREFIX:PATH=/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr -DCMAKE_INSTALL_BINDIR:PATH=bin -DCMAKE_INSTALL_SBINDIR:PATH=sbin -DCMAKE_INSTALL_LIBEXECDIR:PATH=libexec -DCMAKE_INSTALL_SYSCONFDIR:PATH=/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/etc -DCMAKE_INSTALL_SHAREDSTATEDIR:PATH=../com -DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/var -DCMAKE_INSTALL_LIBDIR:PATH=lib -DCMAKE_INSTALL_INCLUDEDIR:PATH=include -DCMAKE_INSTALL_DATAROOTDIR:PATH=share -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DLIB_SUFFIX= -DCMAKE_INSTALL_SO_NO_EXE=0 -DCMAKE_TOOLCHAIN_FILE:FILEPATH=/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/toolchain.cmake -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DFETCHCONTENT_FULLY_DISCONNECTED=ON -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON -DFETCHCONTENT_FULLY_DISCONNECTED=OFF -DCMAKE_SYSROOT= -Wno-dev
| -- The C compiler identification is GNU 9.4.0
| -- The CXX compiler identification is GNU 9.4.0
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Check for working C compiler: /workdir/build/tmp/hosttools/gcc - skipped
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Check for working CXX compiler: /workdir/build/tmp/hosttools/g++ - skipped
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Cloning flatbuffers repository from http://localhost:8080/7d6d99c6befa635780a4e944d37ebfd58e68a108.tar.gz, found in /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/../third_party/flatbuffers/workspace.bzl
| -- Cloning flatbuffers repository revision 7d6d99c6befa635780a4e944d37ebfd58e68a108, found in /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/../third_party/flatbuffers/workspace.bzl
| -- Proceeding with version: 23.5.26.0
| -- Looking for strtof_l
| -- Looking for strtof_l - found
| -- Looking for strtoull_l
| -- Looking for strtoull_l - found
| -- Looking for realpath
| -- Looking for realpath - found
| -- CMAKE_CXX_FLAGS: -isystem/workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/include -O2 -pipe
| -- Cloning https://github.com/protocolbuffers/protobuf...
| [1/9] Creating directories for 'protobuf-populate'
| [1/9] Performing download step (git clone) for 'protobuf-populate'
| Cloning into 'protobuf'...
| fatal: unable to access 'https://github.com/protocolbuffers/protobuf/': Could not resolve host: github.com
| Cloning into 'protobuf'...
| fatal: unable to access 'https://github.com/protocolbuffers/protobuf/': Could not resolve host: github.com
| Cloning into 'protobuf'...
| fatal: unable to access 'https://github.com/protocolbuffers/protobuf/': Could not resolve host: github.com
| -- Had to git clone more than once: 3 times.
| CMake Error at _deps/protobuf-subbuild/protobuf-populate-prefix/tmp/protobuf-populate-gitclone.cmake:39 (message):
| Failed to clone repository: 'https://github.com/protocolbuffers/protobuf'
|
|
| FAILED: protobuf-populate-prefix/src/protobuf-populate-stamp/protobuf-populate-download /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/build/_deps/protobuf-subbuild/protobuf-populate-prefix/src/protobuf-populate-stamp/protobuf-populate-download
| cd /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/build && /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/bin/cmake -P /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/build/_deps/protobuf-subbuild/protobuf-populate-prefix/tmp/protobuf-populate-gitclone.cmake && /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/bin/cmake -E touch /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/build/_deps/protobuf-subbuild/protobuf-populate-prefix/src/protobuf-populate-stamp/protobuf-populate-download
| ninja: build stopped: subcommand failed.
|
| CMake Error at /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/share/cmake-3.28/Modules/FetchContent.cmake:1679 (message):
| Build step for protobuf failed: 1
| Call Stack (most recent call first):
| /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/share/cmake-3.28/Modules/FetchContent.cmake:1819:EVAL:2 (__FetchContent_directPopulate)
| /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/recipe-sysroot-native/usr/share/cmake-3.28/Modules/FetchContent.cmake:1819 (cmake_language)
| /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/lite/tools/cmake/modules/OverridableFetchContent.cmake:537 (FetchContent_Populate)
| /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/lite/tools/cmake/modules/protobuf.cmake:37 (OverridableFetchContent_Populate)
| /workdir/build/tmp/work/x86_64-linux/tensorflow-lite-host-tools-native/2.16.2/git/tensorflow/lite/tools/cmake/modules/Findprotobuf.cmake:18 (include)
| protobuf/CMakeLists.txt:34 (find_package)
|
|
| -- Configuring incomplete, errors occurred!
| WARNING: exit code 1 from a shell command.
| + bb_sh_exit_handler
| + ret=1
| + [ 1 != 0 ]
| + echo WARNING: exit code 1 from a shell command.
| + exit 1
ERROR: Task (virtual:native:/workdir/sources/meta-imx/meta-imx-ml/recipes-libraries/tensorflow-lite/tensorflow-lite-host-tools_2.16.2.bb:do_configure) failed with exit code '1'
NOTE: Tasks Summary: Attempted 5477 tasks of which 5471 didn't need to be rerun and 1 failed.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 9 seconds

 

0 Kudos
Reply

575 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello,

You need to erase everything and restart to download yocto, since I can not reproduce the issue in my building. If you are working with none BSP from NXP, please contact your vendor for this issue.

Regards

0 Kudos
Reply

282 Views
greeran
Contributor IV

hello

after a number of deletions and build i have found a workaround for the problem. it seems that the branch and has for the matter did not match between the downloads/github..matter.git to the bb file. i added to my site.conf 

SRCBRANCH:pn-matter = "release/v1.4.

SRCREV:pn-matter = "9042e9281bf619ce0d06d043d5ac3c0c34c24bbe"

and that solved the parsing error. after that  i have a fetch error that tried to get the thirdparty/nxp_matter_support that i did not find in my downloads (which successfully build an image with internet). so to workaround this problem i removed all the installs for "packagegroup-nxp-matter" and such. 

This did not solve the problem but enabled me to build my image and because i do not need the application that use the matter it is a sufficient workaround 

thanks

0 Kudos
Reply

634 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello,

If you download the complete bsp and build the full-image you should not have any issue, please tell us which BSP are you using to test.

Regards

0 Kudos
Reply

472 Views
greeran
Contributor IV

i am deleted all the build and download and sstate. i am now building from scratch and hopefully tomorrow i will be able to test it with a clean build on a machine without internet