Yocto Build Fail: onnxruntime Fails on do_configure with Eigen Download SHA1 Hash Mismatch

[dongare_anil]

iMX95 IMX8MPLUS 
#yocto

Hello NXP Community,

I am working on a Yocto build for an i.MX 8M Plus target and have run into a consistent build failure with the onnxruntime recipe from the meta-imx-ml layer.

The build fails during the do_configure task with a checksum mismatch when trying to download the Eigen library dependency.

Build Environment:

• BSP Version: Based on imx-linux-kirkstone (or a recent styhead branch)
• Machine: imx8mp-evk
• Distro: fsl-imx-xwayland
• Host OS: Ubuntu 20.04.3 LTS
• Build Command: bitbake imx-image-full

Problem Description:

The build process for onnxruntime uses CMake's FetchContent to download the Eigen library. The download appears successful, but it fails at the verification step because the SHA1 hash of the downloaded file does not match the expected hash in the build script.

Error Log Snippet:

Here is the relevant failure from the log:

Code snippet

-- verifying file...
|        file='/home2/anil/IMX8MP/build-xwayland/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/build/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-e7248b26a1ed53fa030c5c459f7ea095dfd276ac.zip'
| -- SHA1 hash of
|      /home2/anil/IMX8MP/build-xwayland/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/build/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-e7248b26a1ed53fa030c5c459f7ea095dfd276ac.zip
|   does not match expected value
|      expected: 'be8be39fdbc6e60e94fa7870b280707069b5b81a'
|        actual: '32b145f525a8308d7ab1c09388b2e288312d8eba'
| -- Hash mismatch, removing...
| CMake Error at eigen-subbuild/eigen-populate-prefix/src/eigen-populate-stamp/download-eigen-populate.cmake:163 (message):
|   Each download failed!
|
|
| make[2]: *** [CMakeFiles/eigen-populate.dir/build.make:100: eigen-populate-prefix/src/eigen-populate-stamp/eigen-populate-download] Error 1
| make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/eigen-populate.dir/all] Error 2
| make: *** [Makefile:91: all] Error 2
|
| CMake Error at /home2/anil/IMX8MP/build-xwayland/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/recipe-sysroot-native/usr/share/cmake-3.30/Modules/FetchContent.cmake:1918 (message):
|   Build step for eigen failed: 2

Troubleshooting Steps Already Taken:

1. Full Clean: I have run bitbake -c cleanall onnxruntime to remove all cached sstate and work directory files.
2. Manual Download Cache Clearing: I have manually deleted the eigen-*.zip file from my downloads directory to ensure the build attempts a fresh download.
3. Network Check: I have confirmed that this is not a network proxy issue. Manually downloading the file with wget and checking its hash with sha1sum confirms that the "actual" hash (32b145f...) is what is being served from GitLab, indicating the file upstream may have changed or been re-packaged.

My Questions:

1. Is this a known issue with the onnxruntime recipe in the meta-imx-ml layer?
2. What is the recommended "Yocto-way" to resolve this? Is there an official patch available that updates the checksum or, preferably, adds Eigen as a proper SRC_URI dependency to avoid the FetchContent download during do_configure?

Any help or guidance on how to properly patch this recipe would be greatly appreciated.

Thank you,
Anil Dongare.

[Bugger]

Looked at the skyhead Yocto repo for IMX and found that this issue is not fixed in there. Since I am just starting off with Yocto perhaps my interpretation is wrong.  So as I estimate, with the NXP skyhead Yocto repo the build will fail on onnxruntime as well.

NXP uses own onnxruntime repo which version is directly linked to the Yotco image version:

NXP IMX scarthgap 6.6.36.2.1.0 Yocto
meta-imx/meta-imx-ml/recipes-libraries/onnxruntime/onnxruntime_1.17.1.bb

ONNXRUNTIME_SRC ?= "gitsm://github.com/nxp-imx/onnxruntime-imx.git;protocol=https"
SRCBRANCH = "lf-6.6.36_2.1.0"
SRCREV = "813b93f64103847a21227725cb5ab04b6ca5b5ed"

The onnx repo points to the specific onnx repo which has the faulty cmake depts.txt with eigen

onnxruntime-imx/cmake/deps.txt
eigen;https://gitlab.com/libeigen/eigen/-/archive/e7248b26a1ed53fa030c5c459f7ea095dfd276ac/eigen-e7248b26a...

NXP IMX skyhead 6.12.3.1.0 Yocto
meta-imx/meta-imx-ml/recipes-libraries/onnxruntime/onnxruntime_1.17.1.bb
ONNXRUNTIME_SRC ?= "gitsm://github.com/nxp-imx/onnxruntime-imx.git;protocol=https"
SRCBRANCH = "lf-6.12.3_1.0.0"
SRCREV = "3616ba2f9cd2b7b882252a95e171f0c0c0f1826f"

In imx skyhead Yocto repo the onnxruntime is still the faulty one as well

eigen;https://gitlab.com/libeigen/eigen/-/archive/e7248b26a1ed53fa030c5c459f7ea095dfd276ac/eigen-e7248b26a...

There is a newer NXP imx onnxruntime branch lf.6.12.20-2.00 that has the correct eigen checksum for cmake devs.txt
but this has not been included in NXP's the latest Yocto skyhead repo

eigen;https://gitlab.com/libeigen/eigen/-/archive/1d8b82b0740839c0de7f1242a3585e3390ff5f33/eigen-1d8b82b07...

[Bugger]

Bumped into the same issue today with the Full build for FRDM-IMX93. cleaning does not fix it.

When this is going to be resolved. Stuck now. Building full image takes days. Don't want to start all over again and bumped into the same issue due to a potential upstream faulty source code

-- SHA1 hash of
| /home/build/yocto/frdm-imx93/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/build/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-e7248b26a1ed53fa030c5c459f7ea095dfd276ac.zip
| does not match expected value
| expected: 'be8be39fdbc6e60e94fa7870b280707069b5b81a'
| actual: '32b145f525a8308d7ab1c09388b2e288312d8eba'

[Bugger]

My process that worked since I wanted to understand how the patch above worked and how to handle future occasions of similar issues.

1.  bitbake -c cleanall onnxruntime   

clean or cleanall resets the checksum in the dept.txt file  to the wrong value.  Don't run this after patching.

2. cat ~/yocto/frdm-imx93/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/git/cmake/deps.txt

3. find the line with "eigen" 

eigen;https://gitlab.com/libeigen/eigen/-/archive/e7248b26a1ed53fa030c5c459f7ea095dfd276ac/eigen-e7248b26a...

4. wget https://gitlab.com/libeigen/eigen/-/archive/e7248b26a1ed53fa030c5c459f7ea095dfd276ac/eigen-e7248b26a...

5. sha1sum eigen-e7248b26a1ed53fa030c5c459f7ea095dfd276ac.zip

will report as checksum 32b145f525a8308d7ab1c09388b2e288312d8eba

6. edit deps.txt and change eigen line to 

eigen;https://gitlab.com/libeigen/eigen/-/archive/e7248b26a1ed53fa030c5c459f7ea095dfd276ac/eigen-e7248b26a...

7. continue the yocto build 

Best way would be if NXP makes an announcement and releases a patch that does this all  (or create an AI agent that will confirm for this well known problem.. "you are absolutely right, this is a problem in yocto and here is the way to fix it",   That would be better than assuming the download image is corrupt. BTW Instead of patching the checksum, Yocto could use the latest eigen package release ?

[guepy]

Looks like the problem is related to the eigen package versionning and it's well known on ubuntu as it can be described from https://github.com/microsoft/onnxruntime/issues/18286.
A gross temporary fix  is to edit the hash used to verfy the download. I prepare a patch for the onnxruntime package to edit the hash value of the eigen package so that it corresponds to what the hash of the online branch currently gives, which is  32b145f525a8308d7ab1c09388b2e288312d8eba.
You can also find below the bbapend file for onnxruntime. So go to the onnxruntime package source folder located here: sources/meta-imx/meta-imx-ml/recipes-libraries/onnxruntime, and paste the bbappend file. while there, Create a folder named files and paste the patch inside.
Rerun bitbake -c configure onnxruntime, and it should be ok.
It may be the ugliest way to deal with but I don't have much time to investigate further.
Thanks

[Vince100]

Worked for me. The patch format has changed slightly, need to add
Upstream-Status: Pending
but otherwise this worked.

[Bugger]

The patch changes  git/cmake/deps.txt,

I can't seem to find this file in the yocto repo... (?)

[Bugger]

Found it. ~/yocto/frdm-imx93/tmp/work/armv8a-poky-linux/onnxruntime/1.17.1/git/cmake

[davidruffen]

I just found this post which looks to be the error -- not totally sure?

https://github.com/microsoft/onnxruntime/issues/24861 

Just not sure how to pull in this fix from imx-linux-scarthgap.

[davidruffen]

I am also seeing the same issue trying to build IMX95 (build information below) and also looking for a solution to this.

Any help would be appreciated.

Thank you in advance.

Build Environment:

• BSP Version: Based on imx-linux-scarthgap -m imx-6.6.52-2.2.0.xml
• Machine: imx95-19x19-verdin
• Distro: fsl-imx-xwayland
• Host OS: Ubuntu 22.04.05. LTS
• Build Command: bitbake imx-image-full

[Bio_TICFSL]

Hello,

This fail also with full-image on iMX95? this is probable a bug let's check it, since I can not reproduce this issue, is probably a bad package on your host.

Regards

[dongare_anil]

Yes, this also fail with full-image on imx8mp and its reproducible.

[davidruffen]

It is very reproducible.
I did bitbake -f -c cleanall onnxruntime to try to clean it out.
Any other suggestions?

[Bio_TICFSL]

Hello,

Just remove the onnx and clean all:

bitbake -c cleanall 

Regards

[davidruffen]

I have attached the build log and error.

[davidruffen]

I have tried building in a new workspace and this error continues to happen.

bitbake -c clean onnxruntime does not resolve this.

[tatula]

I can confirm that a full clean build will replicate this issue.