[bluebox] Yocto issues with Codeaurora relics on Github ?

kiashade · ‎05-01-2023

Greetings All,

some guidance/orientation would be highly appreciated. Big thank you in advance, if so. knows how to crack that nut !

PROBLEM STATEMENT

we've tried building Bluebox repo obtained from github but do run into issues as early as at recipe parsing stage of Yocto due to a host of old Codeaurora URIs. Seems to affect all BSPs, no matter the rev (tried 36,35,....).

https://github.com/nxp-auto-linux/auto_yocto_bsp/tree/bsp36.0

Lots of URIs to git://source.codeaurora.org in many dozens of .bb files.

ONE OFF - ATTEMPT

Tried changing them to github manually, for instance for optee package. Works. But very tedious to do for all packages.

THEORY SOUGHT

Please see embedded image.

Yocto, in its enigmatic way spits out sth about...
SRCPV -> PV -> PKGV -> EXTENDPKGV -> RDDEPENDS...
... which appears to be job security measure for yoctoproject.org consultants.

Blue arrow (in img): How exactly bitbake catches wind of new github SCM we dont know - grepped the entire folder structure recursively - github URI nowhere to be found.

Blue and Red square (in img): How exactly SRCREV_FORMAT could be used/set to which value, in order to direct bitbake to discard codeaurora URIs and proceed to github ones only is not detailed , nor described on yoctoproject.org where that var is mentioned but its use not really detailed by example.

GENERALIZING ATTEMPT CONSIDERED (could someone confirm / deny?)

Facing all these double URIs confusing bitbake,

and in absence of clear definition of how to set SRCREV_FORMAT globally,

and in rejection to having to change everything manually (in all 60..100 .bb files),

we consider conducting heresy by changing data_smart.py or utils.py in bitbake, where the ambiguity by two SCMs creates the error causing bitbake to abort.

Could someone share some thought on that ?
-OR-

Could someone advise what/if NXP is going to correct those URIs ?

Thanks everyone !

BR !

krivolapov · ‎06-08-2023

Hi,

We needed the SDK for a customer, so I recently spent a few days reshuffling the scripts. Everything is ready-to-go now, let me know if you need it (kid@mcobject.com) and I’ll send the download links

kiashade · ‎05-18-2023

Greetings @Daniel-Aguirre

Thanks, for script and guidance ! Much appreciated.

Firstly, it works !

Secondly, bitbake fails for a couple older BSPs (~24,25,26) and newer BSPs (~30-36). Missing URIs, patch files not found, patch target files not found,...

Bakes do work for BSP29.0.hf1 and BSP23.

Nothing special about our build machine. Single socket Xeon, 18.04.6 Desktop, excuted host-prepare.sh without hiccup. Everything normal.

Machines we build for are {ls2084abbminiubuntu , s32v234bbminiubuntu} , Image built {fsl-image-ubuntu-base}.

Any idea why these many BSP bakes fail ?

Thanks a bunch !

Daniel-Aguirre · ‎05-01-2023

Hi,

The migration from Code Aurora to GitHub is still on going, hence the behavior you are seeing. The internal team has provided the following patch and instructions on how to use it. Given that this was done as quickly as possible to enable the usage of GitHub during the migration, you could encounter some roadblocks non-found by our team. If so, we do apologize.

Instructions below (patch attached):

Hello,

In order to facilitate the build for older BSPs after CAF shutdown, we have implemented a bash script which is able to change the manifest and Yocto files by adapting the old CAF git repository links to corresponding NXP GitHub repository links. This will allow Yocto builds to be performed without the need to pull sources from CAF.

Currently, there are 2 operating modes:
RECOMMENDED: fully-automated mode
in this mode, the user starts from scratch and the script will handle everything: cloning from upstream and adjusting sources to NXP GitHub
how to use:
./migrate.sh --full --work_path ./testfolder --release_branch release/bsp32.0
inside the work_path given (e.g. "./testfolder") it will create the "fsl-image-auto-bsp" folder, and inside this folder it will start performing repo init and repo sync with adjusted URLs
after the script has finished, you can go inside the project (e.g. ./testfolder/fsl-image-auto-bsp) and perform the normal steps for the yocto build:
enter the bitbake shell
source nxp-setup-alb.sh -m <target_machine>
start a build
bitbake <target-image>
EXPERIMENTAL (NOT RECOMMENDED): postsync mode
in this mode, the hypothesis is that the user already has all the repos cloned, and encountered the issue a while after they performed "repo sync"

WARNING: ALL LOCAL REPO CHANGES SHOULD BE SAVED BEFORE STARTING IN THIS MODE!!! (e.g. via "git add" + "git commit" or other desired methods)
how to use:
./migrate.sh --postsync --work_path ./testfolder
ATTENTION: as a current limitation, work_path expects a directory that contains the "fsl-image-auto-bsp" folder inside of it. For example, if "fsl-image-auto-bsp" is inside the "testfolder", we should launch the folder from outside the "testfolder" with the path relative to it. Also, if your "fsl-image-auto-bsp" folder is named otherwise, you should rename it. This limitation will be addressed in the near future to facilitate other folder names as well.

After running in postsync mode, you can resume by entering the bitbake shell and launching a build, as described in the previous section.
Additionally, the help section of the script can be accessed via:

./migrate.sh --help

Other things to consider:

Because of the big scope of this change (all BSPs < 35) and time constraints, limited testing has been performed for this script (e.g. for BSP31.0, BSP 32.0, BSP 33.0). Because of this, failures can be expected when using it.

In addition to the script "migrate.sh", there are few more workarounds required for older BSPs. e.g:

- to build S32G yocto images with "pfe" enabled (BSP3x), the following lines must be added to "conf/local.conf":
DISTRO_FEATURES_append = " pfe"
NXP_FIRMWARE_LOCAL_DIR = "<path-to-pfe-firmware>"
SRC_URI_remove_pn-pfe = "git://github.com/nxp-auto-linux/extra/pfeng;protocol=https"
SRC_URI_prepend_pn-pfe = "git://github.com/nxp-auto-linux/pfeng;protocol=https "
SRC_URI_remove_pn-libfci-cli = "git://github.com/nxp-auto-linux/extra/pfeng;protocol=https"
SRC_URI_prepend_pn-libfci-cli = "git://github.com/nxp-auto-linux/pfeng;protocol=https "
- to build BlueBox QorIQ images (BSP3x, fsl-image-auto, fsl-image-blueboxdt), the following line must be added to "conf/local.conf" (to silence an error for a missing recipe):
BB_DANGLINGAPPENDS_WARNONLY = "1"
- to build S32V images with with BSPs 23.x, the following commands should be run in BSP root directory, after running "migrate.sh", before "nxp-setup-alb.sh" (to patch gcc 6.3 cross compiler):
sed -i 's#https://source.codeaurora.org/external/s32ds/compiler/gnu_nxp/plain/gcc.6-3-1-linaro-backport-armv8r-cortex-r52;name=r52;apply=yes#https://raw.githubusercontent.com/nxp-auto-tools/gnu_nxp/master/gcc.6-3-1-linaro-backport-armv8r-cortex-r52;apply=yes#g' ./sources/meta-alb/recipes-devtools/gcc/gcc-linaro-6.3-fsl.inc
sed -i 's#https://source.codeaurora.org/external/s32ds/compiler/gnu_nxp/plain/gcc.fix_armv8-libgfortran;name=fortran;apply=yes#https://raw.githubusercontent.com/nxp-auto-tools/gnu_nxp/master/gcc.fix_armv8-libgfortran;apply=yes #g' ./sources/meta-alb/recipes-devtools/gcc/gcc-linaro-6.3-fsl.inc
echo -e '\nBB_STRICT_CHECKSUM = "0"\n' >> ./sources/meta-alb/recipes-devtools/gcc/gcc-linaro-6.3-fsl.inc
- to build BlueBox Mini QorIQ images (BSP23.x, fsl-image-auto, fsl-image-blueboxdt), the following lines may be neded in "conf/local.conf" (as lxc package has been moved):
SRC_URI_remove_pn-lxc = "http://linuxcontainers.org/downloads/lxc-2.0.8.tar.gz"
SRC_URI_prepend_pn-lxc = "https://linuxcontainers.org/downloads/lxc/lxc-2.0.8.tar.gz "
BB_DANGLINGAPPENDS_WARNONLY = "1"
This is still work in progress, please do expect changes, patches etc as we validate more and more releases.

Best Regards,

0xEC · ‎06-01-2023

this migrate.sh script is also tryig to reach source.codeaurora.org address which is unavailable. i got following error messages

kiashade · ‎06-01-2023

Hi Erhan,

that NXP provided script did work well for us.

Maybe it helps -> below you see the BSPs we repo-ed using the migrate script along with full CMD.

Note 1: You can repo release/bspXYZ and/or hotfix/bspXYZ.

Note 2: Even though all migrate.sh based repos worked, bitbake failed in many cases - for various reasons. (We have yet to provide corresponding logs to Ionut)

Hope that helps a bit !

IonutV_Vicovan · ‎05-19-2023

Hello,

an article has been written on this topic (initially in other community forum); I have moved it here for better accessibility:

https://community.nxp.com/t5/S32G-Knowledge-Base/How-to-build-older-Auto-Linux-BSPs/ta-p/1637516

It contains updated script and instructions.

@kiashade,

Please do not hesitate to try it and share feedback.

Also please be so kind to share some logs with the failures, from the call to `repo init` or `migrate.sh`, as well as the OS versions of the machines used for the builds. Please note that for older BSPs only Ubuntu 18.04 was supported, for the newer ones (35,36) only Ubuntu 20.04. Please see the corresponding User Manual for details, as it is most important.

Best Regards,

Ionut

kiashade · ‎05-20-2023

Thanks @IonutV_Vicovan .

Quick Q - from which source can we download the corresponding User Manuals ?

The two main ones (plus few smaller ones) we do have are:
1. "Auto_Linux_BSP_23.0_BlueBox_Quick_Start.pdf" (for LS)
2. "Auto_Linux_BSP_23.0_S32V234_User_Manual.pdf" (for S32V)

Would be awesome to access corresponding docs for newer BSPs.

Thanks !

IonutV_Vicovan · ‎05-22-2023

Hi,

User manuals for newer BSPs are available through nxp.flexnetoperations.com.

You should use the same account as for BSP23, and you should search for the desired version of the product there. If you have issues, please contact your NXP representative which guided you for geting BSP23.

Regards,

Ionut

kiashade · ‎05-02-2023

Excellent, thx for speedy reply, will try that one !