Questions/thoughts on the download (fetch) system

[yvongrall]

Hi all,

I'm building the imx6sabresd machine with the Linux 4.1.15_1.2.0 BSP. Unfortunately, the network access to freescale/NXP servers (git://git.freescale.com) is erratic : I had to re-run several times the build (bitbake core-image-minimal) when it comes to fetch u-boot. So I'm trying to understand the download (fetch) system.

When it fails, we can see this in the corresponding log.do_fetch file :

DEBUG: Trying PREMIRRORS
[...]
DEBUG: For url git://git.freescale.com/imx/uboot-imx.git;protocol=git;branch=imx_v2015.04_4.1.15_1.0.0_ga returning http://downloads.yoctoproject.org/mirror/sources/git2_git.freescale.com.imx.uboot-imx.git.tar.gz
[...]
DEBUG: Fetcher failure: Fetch command failed with exit code 8, output:
http://downloads.yoctoproject.org/mirror/sources/git2_git.freescale.com.imx.uboot-imx.git.tar.gz:
2017-02-23 12:00:25 ERROR 404: Not Found.
[...]
DEBUG: Trying Upstream
DEBUG: Fetcher failure: Fetch command failed with exit code 128, output:
Cloning into bare repository '/home/yon/fsl-nxp-1.2.0/downloads//git2/git.freescale.com.imx.uboot-imx.git'...
fatal: read error: Connection reset by peer
[...]
DEBUG: Trying MIRRORS
[...]
DEBUG: Fetcher failure: Fetch command failed with exit code 8, output:
http://sources.openembedded.org/git2_git.freescale.com.imx.uboot-imx.git.tar.gz:
2017-02-23 12:00:29 ERROR 404: Not Found.
ERROR: Fetcher failure: Fetch command failed with exit code 128, output:
Cloning into bare repository '/home/yon/fsl-nxp-1.2.0/downloads//git2/git.freescale.com.imx.uboot-imx.git'...
fatal: read error: Connection reset by peer
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
ERROR: Function failed: Fetcher failure for URL: 'git://git.freescale.com/imx/uboot-imx.git;protocol=git;branch=imx_v2015.04_4.1.15_1.0.0_ga'. Unable to fetch URL from any source.

1. The build system maps git://git.freescale.com/imx/uboot-imx.git to http://downloads.yoctoproject.org/mirror/sources/git2_git.freescale.com.imx.uboot-imx.git.tar.gz, or later to http://sources.openembedded.org/git2_git.freescale.com.imx.uboot-imx.git.tar.gz.
   Q : where are these mappings defined (in which config/include/recipe/anything_else file) ?
2. Some archives on the mirrors are large (e.g. 1 GB for git2_git.freescale.com.imx.linux-2.6-imx.git.tar.gz). So in case when they are fetched, it may take a long time to download.
   Q : Why this huge size, is it because an archive contains all branches/tags of a package ?
   
3. On the other hand, in case when the build system is able to connect to the git repository, I guess it only downloads the desired branch/tag, which is much faster.
   Q : shouldn't NXP provide public git clones as fallbacks instead of compressed archives on http/ftp mirrors ?
   
4. Assuming the DEBUG lines were printed in chronological order, the build system seems to try to fetch at first on the mirror, before attempting to fetch on the primary source.
   Q1 : shouldn't the primary source be tried at first ?
   Q2 : how to disable mirrors (and only use the freescale git repository instead) ?
5. At the time of writing, there's indeed no git2_git.freescale.com.imx.uboot-imx.git.tar.gz within the mirrors. Searching in the NXP community forums, we can see that many users encountered similar issues for some other packages : this issue (of non up-to-date mirrors) seems quite frequent.
   Q : would there be any acceptable generic workaround to that problem ? e.g. such as git-cloning the repositories, and then pointing SRC_URI to that local clone via file://....

Thanks in advance for your help,

-Yvon

[gusarambula · Accepted solution]

Hello Yvon,

The NXP git does allow to download specific branches or commits

http://git.freescale.com/git/cgit.cgi/imx/uboot-imx.git/

The mirrors may not offer all alternatives and are sometimes impractical as they may contain compressed files etc. I’m not sure if there is a unified practice in regards of how mirrors are offered.

You may be able to disable mirrors by setting the variables MIRRORS/PREMIRRORS to an empty string on local.conf although I haven’t tried this. I’m also trying to sort out how mirrors and premirrors are handled in terms of priorities.

A good option when working with a not so stable connection or when you would like to work offline at some point is to fetch the required sources for a specific image or package and building at a later time. You can do this by running:

$ bitbake <image_or_recipe> -c fetchall

I know this is a workaround, but it may be helpful in some scenarios and it’s easier than cloning the repositories and then changing SRC_URI to the local clone.

I hope this information helps! My apologies for the inconvenience.

Regards,