Problems with Network Boot DHCP Hang on iMX6ULL EVK

[jeremy_cole]

I'm trying to set up network boot on an iMX6ULL evk board which has two ethernet ports. Linux image is built with buildroot, using freescale_imx6ullevk_defconfig. I had to apply a patch to get the ethernet working on this revision of the evk, but it's now all working well (including the network) if I boot the image from a uSD card. 

I now want to use network boot and NFS for easier development. I set up TFTP and NFS on my desktop, and shared the kernel image via TFTP and the rootfs via NFS. I removed the kernel and rootfs from the uSD card, and set up u-boot environment variables with the address of my desktop, etc, following this guide. (modified to use DHCP not fixed IP address, although setting a fixed address didn't help).

The u-boot part works great - it boots, finds an address with DHCP, and downloads the kernel. The kernel then boots OK. However, at the end of the boot, it hangs and appears to be trying to get an address via DHCP on both network interfaces. Some errors are displayed. After about a minute, it times out, then seems to get an address. It finishes the boot and I can log in. At this point it DOES have an address on eth0 (but not eth1, although that doesn't matter), and it's all working (and rootfs is definitely coming from my desktop via NFS). Nb I've tried with live cables plugged into both ethernet ports and just one but it doesn't help; the results are much the same.

Here is the output during the last stage of the boot:

8021q: 802.1Q VLAN Support v1.8
Key type dns_resolver registered
cpu cpu0: dev_pm_opp_get_opp_count: device OPP not found (-19)
dhd_module_init in
snvs_rtc 20cc000.snvs:snvs-rtc-lp: setting system clock to 1970-01-01 00:09:17 UTC (557)
fec 20b4000.ethernet eth0: Freescale FEC PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=20b4000.ethernet:01, irq=-1)
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
fec 2188000.ethernet eth1: Freescale FEC PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=20b4000.ethernet:02, irq=-1)
IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
fec 20b4000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sending DHCP requests .
fec 2188000.ethernet eth1: Link is Up - 100Mbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
.
DHCP/BOOTP: Reply not for us, op[2] xid[da6a8b7f]
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
.
DHCP/BOOTP: Reply not for us, op[2] xid[da6a8b7f]
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
.
DHCP/BOOTP: Reply not for us, op[2] xid[da6a8b7f]
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
.
DHCP/BOOTP: Reply not for us, op[2] xid[da6a8b7f]
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
.
DHCP/BOOTP: Reply not for us, op[2] xid[da6a8b7f]
DHCP/BOOTP: Reply not for us, op[2] xid[3fc25848]
random: nonblocking pool is initialized
 timed out!
IP-Config: Retrying forever (NFS root)...
fec 20b4000.ethernet eth0: Freescale FEC PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=20b4000.ethernet:01, irq=-1)
fec 2188000.ethernet eth1: Freescale FEC PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=20b4000.ethernet:02, irq=-1)
Sending DHCP requests .
fec 20b4000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
fec 2188000.ethernet eth1: Link is Up - 100Mbps/Full - flow control off
.,
DHCP/BOOTP: Reply not for us, op[2] xid[d496e039]
 OK
IP-Config: Got DHCP answer from 10.113.129.45, my address is 10.113.130.59
IP-Config: Complete:
     device=eth0, hwaddr=00:04:9f:04:f7:dc, ipaddr=10.113.130.59, mask=255.255.248.0, gw=10.113.128.2
     host=10.113.130.59, domain=nw.local, nis-domain=(none)
     bootserver=0.0.0.0, rootserver=10.113.130.94, rootpath=
     nameserver0=10.113.129.55, nameserver1=10.113.129.57, nameserver2=10.113.224.85
gpio_dvfs: disabling
VSD_3V3: disabling
can-3v3: disabling
ALSA device list:
  #0: wm8960-audio
VFS: Mounted root (nfs filesystem) readonly on device 0:14.
devtmpfs: mounted
Freeing unused kernel memory: 428K (80ae7000 - 80b52000)
devpts: called with bogus options
Starting logging: OK
Initializing random number generator... done.
Starting network: ip: RTNETLINK answers: File exists
FAIL
Starting dropbear sshd: OK

Welcome to i.MX6 EVK
iMX6 login: ...

I have DISABLED eth0 in /etc/network/interfaces, in the hope that it would not bother trying to DHCP, but this made NO difference:

# interface file auto-generated by buildroot

auto lo
iface lo inet loopback

#auto eth0
#iface eth0 inet dhcp
#  pre-up /etc/network/nfs_check
#  wait-delay 15

The "nfs_check" script in /etc/network sounds like it has something to do with stopping DHCP from interfering when the kernel is already using the interface for NFS, but before I disabled the eth0 settings in /etc/network/interfaces, I added a debug message to nfs_check, and it was never called.

I feel like this is very close because it DOES work once the timeout eventually finishes.

Any ideas? Where is the configuration or code which tells it to attempt to bring up eth0 and eth1?

Nb: Here are env variables from u-boot (trimmed):

boot_fdt=try
bootcmd=run findfdt;mmc dev ${mmcdev};mmc dev ${mmcdev}; if mmc rescan; then if run loadbootscript; then run bootscript; else if run loadimage
; then run mmcboot; else run netboot; fi; fi; else run netboot; fi
eth1addr=00:04:9f:04:f7:dc
ethact=FEC1
ethaddr=00:04:9f:04:f7:dd
ethprime=FEC
image=zImage
initrd_addr=0x83800000
initrd_high=0xffffffff
loadaddr=0x80800000
loadbootscript=fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} ${script};
loadfdt=fatload mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${fdt_file}
loadimage=fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} ${image}
mfgtool_args=setenv bootargs console=${console},${baudrate} rdinit=/linuxrc g_mass_storage.stall=0 g_mass_storage.removable=1 g_mass_storage.f
ile=/fat g_mass_storage.ro=1 g_mass_storage.idVendor=0x066F g_mass_storage.idProduct=0x37FF g_mass_storage.iSerialNumber="" clk_ignore_unused
netargs=setenv bootargs console=${console},${baudrate} ${smp} root=/dev/nfs ip=all nfsroot=${serverip}:${nfsroot},v3,tcp
netboot=echo Booting from net ...; run netargs; if test ${ip_dyn} = yes; then setenv get_cmd dhcp; else setenv get_cmd tftp; fi; ${get_cmd} ${
image}; if test ${boot_fdt} = yes || test ${boot_fdt} = try; then if ${get_cmd} ${fdt_addr} ${fdt_file}; then bootz ${loadaddr} - ${fdt_addr};
 else if test ${boot_fdt} = try; then bootz; else echo WARN: Cannot load the DT; fi; fi; else bootz; fi;
nfsroot=/home/matt/imx6/rootfs
serverip=10.113.130.94

[jeremy_cole]

Problem Solved as follows:

1/ Removed unused Ethernet from device tree:

I edited imx6ull-14x14-evk.dts (in build/linux-rel_imx_4.1.15_2.0.0_ga/arch/arm/boot/dts/), found the "&fec1" section, and set the "status" to "disabled", i.e.:

&fec1 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_enet1>;
    phy-mode = "rmii";
    phy-handle = <&ethphy0>;
    status = "disabled";
};

I didn't edit &fec2. Note that logically you would expect to disable fec2, but I tried various combinations and the above worked. Disabling fec2 confused the kernel; it looks like u-boot is using fec2, but the kernel tried to use fec1 as "eth0".

I created a patch from this and added it to my patches to make the change persistent.

2/ Fixed u-boot "netargs" env value:

The "ip" parameter in "netargs" must be "ip=dhcp" when using DHCP along with NFS rootfs. E.g.:

 setenv netargs 'setenv bootargs console=${console},${baudrate} ${smp} root=/dev/nfs ip=dhcp nfsroot=${serverip}:${nfsroot},v3,tcp'

3/ Hacked resolv.conf:

The above works well, apart from one problem: DNS doesn't work for some reason. /etc/resolv.conf points to /tmp/resolv.conf, but this is empty.

To fix, I grabbed name servers which get set in /proc/net/pnp:

cat /proc/net/pnp | grep nameserver > /tmp/resolv.conf

I added this to a startup script. This is a bit hacky, but I am only using NFS rootfs for dev.