iMX8 MacOS/Apple M1 LLVM/clang cross-compile toolchain

[jaredwheeler]

Hello, I am bringing up a custom SoM based on the IMX8. This work includes standing up a cross-compile toolchain, spec'd thus:

HOST: MacOS/Apple M1(Arm64)
CC: clang/LLVM

The toolchain is designed to support a "Digital Nomad" workflow. This carries a couple requirements:

1. Must function efficiently on battery
2. Must support edit/compile/load/debug workflow in network-constrained environment
3. Portability - minimize wiring/physical footprint

I have enough of the above working at this point to feel like it's possible. Sending up a flare to gauge community experience/interest. Anyone else out there working along similar lines? Links to active discussion threads/forum sections would be welcome.

I'll also note that I did explore the imx-yocto project. I quickly found myself searching for something less fussy.

Cheers!

[jaredwheeler · Accepted solution]

Great to hear there's interest!

I broke to the problem space into two pieces:

1. Cross-compile and load bootloader/kernel/rootfs
2. Edit/compile/load/debug workflow for user-space executable
   • IDE GUI integration (targeting vscode)
   • LLDB breakpoint debugging

Here's where I'm at with the above:

1. Cross-compile and load bootloader/kernel/rootfs

	cross-compile (in macOS)	create image (in macOS)	uuu load to target (in macOS)
u-boot		Y	Y
linux kernel	Y**	Y	Y
rootfs	***	Y	Y

**takes a number of sysroot hacks, but I currently have the kernel 6.0 'make' compiling to --target=aarch64-linux-gnu from an M1 MBP host. Happy to share details on this and maybe clean it up a bit.

*** Currently using buildroot to generate the rootfs image.  Able to subsequently modify the image in macos, then reload via UUU.

2. Edit/compile/load/debug workflow for user-space executable
This is still theoretical. Aiming to:

• Cross-compile for aarch64-linux-gnu via clang
• Load executable to target over USB
• Open LLDB session over the USB interface
• Set up LLDB breakpoint debugging in VSCode via extensions

That's the idea. Basically trying to bring the workflow as close as possible to existing iOS/Android mobile development systems.

Shout if there's interest in trying/further developing any of the above.

[KubaBE]

Heya, how exactly are you loading your kernel image and rootfs with uuu? I've been struggling with the emmc_burn_all script getting stuck on a jump instruction after doing the bootloader and I was wondering it's related to me using a mac.

[jaredwheeler]

Glad to have some company down here in this rabbit hole. I'm using a bash script in MacOS to do the following uuu invocation:

#!/bin/bash
sc_root=$(pwd)
bootloader_image=${sc_root}/build/images/bootloader.bin
emmc_image=${sc_root}/build/images/emmc_img
sudo uuu -b emmc_all ${bootloader_image} ${emmc_image}

Notes:

• bootloader.bin is the output from imx-uboot sources, compiled in Linux (Ubuntu running inside qemu running inside UTM).
• emmc_image contains the bootable Linux partition and the Ext4 rootfs partition -- also compiled/built in Linux
• uuu is compiled from source and invoked in MacOS on an Apple M1 (Arm64).
• uuu is loading to MMC device 2 -- an eMMC soldered to the SoM board

It took a bit of fiddling, but the above eventually worked and remains pretty reliable. I'm currently bringing up a BSP + kernel drivers for the application board, so this script has seen heavy use.

Questions:
1. Are you trying this against a custom SoM, or are you using an EVK?
2. How are you troubleshooting the connection?
3. Does uuu -list show the target as expected?
4. I found the u-boot console helpful when trying to rule out connection/functional issues with MMC devices -- are you able to connect a terminal emulator to the i.MX8M's debug UART?

DM if it's helpful to do a screen share session. I'd be happy to walk thru my setup and show what I've been able to get working so far.

Also, I'm in the process of bringing up a USB driver on MacOS and it's 8M counterpart. I'll update the accepted answer once that's working. The idea is to expose 3 USB endpoints:

• tty console
• lldb/gdb serial transport
  • allows for app loading/breakpoint debugging from MacOS host (VSCode)
• eMMC partition mounted in macos
  • allows MacOS to transfer files directly to eMMC on SoM.

Hoping to eliminate multiple USB UART cables and do everything over a single USB port on the target device. Let me know if you have any experience with this stuff.

[Jasen]

Hi,
1. have you try cross-compile on macos via Docker? I tried x86 ubuntu emulation(Ubuntu running inside qemu running inside UTM) , unfortunately it has bad performance.
2. Could you share how to compile uuu from sorce for macos?

[jaredwheeler]

1. I haven't tried Docker, but I totally agree that the UTM/qemu/Ubuntu turducken isn't optimal.  If you do get a kernel/rootfs/bootloader compilation toolchain running in a Docker instance, PLEASE SHARE!  

Maybe start a new thread with a writeup of how you did it?  Battery-friendly macOS bringup for 8Ms could use some attention.

2. Im using a clone of the NXP mfgtools repo to build uuu.  

https://github.com/nxp-imx/mfgtools 

The macOS build instructions in the repo readme work on my machine with a fresh clone (tested just now). I may have had some weird issues with the openSSL dependency on my M1 machine if I recall correctly...  It was a while ago, so whatever I may have done is lost down the memory hole.

[Jasen]

Thank you, i will try Docker and compiling uuu on MacOS, and will feedback it here if i have some good news.

[jaredwheeler]

Hello!  Quick clarifying question: Are you on an ARM (M1/Apple Silicon) MacOS machine, or an Intel/x86 machine?

I find the compilation flow is super different between the two.  Apple ARM ABI support in gcc thread is here... https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96168

[Jasen]

Apple Silicon M2 machine

[dvlogic]

I would be interested to learn more of what you are doing.  I'm currently developing for a custom i.MX8 on a M1 mac, but building with a imx-yocto setup running in a Parallels Ubuntu Arm Linux VM.  

[Bio_TICFSL]

Hello,

You are working on MX8, but what MX8QM or MX8MQ? I guess this should be for MX8QM.

regards

[jaredwheeler]

I’ll note that the above solution targets the IMX8MN.  Currently generating a boot loader/mainline kernel/rootfs which exposes a login/shell on uart2.

[jaredwheeler]

As an update, I have a login shell exposed over USB.

Now running as a single wire-- serial+power over a USB (A and C both working) port.  It's reliable enough that I don't need the UART port unless something goes seriously wrong.  Daily development on the platform only needs the USB cable.

It uses the Linux USB Serial Gadget stuff to expose a serial device (also a parallel Mass Storage device).

USB Serial Gadget was funky owing to its dependencies on most the rest of the kernel booting up successfully-- I WOULD NOT attempt to do this without first bringing up a UART serial port unless you're more experienced with kernel configuration than I am.  

The USB serial connection comes late in the kernel boot process so you'll just get a silent failure to connect if anything goes wrong before kernel boot makes it to the USB stack.  

The UART port gives you feedback starting from the boot loader, so it's super helpful to debug all the things that go wrong during a bootloader/kernel/rootfs bringup on a custom board.

If anyone else has experience doing something similar, would love to hear stories.