OPTEE with Android on IMX6 quad

[saidgani_musaev]

Hello everyone, I am using Nitrogen6x board from boundarydevice. It's based on IMX6Q chip. I am working on studying project related to Trusted execution environment. I could boot OPTEE with Linux kernel from linaro repo. When I try to boot Android with OPTEE it is stuck. I see logs from OPTEE itself: the last log message is "Primary CPU switching to normal world boot". I got android sources from boundary device private repo. Without OPTEE Android boots correctly. With the same OPTEE kernel,  Linux kernel boots correctly as well( all client tests pass ). I recompiled Android kernel with CONFIG_EARLY_PRINTK and CONFIG_EARLY_PRINTK_DIRECT. Also I disabled SMP to not boot secondary CPUs.  I don't see any message after OPTEE executes "smc #0" for first time. I assume Android booting code does do something which is forbidden in normal world. I wrote to Boundary Device support (Android Getting Started Guide - Boundary Devices ), but they directed me here :smileyhappy:. What could be a good starting point for me to get deep and to port Android to my board? As I understand this Android distribution will not work with "Trusty"-TEE, right?  

[chenlifu2015]

Hi,Saidgani Musaev 

   I am also trying to load Android img with OP-TEE,but how do you boot  tee.bin and boot.img, use boota cmd or bootm? can you tell me the environment in  U-Boot ? I have been blocked by this problem, Thanks 

[saidgani_musaev]

Hello, yes, like bootm 0x20000000 ..., but you miss ramdisk. I have attached my script here.

# ==================== script ================================

setenv bootpart 1

setexpr rval *0x020CC068 \& 0x180
if itest.s "x$rval" -eq "x100"; then
    echo "Starting fastboot..."
    fastboot 0
else
    if itest.s "x$rval" -eq "x80"; then
        setenv bootpart 2;
    fi
fi
mw.l 0x020cc068 0 1

setenv bootargs ''

m4=''
a_base=0x10000000
if itest.s x51 == "x${cpu}" ; then
    a_base=0x90000000
elif itest.s x53 == "x${cpu}"; then
    a_base=0x70000000
elif itest.s x6SX == "x${cpu}" || itest.s x7D == "x${cpu}"; then
    a_base=0x80000000
    if itest.s "x1" == "x$m4enabled" ; then
        run m4boot;
        m4='-m4';
    fi
fi

setexpr a_script  0x10000000
setexpr a_zImage  0x12000000
setexpr a_fdt     0x18000000
setexpr a_ramdisk 0x13800000

if itest.s "x" == "x${board}" ; then
    echo "!!!! Error: Your u-boot is outdated. Please upgrade.";
    exit;
fi

if itest.s "x" == "x${dtbname}" ; then
    if itest.s x6SOLO == "x${cpu}" ; then
        dtbname=imx6dl-${board}.dtb;
    elif itest.s x6DL == "x${cpu}" ; then
        dtbname=imx6dl-${board}.dtb;
    elif itest.s x6QP == "x${cpu}" ; then
        dtbname=imx6qp-${board}.dtb;
    elif itest.s x6SX == "x${cpu}" ; then
        dtbname=imx6sx-${board}${m4}.dtb;
    elif itest.s x7D == "x${cpu}" ; then
        dtbname=imx7d-${board}${m4}.dtb;
    elif itest.s x51 == "x${cpu}" ; then
        dtbname=imx51-${board}${m4}.dtb;
    elif itest.s x53 == "x${cpu}" ; then
        dtbname=imx53-${board}${m4}.dtb;
    else
        dtbname=imx6q-${board}.dtb;
    fi
fi

if load ${dtype} ${disk}:${bootpart} ${a_script} uEnv.txt ; then
    env import -t ${a_script} ${filesize}
fi

if load ${dtype} ${disk}:${bootpart} ${a_fdt} ${bootdir}${dtbname} ; then
    fdt addr ${a_fdt}
    setenv fdt_high 0xffffffff
else
    echo "!!!! Error loading ${bootdir}${dtbname}";
    exit;
fi

cmd_xxx_present=
fdt resize
if itest.s "x" != "x${cmd_custom}" ; then
    run cmd_custom
    cmd_xxx_present=1;
fi

# ------------------- Displays detection
if itest.s "x" != "x${cmd_hdmi}" ; then
    run cmd_hdmi
    cmd_xxx_present=1;
    if itest.s x == x${allow_noncea} ; then
        setenv bootargs ${bootargs} mxc_hdmi.only_cea=1;
        echo "only CEA modes allowed on HDMI port";
    else
        setenv bootargs ${bootargs} mxc_hdmi.only_cea=0;
        echo "non-CEA modes allowed on HDMI, audio may be affected";
    fi
fi
if itest.s "x" != "x${cmd_lcd}" ; then
    run cmd_lcd
    cmd_xxx_present=1;
    setenv bootargs $bootargs tsc2004.calibration=-67247,-764,272499173,324,69283,-8653010,65536
fi
if itest.s "x" != "x${cmd_lvds}" ; then
    run cmd_lvds
    cmd_xxx_present=1;
fi
if itest.s "x" != "x${cmd_lvds2}" ; then
    run cmd_lvds2
    cmd_xxx_present=1;
fi
if itest.s "x" == "x${cmd_xxx_present}" ; then
    echo "!!!!!!!!!!!!!!!!"
    echo "warning: your u-boot may be outdated, please upgrade"
    echo "!!!!!!!!!!!!!!!!"
fi

setenv bootargs "$bootargs console=${console},115200 vmalloc=128M consoleblank=0 androidboot.selinux=permissive androidboot.selinux=enforcing rootwait"
setenv bootargs $bootargs androidboot.hardware=freescale

if itest.s "xsata" == "x$dtype" ; then
    bootdev=sda
elif itest.s "xusb" == "x$dtype" ; then
    bootdev=sda
else
    bootdev=mmcblk${disk}
fi
setenv bootargs $bootargs androidboot.bootdev=$bootdev

if itest.s "x" != "x$selinux" ; then
    setenv bootargs $bootargs androidboot.selinux=$selinux
fi

# add serial number based on MAC address
setexpr mac_hi *0x021bc630
setexpr mac_lo *0x021bc620
setenv bootargs $bootargs androidboot.serialno="$mac_hi$mac_lo"

if itest.s "x" != "x$hwrotation" ; then
    setenv bootargs $bootargs androidboot.hwrotation=$hwrotation
fi

if itest.s "x" != "x$disable_giga" ; then
        setenv bootargs $bootargs fec.disable_giga=1
fi

if itest.s "x" != "x$wlmac" ; then
    setenv bootargs $bootargs wlcore.mac=$wlmac
    setenv bootargs $bootargs androidboot.wlan.mac=$wlmac
    # create a bt mac address from the wlan one
    setexpr btmac sub "(..:..:..:).(.*)" "\\1b\\2" ${wlmac}
    setenv bootargs $bootargs androidboot.btmacaddr=${btmac}
fi

if itest.s "x" == "x$cma" ; then
    cma=448M
fi
setenv bootargs $bootargs cma=$cma

if itest.s "x" != "x$gpumem" ; then
    setenv bootargs $bootargs galcore.contiguousSize=$gpumem
fi

if itest.s "x" != "x$show_fdt" ; then
    fdt print /
fi

if itest.s "x" != "x$show_env" ; then
    printenv
fi

tftp 0x20000000 uTee

if ${fs}load ${dtype} ${disk}:${bootpart} ${a_zImage} zImage ; then
    if ${fs}load ${dtype} ${disk}:${bootpart} ${a_ramdisk} uramdisk.img ; then
        bootm 0x20000000 ${a_ramdisk} ${a_fdt} ;
    fi
fi

echo "Error loading the OS"

[chenlifu2015]

Hi，Saidgani Musaev

   Thanks for your apply，I just load optee on IMX6Q by referencing your idea，I just modify the function ：do_boota，because do_boota finally used function :do_bootm，so I fixed the args for the do_bootm like:"sprintf(boot_addr_start, "0x%lx", tee_entry_addr);" (tee_entry_addr is 0x4E000000).In this way，I just load boot.img and the use boota cmd，then，uboot will load the optee correctly，but I also stuck in the log：“Primary CPU switching to normal world boot”， and I think the question is not about android code，I think the reason is we changed the process of loading kernel(like analysis zImge or boot.img)，so when optee has been load and return to normal world for loading kernel，can not get the args(like length load address entry address and bootargs) for loading kernel 

[saidgani_musaev]

I don't think so.

Here is code from core/arch/arm/kernel/geberic_entry_a32.S:

/*
      * Save boot arguments
      * entry r0, saved r4: pagestore
      * entry r1, saved r7: (ARMv7 standard bootarg #1)
      * entry r2, saved r6: device tree address, (ARMv7 standard bootarg #2)
      * entry lr, saved r5: non-secure entry address (ARMv7 bootarg #0)
      */
     .macro bootargs_entry
#if defined(CFG_NS_ENTRY_ADDR)
     ldr     r5, =CFG_NS_ENTRY_ADDR
#else
     mov     r5, lr
#endif
#if defined(CFG_PAGEABLE_ADDR)
     ldr     r4, =CFG_PAGEABLE_ADDR
#else
     mov     r4, r0
#endif
#if defined(CFG_DT_ADDR)
     ldr     r6, =CFG_DT_ADDR
#else
     mov     r6, r2
#endif
     mov     r7, r1
     .endm
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So as you see bootargs are being saved for normal world. Moreover - I could boot Android with OPTEE. But I have weird problem with Android itself - it remounts filesystem, restart services and then reboots itself. I used Android kernel from linaro for hikey( which is ported for using with OPTEE).  (GitHub - linaro-swg/optee_android_manifest: Android manifest for building OP-TEE in AOSP )

[chenlifu2015]

Hi,

   It is my bad,but I still think the problem is not about the Android code,maybe we should read the code of the bootloader(u-boot) more carefully，or OP-TEE may not be load correctly（I supprt the SMP，the Second Cpu init will print another log，but I do not see it）because in this case，system.img  is load behind boot.img, when we resolve this problem,system.img maybe load.

[saidgani_musaev]

Hi, I could boot everything with different version of the kernel. But currently I have another problem - healthd daemon and some other services don't work. I am still looking for different configurations of AOSP( but it takes a lot of time to download it,  build it, reconfigure it). But at least I could boot it. The main problem with current build is that I can't enable adb shell. And I can't get root access through serial console. So, as far as different Kernel works ( with absolutely the same environment, filesystem, init scripts etc.) - I think the problem is in Kernel(boot) code :smileyhappy: .

P.S. You can find link to the working kernel in my previous message.

[chenlifu2015]

Hi，I'm sorry to bother you，but I have tried so many versions of kernel, but I still can not boot optee and boot.img correctly,so,can you tell me the link to the working kernel ,and the version you use, Thanks a lot

[saidgani_musaev]

Hello , I am really sorry for "ignoring" your questions. I was busy recent days.  I would really love to help you with that. Could we move our discussion in other chat. Because of it's hard to discuss it here. You can find me on Facebook by my name. Or Send me a pm, please. Thank you.

[chenlifu2015]

Hi,Saidgani Musaev 

   I have send a message to you in facebook,my name is the same with the username in NXP: 陈礼夫，I enable the CONFIG_DEBUG_PRINTK and CONFIG_DEBUG_LL,I got more details about the problem: 

    Unhandled fault: imprecise external abort (0xc06) at 0x12402d8c
10:33:41.297: pgd = c0004000
10:33:41.301: [12402d8c] *pgd=00000000
10:33:41.301: Internal error: : c06 [#1] PREEMPT ARM
10:33:41.305: Modules linked in:
10:33:41.309: CPU: 0 PID: 0 Comm: swapper Tainted: G W 4.1.15-dirty #1
10:33:41.313: Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
10:33:41.318: task: c110a3f8 ti: c1100000 task.ti: c1100000
10:33:41.323: PC is at find_static_vm_vaddr+0x18/0x6c
10:33:41.326: LR is at __iounmap+0x14/0x50
10:33:41.329: pc : [<c01141d0>] lr : [<c011439c>] psr: 20000093
10:33:41.333: sp : c1101fa0 ip : 000000c0 fp : 00000000
10:33:41.338: r10: ee7572c0 r9 : c10846e0 r8 : c11f4000
10:33:41.342: r7 : ffffffff r6 : c1102000 r5 : c11f4000 r4 : f4a02000
10:33:41.347: r3 : eefffff8 r2 : eeffffd8 r1 : c110bf0c r0 : f4a02000
10:33:41.352: Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
10:33:41.359: Control: 10c57c7d Table: 10004059 DAC: 00000015

   so I am sure the the problem is in Kernel(boot) code,so can you tell me the right version for kernel code you use

Thanks a lot 

[chenlifu2015]

Hi,I have used the kernel with the link： GitHub - linaro-swg/optee_android_manifest: Android manifest for building OP-TEE in AOSP，but it does not work ，too.Should I download another version of the kernel,I have been drived mad by this question，I am looking forward to your apply，thanks for your help in this week。

[chenlifu2015]

Hi，Thanks for your share，I will use your kernel code to build boot.img and try again，The link to working kernel is GitHub - linaro-swg/optee_android_manifest: Android manifest for building OP-TEE in AOSP？Should I download another version of the kernel，because you said " I could boot everything with different version of the kernel",can you tell me the link to working OP-TEE and Android Code，I want to use the same code exactly like you，so I will rule out the  possibilities of wrong code。

[saidgani_musaev]

Hello,

Which chip do you use? In general it is similar to what you are doing for booting Linux Kernel. I modified 6x_bootscript which is located on boot partition on my boot flash( note that I am working with ready sdcard. I put link to instruction in my question). In this script I have changed addresses of zImage, fdt, ramdisk and script. It really depends on how did you compile OPTEE( in my case FDT is supposed to be on address 0x18000000, NS_Entry_addr is 0x12000000 ), so in my case I used these addresses:

setexpr a_script  0x10000000
setexpr a_zImage  0x12000000
setexpr a_fdt     0x18000000
setexpr a_ramdisk 0x13800000

just before booting Android Kernel I loaded tee.bin (I used address 0x20000000 for this purpose), and then I just changed bootz {a_zImage}... to bootm 0x20000000. I hope it helped, Please let me know about your progress and which board do you use. Anyway I am going to put step by step guide to boot it on for my board. Thank you.

[chenlifu2015]

Hi，Saidgani Musaev

   Thanks for your apply，the type of board which I am using is IMX6Q，the method of booting the IMX6Q board is by sdcard，the address of FDT and NS_ENTRY are the same with your board.I do not understand the meaning of "then I just changed bootz {a_zImage}... to bootm 0x20000000", like bootm 0x20000000 - 0x18000000 ? and I can not find your 6x_bootscript  too, can you just copy and paste your bootscript in the answer,thank you very mach!