iMX7D Yocto with OP-TEE

[medaliyousfi]

Hey, I have a Yocto project build as following:

Spoiler

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-20.04"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "imx7dsabresd"
DISTRO = "fsl-imx-fb"
DISTRO_VERSION = "5.4-zeus"
TUNE_FEATURES = "arm vfp cortexa7 neon thumb callconvention-hard"
TARGET_FPU = "hard"

with OP-TEE enabled. I can successfully build my custom image for my custom hardware.

When i flash the image using uuu utility and boot the system with the following order boot

ROM -> u-boot -> OP-TEE -> Linux Kernel

OP-TEE starts with no (E/TC) debug message and loads my kernel and i can login and manipulate the system correctly.

The issue is as soon as i do a restart or a shutdown (no matter how , even a physical power off), and when i try to reboot my system, OP-TEE starts again and loads my kernel, but the Kernel panics and no longer able to mount the rootfs from mmcblk2p2.

Even from u-boot I no longer able to 'ls mmc 2' or 'ext4ls' the rootfs partition as if it was somehow corrupted.

Remember this only occurs after a first successful boot.

Somehow there is something corrupting my rootfs or changing mmc parts and causing linux to panic.

Also after a successful boot, when i issue the command 'xtest' I get an error as follows:

Spoiler

root@imx7dsabresdv4 xtest Run test suite with level=0E/TC:0 0 std_smc_entry:186 Bad arg address 0xb89ac000
TEE test application started over default TEE instance ###################################################### # regression+regression_nxp ######################################################
* regression_1001 Core self tests /usr/src/debug/optee—test/3.10.0.imx—r0/git/host/xtest/regression_1000.c:241: res has an unexpected value: Oxffff 000e = TEEC_ERROR_COMMUNICATION, expected 0x0 = TEEC_SUCCESS Segmentation fault 

Also note that we use a custom memory layout and we reserve A7 Core 0 for linux and A7 Core 1 for a custom application, the following device tree snippet should illustrate the layout.

Spoiler

/ {
memory {
//linux,usable-memory = <0x80000000 0x1ff00000>,
// <0xa0000000 0x1ff00000>;
reg = <0x80200000 0x3fd00000>;
linux,usable-memory = <0x80200000 0x3fd00000>;


};

m4_tcm: tcml@7f8000 {
compatible = "fsl, m4_tcml";
reg = <0x7f8000 0x8000>;
};

reserved-memory {
#address-cells = <1>;
#size-cells = <1>;
ranges;

m4_0: m4_app@80000000 {
reg = <0x80000000 0x00100000>;
};

rpmsg_mem: rpmsg_mem_reg@bfff0000 {
no-map;
reg = <0xbfff0000 0x00010000>;
};

a7_1: a7_1_app@90000000 {
reg = <0x90000000 0x10000000>;
};

a7_1_2: a7_1_2_app@A0000000 {
reg = <0xA0000000 0x10000000>;
};

};

[Sanket_Parekh]

Hi @medaliyousfi 

I hope you are doing well.

Please make sure you have followed the steps mentioned in 5 Configuring OP-TEE in i.MX Porting Guide for custom board and DRAM.

* regression_1001 Core self tests /usr/src/debug/optee—test/3.10.0.imx—r0/git/host/xtest/regression_1000.c:241: res has an unexpected value: Oxffff 000e = TEEC_ERROR_COMMUNICATION, expected 0x0 = TEEC_SUCCESS Segmentation fault 

Above error could be due to memory conflict between OP-TEE and kernel.

Please define the memory node as below :

	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x80000000>;
	};

You can reserve memory for applications in reserved-memory node.

Did you change the kernel load address?

Please mention the size of the DRAM.

Please provide me with boot logs and reboot logs when the kernel crashes for further debugging.

Thanks & Regards,

Sanket Parekh

[medaliyousfi]

Hello ,

I managed to get the xtest working by disabling the dynamic shared memory configuration,
my optee-os bbappend is as follows:

EXTRA_OEMAKE += " \
  CFG_CORE_DYN_SHM=n \
  CFG_RPMB_FS=y \
  CFG_RPMB_FS_DEV_ID=2 \
  CFG_RPMB_WRITE_KEY=y \
  CFG_CORE_DEBUG_CHECK_STACKS=y \
  CFG_TEE_CORE_DEBUG=y \ 
  CFG_TEE_CORE_MALLOC_DEBUG=y \
  CFG_TEE_CORE_LOG_LEVEL=4 \
  CFG_TEE_TA_LOG_LEVEL=4 \
  DEBUG=y \
"

• Defining the memory node as you have mentioned has no impact on the system & didn't help, my DRAM size is 1GB, with 1st 1MB reserved for CM4, the next 255MB for Linux, then next 256 MB for CA7_Core1 APP, the next for CA7_Core1 APP DATA, the resting 256MB for Linux.
• I followed the exact steps mentioned in the iMX Porting Guide .
• The load address and memory layout is as follows:

load address    = 0x80800000
fdt address     = 0x83000000
uTee address    = 0x84000000
custom_A7_1 App = 0x89000000​

boot sequence is as follows:

OPTEE Disabled:

Boot ROM -> u-Boot [Load FDT, Load A7 App, Load CM4 FW, bootz] -> Linux Kernel

OPTEE Enabled:

Boot ROM -> u-Boot -> [Load FDT, Load A7 App, Load CM4 FW, bootz] -> OPTEE -> Linux Kernel

Now the issues are :

1. I have a new issue, when loading Cortex M4 , I am no longer able to boot to Kernel
   Without OPTEE : the kernel hangs at "Starting Kernel ..."
   With OPTEE : OPTEE hangs at "Switching to Normal world ..."
   My CM4 is linked against OCRAM EPDC (0x920000) and loaded to that address (0x920000), but i think OCRAM is being used by OPTEE.
   PS: The same application is working fine on kernel 4.19 (BSP Warrior) & without OPTEE.
2. Xtests are failing, the logs are in the attachement.
   
   

[Sanket_Parekh]

Hi @medaliyousfi 

I hope you are doing well.

Please make sure you have modified CFG_DDR_SIZE in conf.mk in op-tee.

Please refer to arch/arm/boot/dts/imx7d-sdb-m4.dtsi to reserve memory and peripherals for M4 core.

Please reserve 1MB at 0x9ff00000 for M4 core as shown in above dts file,As Linker files maps the same memory as in dtsi. 

PATH : FreeRTOS_BSP_1.0.1_iMX7D/platform/devices/MCIMX7D/linker/gcc/

MEMORY
{
  m_interrupts          (RX)  : ORIGIN = 0x9ff00000, LENGTH = 0x00000240
  m_text                (RX)  : ORIGIN = 0x9ff00240, LENGTH = 0x00007DC0
  m_data                (RW)  : ORIGIN = 0x20000000, LENGTH = 0x00008000
}

One should use OCRAM_S instead of OCRAM_EPDc as suggested in AN5317.

Please share full bootlogs when the kernel hangs at Starting kernel... 

Have you made any other changes?

Thanks & Regards,

Sanket Parekh

[medaliyousfi]

Hey @Sanket_Parekh , I hope you are doing well too, and Thanks for your reply !

Well, after rechecking my kernel patches & device trees (I am porting from 4.19 Warrior to 5.4 Zeus) I have missed patching a DT node and that was the cause of the hang at Starting Kernel.

After that everything works correctly and according to the Porting Guide on Chapter 5.5.1

OCRAM_S will be used by OP-TEE to save PM data, that's why I left CM4 linked against OCRAM_EPDC 0x920000 and above.

Now issues remains with OP-TEE, I still have failed Tests (check the attachment logs).
I had to disable RPMB because it was causing most of the tests to fail.
CFG_DDR_SIZE is defined correctly, please find attached the conf.mk file.

If you see any bad/wrong configuration statement within the conf.mk please correct me.

[Sanket_Parekh]

Hi  @medaliyousfi 

I hope you are doing well.

Please keep the CFG_CORE_DYN_SHM value and RPMB as default and try using the Latest BSP.

Please load the M4 binary in TCM.

Thanks & Regards,

Sanket Parekh