example application from sdk i.e. ethosu_apps is not running as expected

[phanichinimilli]

I'm trying to following instructions (from section 6.2.7 ) on building Cortex-M33 firmware from the SDK downloaded.
using GNU arm toolchain 13.x , I'm able to compile the demo application
ethosu_apps
But from uboot when I ran it, I'm seeing error logs in COM/serial port of Cortex M33

==================
Initialize Arm Ethos-U
E: NPU error(s) occured during inference. (ethosu_driver.c:542)
E: NPU status=0x00000804, qread=0, cmd_end_reached=0 (ethosu_device_u55_u65.c:182)
Node ethos-u (number 0) failed to invoke with status 1
E: Invoke failed for inference: job=job (inference_process.cpp:281)
Inference status: failed
==================
I expected it to work as is, can someone help me here ?

[Chavira]

HI @phanichinimilli!

Thank you for contacting NXP Support!

I tried with the iMX93-QSB and I didn't see errors.

I compiled the example using MCUXPRESSO for VS CODE the version of the SDK that I am using is 2.16.

Best Regards!

Chavira

[phanichinimilli]

Thanks for the response.
I'm using sdk of imx93_evk (not imx93_qsb), Im not sure if it makes any difference.
>tftp 0x80000000 ethosu_apps.bin
>cp.b 0x80000000 0x201e0000 0x20000
>bootaux 0x1ffe0000 0
Do I have set any env variables in uboot ? Please let me know if I'm missing any step.
P.S: 
The ethosu_apps_rpmsg application from sdk is not behaving, similar to ethosu_firmware

[Chavira]

HI @phanichinimilli!

I was thinking that you have the problem on iMX93-QSB board.

I tried with the iMX93-EVK and the ethosu_apps_rpmsg is working as expected but the ethosu_apps have an issue in iMX93-EVK.

I will report this issue to our internal team and I will update this community post as soon as possible with the solution of the apps  team.

Best Regards!

Chavira

[phanichinimilli]

Chavira, Thanks for confirmation. on ethosu_apps
A small clarification is needed on ethosu_apps_rpmsg, can you please confirm at your side?
What I have done ?
1. using 2.16 sdk build ethosu_apps_rpmsg application
2. boot linux, replace ethosu_firmware with elf of #1
3. run default inference test on ethosu from linux

./label_image_release -m output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l lab
INFO: Loaded model output/mobilenet_v1_1.0_224_quant_vela.tflite
INFO: resolved reporter
INFO: before CreateAllDelegates
INFO: Ethosu delegate: device_name set to /dev/ethosu0.
INFO: Ethosu delegate: cache_file_path set to .
INFO: Ethosu delegate: timeout set to 60000000000.
INFO: Ethosu delegate: enable_cycle_counter set to 0.
INFO: Ethosu delegate: enable_profiling set to 0.
INFO: Ethosu delegate: profiling_buffer_size set to 2048.
INFO: Ethosu delegate: pmu_event0 set to 0.
INFO: Ethosu delegate: pmu_event1 set to 0.
INFO: Ethosu delegate: pmu_event2 set to 0.
INFO: Ethosu delegate: pmu_event3 set to 0.
INFO: EXTERNAL delegate created.
INFO: EthosuDelegate: 1 nodes delegated out of 1 nodes with 1 partitions.
INFO: Applied EXTERNAL delegate.
INFO: settings->number_of_warmup_runs 2
ERROR: Ethos_u inference failed

ERROR: Node number 1 (EthosuDelegate) failed to invoke.
ERROR: Failed to invoke tflite!
4. observe logs in M33 com-port
I can see logs are same , when compared to default ethosu_firmware executable

P.S:
while inference_runner command works fine with my ethosu_firmware
./inference_runner -d /dev/ethosu0 -n output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt -o output_data_detection -t 1080000000

[Chavira]

Hi @phanichinimilli!

I got the next response from the apps team:

I can reproduce identical ethosu_apps_rpmsg.bin startup success from U-Boot (I assume because the rpmsg app doesn't attempt any inferences) and the same ethosu_apps.bin failure on SDK 2.16.000, gcc-arm-eabi-none-10-2020-q4-major (as noted in UG10166) and BSP LF6.6.52-2.2.0. I launched it from U-Boot as follows (the bootaux 0x201e0000 0 address in UG10166 is suspect):

u-boot=> fatload mmc 1:1 ${loadaddr} ethosu_apps.bin
49800 bytes read in 3 ms (15.8 MiB/s)
u-boot=> cp.b ${loadaddr} 0x201e0000 ${filesize}
u-boot=> bootaux 0x1ffe0000 0
## Starting auxiliary core addr = 0x1FFE0000...
u-boot=> █

Just as you observed on the Cortex-M33 console:

Initialize Arm Ethos-U
E: NPU error(s) occured during inference. (ethosu_driver.c:542)
E: NPU status=0x00000804, qread=0, cmd_end_reached=0 (ethosu_device_u55_u65.c:182)
Node ethos-u (number 0) failed to invoke with status 1
E: Invoke failed for inference: job=job (inference_process.cpp:281)
Inference status: failed

However, I find some of the steps in UG10166 a bit confusing. Lack of familiarity isn't helping matters. E.g., there is no middleware/ethos-u-core-software/examples directory. E.g., .out vs. .bin.

Repeating your steps:

1. using 2.16 sdk build ethosu_apps_rpmsg application
2. boot linux, replace ethosu_firmware with elf of #1

Yikes…parameters would make this software less fiddly. Before replacing anything in the imx-image-full YP build it behaved as demonstratred in §2.7. Then I replaced ethosu_firmware as in steps 1-2:

$ cd /lib/firmware
$ mv ethosu_firmware ethosu_firmware.orig
$ ln -s ethosu_apps_rpmsg.elf ethosu_firmware

Note: Booting Linux with the rpmsg device tree: setenv fdtfile imx93-11x11-evk-rpmsg.dtb; boot.

3. run default inference test on ethosu from linux

./label_image_release -m output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l lab

I was stymied for a bit finding label_image_release in UG10166. (Replacing the obnoxiously long default shell prompt with $␣ in my examples for readability.)

Then returning to run label_image again…success. Just to be certain, I rebooted and power cycled and tried again.

$ cd /usr/bin/tensorflow-lite-2.16.2/examples/
$ ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt
INFO: Loaded model mobilenet_v1_1.0_224_quant.tflite
INFO: resolved reporter
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: invoked
INFO: average time: 153.342 ms
INFO: 0.768627: 653 military uniform
INFO: 0.105882: 907 Windsor tie
INFO: 0.0196078: 458 bow tie
INFO: 0.0117647: 466 bulletproof vest
INFO: 0.00784314: 835 suit

4. observe logs in M33 com-port

No output. Was the Linux label_image program supposed to do something with ethosu_firmware? I don't see it or the sysfs paths referenced in a dump of the program's string table. E.g.,

$ strings label_image | grep remoteproc                     
$ strings label_image | grep ethosu
$ █

So I'm not sure what steps 1 & 2 have to do with label_image. The CM33 isn't running. I saw nothing in the CM33 serial console. Remoteproc reports offline. So I tried running my symlinked ethosu_firmware from step 2 above:

$ cd /sys/devices/platform/remoteproc-cm33/remoteproc/remoteproc0
$ cat state
offline
$ cat firmware
rproc-imx-rproc-fw
$ echo /lib/firmware/ethosu_firmware > firmware
$ cat firmware
/lib/firmware/ethosu_firmware
$ echo start > state

I observed the expected Initialize Arm Ethos-U on the CM33 serial console. Re-running step 3 gave the same results as above with nothing more printer on the CM33 serial console — it's not interacting with ethosu_firmware.

Where am I out of sync with you? I've tried to be copy-paste exact in my procedure.

P.S: while inference_runner command works fine with my ethosu_firmware ./inference_runner -d /dev/ethosu0 -n output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt -o output_data_detection -t 1080000000

Did your postscript get cut off while you were composing this message?

FWIW, my attempt to run ethosu_apps.elf from remoteproc failed:

$ echo /lib/firmware/ethosu_apps.elf >firmware 
$ cat state
offline
$ echo start >state
-bash: echo: write error: Invalid argument
$ cat state
offline
$ dmesg | tail
[   61.593023] audit: type=1327 audit(1709089028.676:26): proctitle="(systemd)"
[   61.593650] audit: type=1334 audit(1709089028.676:27): prog-id=22 op=LOAD
[  252.292643] remoteproc remoteproc0: powering up imx-rproc
[  252.292727] remoteproc remoteproc0: Direct firmware load for /lib/firmware/ethosu_apps.elf failed with error -2
[  252.292737] remoteproc remoteproc0: Falling back to sysfs fallback for: /lib/firmware/ethosu_apps.elf
[  252.295520] remoteproc remoteproc0: Booting fw image /lib/firmware/ethosu_apps.elf, size 227536
[  252.308739] imx-rproc remoteproc-cm33: Translation failed: da = 0xffec288 len = 0x60000
[  252.308761] remoteproc remoteproc0: bad phdr da 0xffec288 mem 0x60000
[  252.308766] remoteproc remoteproc0: Failed to load program segments: -22
[  252.309016] remoteproc remoteproc0: Boot failed: -22

[phanichinimilli]

Its pleasing to see quick responses.

1. In your step #3 I'm afraid , you need to pass external_delegate argument to label_image
   for the inference to run on ETHOSU, else the inference runs on A55, that is why the inference results are in A55 lines.
   (Please forgive me for confusing you with label_image_release vs label_image, it is actually label_image )
   If you provide external delegate argument to label_image, logs on M33 console would be seen, ref to section-2.7.2 
   (you might have to provide vela model if the command doesnt work -- for me it works only with vela model)

> cd /usr/bin/tensorflow-lite-2.16.2/examples
> vela mobilenet_v1_1.0_224_quant.tflite
> ./label_image -m output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libethosu_delegate.so

The above procedure works okay(below) for me with default/original ethosu_firmware,

/usr/bin/tensorflow-lite-2.16.2/examples# ./label_image -m output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt  -external_delegate_path=/usr/lib/libethosu_delegate.so
INFO: Loaded model output/mobilenet_v1_1.0_224_quant_vela.tflite
INFO: resolved reporter
INFO: before CreateAllDelegates 
INFO: Ethosu delegate: device_name set to /dev/ethosu0.
INFO: Ethosu delegate: cache_file_path set to .
INFO: Ethosu delegate: timeout set to 60000000000.
INFO: Ethosu delegate: enable_cycle_counter set to 0.
INFO: Ethosu delegate: enable_profiling set to 0.
INFO: Ethosu delegate: profiling_buffer_size set to 2048.
INFO: Ethosu delegate: pmu_event0 set to 0.
INFO: Ethosu delegate: pmu_event1 set to 0.
INFO: Ethosu delegate: pmu_event2 set to 0.
INFO: Ethosu delegate: pmu_event3 set to 0.
INFO: EXTERNAL delegate created.
INFO: EthosuDelegate: 1 nodes delegated out of 1 nodes with 1 partitions.
INFO: Applied EXTERNAL delegate.
INFO: settings->number_of_warmup_runs 2
INFO: invoked
INFO: average time: 3.762 ms
INFO: Predicted label = 907

But If I replace default one with compiled one I see error(below) and can see logs in M33 console, My expectation is that with compiled ethosu_firmware inference should be successful

/lib/firmware/# reboot
/lib/firmware/# cp ethosu_apps_rpmsg.elf ethosu_firmware

/usr/bin/tensorflow-lite-2.16.2/examples# ./label_image -m output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt  -external_delegate_path=/usr/lib/libethosu_delegate.so

INFO: Loaded model output/mobilenet_v1_1.0_224_quant_vela.tflite
INFO: resolved reporter
INFO: before CreateAllDelegates 
INFO: Ethosu delegate: device_name set to /dev/ethosu0.
INFO: Ethosu delegate: cache_file_path set to .
INFO: Ethosu delegate: timeout set to 60000000000.
INFO: Ethosu delegate: enable_cycle_counter set to 0.
INFO: Ethosu delegate: enable_profiling set to 0.
INFO: Ethosu delegate: profiling_buffer_size set to 2048.
INFO: Ethosu delegate: pmu_event0 set to 0.
INFO: Ethosu delegate: pmu_event1 set to 0.
INFO: Ethosu delegate: pmu_event2 set to 0.
INFO: Ethosu delegate: pmu_event3 set to 0.
INFO: EXTERNAL delegate created.
INFO: EthosuDelegate: 1 nodes delegated out of 1 nodes with 1 partitions.
INFO: Applied EXTERNAL delegate.
INFO: settings->number_of_warmup_runs 2
ERROR: Ethos_u inference failed

ERROR: Node number 1 (EthosuDelegate) failed to invoke.
ERROR: Failed to invoke tflite

*** In both the cases Im seeing expected logs on M33 COM port

2. My observations of running M33 executables from linux i.e. by using "remoteproc0" are exactly same as yours.
3. Inference runner is different example that runs inference on ethosu. This example seems to work with both firmwares in step#1

#cd /usr/bin/ethosu/examples
#./inference_runner -d /dev/ethosu0 -n output/mobilenet_v1_1.0_224_quant_vela.tflite -i grace_hopper.bmp -l labels.txt -o output_data_detection -t 1080000000