topic Object detection Execution issue in i.MX Processors

Object detection Execution issue

RevathyR — Fri, 29 Dec 2023 17:23:35 GMT

I am trying to run a object detection in the imx 8m plus evk by referring the https://www.nxp.com/docs/en/user-guide/IMX-MACHINE-LEARNING-UG.pdf document(Section 8.1)

When I initially tried with that steps I got this error "gi.repository.GLib.GError: gst_parse_error: no element "tensor_converter" (1)".I solved this error by adding nnstreamer and tensorflow-lite support to the image.

Now Im facing the below issue

** (gst-launch-1.0:1891): WARNING **: 11:21:54.892: Cannot identify the given neural network framework, tensorflow-lite

** (gst-launch-1.0:1891): CRITICAL **: 11:21:54.909: _check_tensors: assertion 'config->info.num_tensors >= limit' failed

** (gst-launch-1.0:1891): CRITICAL **: 11:21:54.909: _check_tensors: assertion 'config->info.num_tensors >= limit' failed
WARNING: erroneous pipeline: could not link tensorconverter0 to tensorfilter0

I have checked online for solutions but nothing worked.

Please help me resolve the issue at the earliest.

Re: Object detection Execution issue

Zhiming_Liu — Tue, 02 Jan 2024 02:31:15 GMT

Hi @RevathyR

Are you flashing imx-image-full ?

Re: Object detection Execution issue

RevathyR — Tue, 02 Jan 2024 03:39:32 GMT

Hi @Zhiming_Liu

Thanks for your response.

I have flashed imx-image-multimedia.

Is that the problem?

Re: Object detection Execution issue

Zhiming_Liu — Tue, 02 Jan 2024 07:26:25 GMT

Please try imx-image-full.

Re: Object detection Execution issue

RevathyR — Tue, 02 Jan 2024 11:42:26 GMT

@Zhiming_Liu

Yes sure.

I will try with imx-image-full image.

Re: Object detection Execution issue

RevathyR — Tue, 09 Jan 2024 04:15:37 GMT

@Zhiming_Liu

I have build the imx-image-full image and flashed it in the evk. After that I have transferred the necessary files like tflite model,label file and a video as I don't have connected camera.

I have used this command (referred from nxp machine learning user guide)

gst-launch-1.0 --no-position filesrc location=./two_persons_talking.mp4 ! video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=/home/root/ml_demo/ssd_mobilenet_v2_coco_quant_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=/home/root/ml_demo/coco_labels.txt option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

The above command has been modified to use a video file instead of a camera source.Other than that no change has been made.

Now Im facing this error

** Message: 12:42:56.003: accl = cpu
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/GstCapsFilter:capsfilter0: Filter caps do not completely specify the output format
Additional debug info:
/usr/src/debug/gstreamer1.0/1.22.0.imx-r0/plugins/elements/gstcapsfilter.c(458): gst_capsfilter_prepare_buf (): /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
Output caps are unfixed: video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, views=(int)1, interlace-mode=(string)progressive, format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR, I420, NV12, UYVY, YUY2, YVYU, YV12, NV16, NV21 }
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstFileSrc:filesrc0: Internal data stream error.
Additional debug info:
/usr/src/debug/gstreamer1.0/1.22.0.imx-r0/libs/gst/base/gstbasesrc.c(3132): gst_base_src_loop (): /GstPipeline:pipeline0/GstFileSrc:filesrc0:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...
root@NXP_iMX_8M_plus_EVK:~/ml_demo#
FATAL: read zero bytes from port
term_exitfunc: reset failed for dev UNKNOWN: Input/output error

Can you help on this?

Re: Object detection Execution issue

RevathyR — Tue, 09 Jan 2024 05:52:12 GMT

@Zhiming_Liu

After adding format=RGB Im getting the following warnings and the process quits.

WARNING: Fallback unsupported op 32 to TfLite
WARNING: erroneous pipeline: could not link queue0 to imxvideoconvert_g2d0

The modified command is

gst-launch-1.0 --no-position filesrc location=./two_persons_talking.mp4 ! video/x-raw,width=640,height=480,framerate=30/1,format=RGB ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=/home/root/ml_demo/ssd_mobilenet_v2_coco_quant_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=/home/root/ml_demo/coco_labels.txt option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

I'm trying to run the command for a video.Is that the problem.
Kindly have a look and provide me with the solution.

Thanks in advanace

Re: Object detection Execution issue

Zhiming_Liu — Tue, 09 Jan 2024 07:45:48 GMT

You should use v4l2h264dec like below command to replace imxvideoconvert_g2d and videoconvert

https://www.nxp.com/docs/en/user-guide/IMX_LINUX_USERS_GUIDE.pdf

Re: Object detection Execution issue

RevathyR — Tue, 09 Jan 2024 09:24:05 GMT

@Zhiming_Liu

Should I replace imxvideoconvert_g2d with video/quicktime ! queue max-size-time=0 ! v4l2h264dec !

and videoconvert with video/quicktime ! queue max-size-time=0 ! v4l2h264dec ! ?

Or in total I have to include video/quicktime ! queue max-size-time=0 ! v4l2h264dec !.

Im new to this device.

Please clarify.

Re: Object detection Execution issue

RevathyR — Wed, 10 Jan 2024 04:16:18 GMT

@Zhiming_Liu
I have tried replacing the imxvideoconvert_g2d with v4l2h264dec but got

“WARNING: erroneous pipeline: could not link queue0 to v4l2h264dec0”.

Should I try any other way ?

Re: Object detection Execution issue

RevathyR — Fri, 12 Jan 2024 05:08:37 GMT

Hi @Zhiming_Liu

Kindly have a look at the things I have tried and correct me if anything is wrong.

Thanks in Advance

Re: Object detection Execution issue

RevathyR — Fri, 19 Jan 2024 05:31:13 GMT

Hi @Zhiming_Liu

I have successfully executed the object detection in the imx 8m plus evk.But it is running in cpu.I would like to run it on the npu.

Kindly help me run the same in npu.

The command is as follows

gst-launch-1.0 --no-position v4l2src device=/dev/video3 ! video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=${MODEL} custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=${LABELS} option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

Attaching the logs,

Logs :

** Message: 05:22:29.479: accl = cpu
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...

Thanks in advance

Re: Object detection Execution issue

Zhiming_Liu — Fri, 19 Jan 2024 06:09:37 GMT

Hi @RevathyR

Have you checked the cpu load?

Some operators run on cpu and most operators run on NPU. This is normal.

Re: Object detection Execution issue

RevathyR — Fri, 19 Jan 2024 06:40:48 GMT

@Zhiming_Liu

Yes I have checked the cpu load after executing the process.

Initially it was 100% cpu, then reduced to 80% and then the process was not there in top.

How can we check if that process runs on cpu or npu?
Can we list the processes running on npu ?

Attaching the snapshots of top commnd for reference.

Thanks in advance.

Re: Object detection Execution issue

Fabien1 — Thu, 01 Feb 2024 15:33:05 GMT

Hi,

I'm facing the same issue when trying to use tensor_filter with tensorflow_lite backend. It looks like you are able to resolve this issue (first post) "Cannot identify the given neural network framework, tensorflow-lite".
Could you please share the fix?

tensorflow-lite and nnstreamer are correctly installed in my iMX8M-Plus image.

THANKS

Re: Object detection Execution issue

RevathyR — Tue, 06 Feb 2024 05:36:08 GMT

@Fabien1
In nxp_sdk/sources/meta-imx/meta-sdk/recipes-fsl/images/imx-image-multimedia.bb file add tensorflow-lite in the end and do the below steps

bitbake tensorflow-lite

bitbake imx-image-multimedia.

This resolved my issue.
Hope it helps.