Object detection Execution issue

[RevathyR]

I am trying to run a object detection in the imx 8m plus evk by referring the https://www.nxp.com/docs/en/user-guide/IMX-MACHINE-LEARNING-UG.pdf document(Section 8.1)

When I initially tried with that steps I got this error "gi.repository.GLib.GError: gst_parse_error: no element "tensor_converter" (1)".I solved this error by adding nnstreamer and tensorflow-lite support to the image.

Now Im facing the below issue

** (gst-launch-1.0:1891): WARNING **: 11:21:54.892: Cannot identify the given neural network framework, tensorflow-lite

** (gst-launch-1.0:1891): CRITICAL **: 11:21:54.909: _check_tensors: assertion 'config->info.num_tensors >= limit' failed

** (gst-launch-1.0:1891): CRITICAL **: 11:21:54.909: _check_tensors: assertion 'config->info.num_tensors >= limit' failed
WARNING: erroneous pipeline: could not link tensorconverter0 to tensorfilter0

I have checked online for solutions but nothing worked.

Please help me resolve the issue at the earliest.

[Fabien1]

Hi,

I'm facing the same issue when trying to use tensor_filter with tensorflow_lite backend. It looks like you are able to resolve this issue (first post) "Cannot identify the given neural network framework, tensorflow-lite".
Could you please share the fix?

tensorflow-lite and nnstreamer are correctly installed in my iMX8M-Plus image.

THANKS

[RevathyR]

@Fabien1 
In nxp_sdk/sources/meta-imx/meta-sdk/recipes-fsl/images/imx-image-multimedia.bb file add tensorflow-lite in the end and do the below steps

bitbake tensorflow-lite

bitbake imx-image-multimedia.

This resolved my issue.
Hope it helps.

[RevathyR]

Hi  @Zhiming_Liu 

I have successfully executed the object detection in the  imx 8m plus evk.But it is running in cpu.I would like to run it on the npu.

Kindly help me run the same in npu.

The command is as follows

gst-launch-1.0 --no-position v4l2src device=/dev/video3 ! video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=${MODEL} custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=${LABELS} option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

Attaching the logs,

Logs :

** Message: 05:22:29.479: accl = cpu
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...

Thanks in advance

[Zhiming_Liu]

Hi @RevathyR 

Have you checked the cpu load?

Some operators run on cpu and most operators run on NPU. This is normal.

[RevathyR]

@Zhiming_Liu 

Yes I have checked the cpu load after executing the process.

Initially it was 100% cpu, then reduced to 80% and then the process was not there in top.

How can we check if that process runs on cpu or npu?
Can we list the processes running on npu ?

Attaching the snapshots of top commnd for reference.

Thanks in advance.

[Zhiming_Liu]

Hi @RevathyR 

Are you flashing imx-image-full ?

[RevathyR]

Hi @Zhiming_Liu 

Thanks for your response.

I have flashed imx-image-multimedia.

Is that the problem?

[Zhiming_Liu]

Please try imx-image-full.

[RevathyR]

@Zhiming_Liu 

I have build the imx-image-full image and flashed it in the evk. After that I have transferred the necessary files like tflite model,label file and a video as I don't have connected camera.

I have used this command (referred from nxp machine learning user guide)

gst-launch-1.0 --no-position filesrc location=./two_persons_talking.mp4 ! video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=/home/root/ml_demo/ssd_mobilenet_v2_coco_quant_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=/home/root/ml_demo/coco_labels.txt option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

The above command has been modified to use a video file instead of a camera source.Other than that no change has been made.

Now Im facing this error

** Message: 12:42:56.003: accl = cpu
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/GstCapsFilter:capsfilter0: Filter caps do not completely specify the output format
Additional debug info:
/usr/src/debug/gstreamer1.0/1.22.0.imx-r0/plugins/elements/gstcapsfilter.c(458): gst_capsfilter_prepare_buf (): /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
Output caps are unfixed: video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, views=(int)1, interlace-mode=(string)progressive, format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR, I420, NV12, UYVY, YUY2, YVYU, YV12, NV16, NV21 }
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstFileSrc:filesrc0: Internal data stream error.
Additional debug info:
/usr/src/debug/gstreamer1.0/1.22.0.imx-r0/libs/gst/base/gstbasesrc.c(3132): gst_base_src_loop (): /GstPipeline:pipeline0/GstFileSrc:filesrc0:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...
root@NXP_iMX_8M_plus_EVK:~/ml_demo#
FATAL: read zero bytes from port
term_exitfunc: reset failed for dev UNKNOWN: Input/output error

Can you help on this?

[RevathyR]

@Zhiming_Liu 

After adding format=RGB Im getting the following warnings and the process quits.

WARNING: Fallback unsupported op 32 to TfLite
WARNING: erroneous pipeline: could not link queue0 to imxvideoconvert_g2d0

The modified command is 

gst-launch-1.0 --no-position filesrc location=./two_persons_talking.mp4 ! video/x-raw,width=640,height=480,framerate=30/1,format=RGB ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=/home/root/ml_demo/ssd_mobilenet_v2_coco_quant_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=/home/root/ml_demo/coco_labels.txt option3=0:1:2:3,50 option4=640:480 option5=300:300 ! mix. t. ! queue max-size-buffers=2 ! imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 sink_0::zorder=2 sink_1::zorder=1 ! waylandsink

I'm trying to run the command for a video.Is that the problem.
Kindly have a look and provide me with the solution.

Thanks in advanace

[Zhiming_Liu]

You should use v4l2h264dec like below command to replace imxvideoconvert_g2d  and videoconvert 

https://www.nxp.com/docs/en/user-guide/IMX_LINUX_USERS_GUIDE.pdf

[RevathyR]

Hi @Zhiming_Liu 

Kindly have a look at the things I have tried and correct me if anything is wrong.

Thanks in Advance

[RevathyR]

@Zhiming_Liu 
I have tried replacing the imxvideoconvert_g2d with v4l2h264dec but got

“WARNING: erroneous pipeline: could not link queue0 to v4l2h264dec0”.

Should I try any other way ?

[RevathyR]

@Zhiming_Liu 

Should I replace imxvideoconvert_g2d with video/quicktime  ! queue max-size-time=0 ! v4l2h264dec !

and videoconvert with video/quicktime  ! queue max-size-time=0 ! v4l2h264dec ! ?

Or in total I have to include video/quicktime  ! queue max-size-time=0 ! v4l2h264dec !.

Im new to this device.

Please clarify.

[RevathyR]

@Zhiming_Liu 

Yes sure.

I will try with imx-image-full image.