GStreamer and NNStreamer use VX Delegate Fails

KrisJ1 — Fri, 23 Jun 2023 21:52:11 GMT

I am using a BoundaryDevices iMX8MP board and want to run predictions using TFLite and VX Delegate on the NPU. Here is my GStreamer command:

gst-launch-1.0 \
    videotestsrc is-live=1 ! \      
    video/x-raw,width=640,height=480 ! \
    tee name=t \    
    t. ! queue name=thread-nn max-size-buffers=2 leaky=2 ! \
    imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! \      
    tensor_converter ! \
    tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 ! \      
    tensor_filter framework=tensorflow-lite model=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/ssdlite_mobilene
t_v2_coco_no_postprocess.tflite custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! \      
    tensor_decoder mode=bounding_boxes option1=mobilenet-ssd option2=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/coco_labels_list.txt option3=/home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/box_priors.txt option4=640:480 option5=300:300 ! \
    videoconvert ! \      
    mix. \
    t. ! queue name=thread-img max-size-buffers=2 leaky=2 ! \      
    videoconvert ! \
    imxcompositor_g2d name=mix sink_0::zorder=2 sink_1::zorder=1 latency=300000000 min-upstream-latency=300000000 ! \      
    fpsdisplaysink video-sink=$VID_SINK name=test sync=false text-overlay=true -v 2>&1

I am using custom=Delegate:External,ExtDelegateLib:libvx_delegate.so to use the VX Delegate on the NPU. I have confirmed that this dynamic library is there, and it is usable since the benchmark-model and modelrunner tools all allow the NPU to be used. When I run this GStreamer command, I get the following:

...
ERROR: Invoke called on model that is not ready.

** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.018: Failed to invoke

** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.019: /usr/lib/libnnstreamer-single.so(_backtrace_to_string+0x40) [0xffff6edb73c0]/usr/lib/gstreamer-1.0/libnnstreamer.so(+0x3d9ac) [0xffff6ee1d9ac]/usr/lib/libgstbase-1.0.so.0(+0x48704) [0xffff87198704]/usr/lib/libgstbase-1.0.so.0(+0x47dd4) [0xffff87197dd4]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/gstreamer-1.0/libnnstreamer.so(+0x1fba0) [0xffff6edffba0]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]/usr/lib/libgstreamer-1.0.so.0(+0x90798) [0xffff87840798]/usr/lib/libgstbase-1.0.so.0(+0x47ee8) [0xffff87197ee8]/usr/lib/libgstreamer-1.0.so.0(+0x8e908) [0xffff8783e908]


** (gst-launch-1.0:1613): CRITICAL **: 21:30:40.019: Calling invoke function (inference instance) of the tensor-filter subplugin (tensorflow-lite for /home/root/nxp-nnstreamer-examples/detection/../downloads/models/detection/ssdlite_mobilenet_v2_coco_no_postprocess.tflite) has failed with error code (-1).

ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.063685468
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstQueue:thread-nn: Internal data stream error.
Additional debug info:
../git/plugins/elements/gstqueue.c(992): gst_queue_handle_sink_event (): /GstPipeline:pipeline0/GstQueue:thread-nn:
streaming stopped, reason error (-5)
...

Any help resolving this will be greatly appreciated. If this is not the right stack exchange to post this question, please inform me where I can post to get an answer. Also, let me know if you need any more information about the system I am running.

More information about the system:

Linux info: Linux nitrogen8mp 5.15.71-2.2.0+yocto+gd4a57b30c1d1
Custom Yocto build with GStreamer, NNStreamer, and TFLite VX Delegate installed.

Re: GStreamer and NNStreamer use VX Delegate Fails

Zhiming_Liu — Tue, 27 Jun 2023 06:48:21 GMT

Which kind of video source you want to use? from camera or from video file?

I can't find you set the valid input source.

Please refer chapter 10 Vision Pipeline with NNStreamer in this guide: https://www.nxp.com/docs/en/user-guide/IMX-MACHINE-LEARNING-UG.pdf

Re: GStreamer and NNStreamer use VX Delegate Fails

KrisJ1 — Wed, 28 Jun 2023 16:25:30 GMT

I am using videotestsrc which is random video generated indefinitely by GStreamer and doesn't have a framerate limit allowing me to test the true performance of the model. https://gstreamer.freedesktop.org/documentation/videotestsrc/index.html?gi-language=c

Re: GStreamer and NNStreamer use VX Delegate Fails

KrisJ1 — Fri, 30 Jun 2023 19:11:56 GMT

I also tried using v4l2src to get a camera feed and filesrc from an .AVI file. Both attempts resulted in the same error.

i.MX ProcessorsのトピックRe: GStreamer and NNStreamer use VX Delegate Fails

GStreamer and NNStreamer use VX Delegate Fails

Re: GStreamer and NNStreamer use VX Delegate Fails

Re: GStreamer and NNStreamer use VX Delegate Fails

Re: GStreamer and NNStreamer use VX Delegate Fails