OpenCV vs GStreamer Pipeline for SSD-MobileNetV2 Inference on i.MX8MP (BSP 5.15.52) using VX Delegat

[ashwanipal01]

Hello NXP Community,

I'm working on an object detection pipeline on the i.MX8MP platform, using NXP Linux BSP 5.15.52, with a quantized SSD-MobileNetV2 model and 480p video input. I’ve integrated the VX Delegate (libvx_delegate.so) into a TFLite runtime pipeline.

1. Current Setup:
Using OpenCV + TFLite with VX Delegate:

python
cap = cv2.VideoCapture(args.video)
if not cap.isOpened():
print(f"Failed to open video: {args.video}")
sys.exit(1)

interpreter = Interpreter(
model_path=args.model,
experimental_delegates=[load_delegate(DELEGATE_PATH)]
)
interpreter.allocate_tensors()
This works, but I observe lower FPS than expected.

2. Questions:
Performance:
Would building a full GStreamer-based pipeline (instead of OpenCV VideoCapture) improve real-time inference FPS when using TFLite + VX Delegate on i.MX8MP?

Model Quality (eIQ Toolkit vs TensorFlow CLI):
I noticed that models created using the eIQ Toolkit show lower performance (accuracy/latency) compared to models converted via TensorFlow command-line tools.
Is this expected? Are there differences in quantization strategy, postprocessing, or layer support between the two?

NXP Model Zoo:
Are there recommended pipelines or examples from the NXP Model Zoo for mobilenet_ssd_v2 that are optimized for VX Delegate on i.MX8MP BSP 5.15.52?

Any guidance, benchmarks, or sample pipelines (especially using GStreamer + TFLite + VX Delegate on i.MX8MP) would be highly appreciated!

Thanks in advance!

[Zhiming_Liu]

Hi @ashwanipal01 

For the object detection in video, please refer 8 Vision Pipeline with NNStreamer in i.MX Machine Learning User's Guide

https://www.nxp.com/docs/en/user-guide/UG10166.pdf

Best Regards,
Zhiming

[ashwanipal01]

hi @Zhiming_Liu ,

We followed the nnstreamer-example for the pipeline. Modified the pipeline according to the requirement.

       self.pipeline = Gst.parse_launch(
            'filesrc location=/home/weston/nnstre/detect_example.mp4 ! decodebin ! '
            'videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGBA ! tee name=t '  # tee to split for inference and display

            # ➤ Inference Branch
            't. ! queue max-size-buffers=20 leaky=downstream ! '
            'imxvideoconvert_g2d ! video/x-raw,format=RGBA,width=640,height=480 ! '
            'videoscale ! video/x-raw,width=300,height=300,format=RGBA ! '
            'videoconvert ! video/x-raw,format=RGB ! '
            'tensor_converter ! '
            'tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 ! '
            'identity silent=false ! '
            'tensor_filter framework=tensorflow-lite model=' + self.tflite_model + ' '
                'custom=Delegate:External,ExtDelegateLib:libvx_delegate.so name=detection_filter ! '
            'tensor_decoder mode=bounding_boxes option1=mobilenet-ssd '
                'option2=' + self.tflite_label + ' '
                'option3=' + self.tflite_box_prior + ' '
                'option4=640:480 option5=300:300 ! '
            'imxvideoconvert_g2d ! mix. '

            # ➤ Display Branch
            't. ! queue max-size-buffers=20 leaky=downstream ! '
            'imxvideoconvert_g2d ! video/x-raw,format=RGBA,width=640,height=480 ! '
            'imxcompositor_g2d name=mix sink_0::zorder=2 sink_1::zorder=1 latency=40000000 min-upstream-latency=40000000 ! '
            'fpsdisplaysink name=img_tensor text-overlay=true video-sink=waylandsink sync=true'
        )

attaching the log file and output video.
 
we are getting the fps of 25 but the output video is not sync as input video and feels like stuck or freeze.

[Zhiming_Liu]

Hi @ashwanipal01 

Can you refer the pipieline setting here? Don't use video scale.

https://github.com/nxp-imx-support/nxp-demo-experience-demos-list/tree/lf-6.12.3_1.0.0/scripts/machi...

gst-launch-1.0 v4l2src name=cam_src device=/dev/video3 num-buffers=-1 ! \
  video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t 
    t. ! queue name=thread-nn max-size-buffers=2 leaky=2 ! \
      imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! \
      videoconvert ! video/x-raw,format=RGB ! tensor_converter ! \
      tensor_filter latency=1 framework=tensorflow-lite \
      model=/opt/gopoint-apps/downloads/ssdlite_mobilenet_v2_coco_quant_uint8_float32_no_postprocess.tflite \
      custom=Delegate:External,ExtDelegateLib:libvx_delegate.so name=detection_filter ! \
      tensor_decoder mode=bounding_boxes option1=mobilenet-ssd \
      option2=/opt/gopoint-apps/downloads/coco_labels_list.txt \
      option3=/opt/gopoint-apps/downloads/box_priors.txt option4=640:480 option5=300:300 ! \
      imxvideoconvert_g2d ! mix. \
    t. ! queue name=thread-img max-size-buffers=2 leaky=2 ! \
      imxcompositor_g2d name=mix sink_0::zorder=2 sink_1::zorder=1 latency=20000000 min-upstream-latency=20000000 ! \
      cairooverlay name=perf ! \
      fpsdisplaysink name=img_tensor text-overlay=false video-sink=waylandsink sync=false

Best Regards,
Zhiming