Bad Qt 5.15.10 qtvideosink performance with IMX8MM and vpudec

cancel
Showing results for 
Search instead for 
Did you mean: 

Bad Qt 5.15.10 qtvideosink performance with IMX8MM and vpudec

Jump to solution
995 Views
VincasD
Contributor III

Hi,

I have this device (some tablet):

 

 

 

# uname -a
Linux imx8mmevk 4.14.98-2.3.0+ #1 SMP PREEMPT Thu Jan 20 11:24:01 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

# cat /etc/issue
NXP i.MX Release Distro 4.14-sumo

 

 

 

If I use this pipeline as a source for Qt's MediaPlayer QML type in I get really good performance:

 

 

 

source: "gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! fpsdisplaysink video-sink=glimagesink"

 

 

 

fpsdisplaysink shows ~24fps (see glimagesink.png).

Now if I change to use qtimagesink so that video is embedded inside QtQuick interface like this:

 

 

 

source: "gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! fpsdisplaysink video-sink=qtvideosink"

 

 

 

I  get only about 8-9 fps (see qtvideosink.png)...

I've tried changing output-format but with no luck.

Am I doing something wrong, or qtvideoisink implementation is inherently incompatible with imx8 HW acceleration?

Thanks!

P.S. I've attached example application too.

Tags (1)
0 Kudos
1 Solution
975 Views
VincasD
Contributor III

I've found solution:

  • DO NOT set QT_GSTREAMER_USE_OPENGL_PLUGIN=1
  • Copy plugins/video/videonode/libimx6vivantevideonode.so so it is available to your deployed application (I've never seen anything like that in my Desktop Qt installations, these videonode plugins was new for me).
  • Simply use "...vpudec ! queue ! qtvideosink" pipeline.

My applicaiton now displays FullHD RTSP stream using only ~8% of CPU!

View solution in original post

0 Kudos
7 Replies
963 Views
VincasD
Contributor III

It seems if I remove fpsdisplaysink as intermediary, I get rather stable 23-25fps with qtvideosink too.

0 Kudos
963 Views
VincasD
Contributor III

OK so it seems I can reach 25fps using this pipeline:

gst-launch-1.0 rtspsrc "location=rtsp://login:pass@192.168.1.110:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1" protocols=udp latency=100 ! queue max-size-buffers=0 ! rtpjitterbuffer latency=100 ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! vpudec ! queue max-size-buffers=0 ! fpsdisplaysink video-sink=glimagesink

Thanks to: https://imxdev.gitlab.io/tutorial/i.MX_6Q_RTSP_video_and_camera_stream/

Though video stutters (0 - 25fps) when I use qtvideosink in my Qt application .

0 Kudos
967 Views
VincasD
Contributor III

Well,  even if CPU usage looks great (small), the FPS is rather low...

If I use (avoiding Qt' qtvideosink)

gst-launch-1.0 filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! fpsdisplaysink video-sink=glimagesink

to play back Big Buck Bunny 30fps Full HD movie I get about 23-25fps, NOT 30fps.

And if I use

gst-launch-1.0 rtspsrc "location=rtsp://login:pass@192.168.1.110:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1" protocols=udp latency=100 ! rtph264depay ! vpudec ! queue ! fpsdisplaysink video-sink=glimagesink

to display RTSP stream from Hikvision IP camera, I get only about 12fps.

Attached screenshots.

Is this limitation of imx8mm decoder, or maybe I have to tune gstreamer pipeline?

0 Kudos
976 Views
VincasD
Contributor III

I've found solution:

  • DO NOT set QT_GSTREAMER_USE_OPENGL_PLUGIN=1
  • Copy plugins/video/videonode/libimx6vivantevideonode.so so it is available to your deployed application (I've never seen anything like that in my Desktop Qt installations, these videonode plugins was new for me).
  • Simply use "...vpudec ! queue ! qtvideosink" pipeline.

My applicaiton now displays FullHD RTSP stream using only ~8% of CPU!

0 Kudos
980 Views
VincasD
Contributor III

I've tried glupload & glcolorconvert to convert to the format qtvideosink needs like this:

source: "gst-pipeline: filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! queue ! glupload ! glcolorconvert ! qtvideosink"

but now it's even slower when without QT_GSTREAMER_USE_OPENGL_PLUGIN...

 

0 Kudos
981 Views
VincasD
Contributor III

I've tried to use imxvideoconvert_g2d to convert pixels, but it does not connect to qtvideosink:

"gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! imxvideoconvert_g2d ! qtvideosink"
0:00:00.635955725  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:1172:gst_element_get_compatible_pad: incompatible pads
0:00:00.635969849  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:1178:gst_element_get_compatible_pad: unreffing pads
0:00:00.635987849  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:1201:gst_element_get_compatible_pad:<imxvideoconvert_g2d0> Could not find a compatible unlinked always pad to link to qgstvideorenderersink1:sink, now checking request pads
0:00:00.636004223  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:3058:gst_pad_query_caps:<qgstvideorenderersink1:sink> get pad caps with filter (NULL)
0:00:00.636020223  6874     0x3d9e4e00 DEBUG                  query gstquery.c:675:gst_query_new_custom: creating new query 0x3dcf59e0 caps
0:00:00.636039847  6874     0x3d9e4e00 DEBUG               GST_PADS gstpad.c:4049:gst_pad_query:<qgstvideorenderersink1:sink> doing query 0x3dcf59e0 (caps)
0:00:00.636066471  6874     0x3d9e4e00 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<qgstvideorenderersink1:sink> sent query 0x3dcf59e0 (caps), result 1
0:00:00.636258215  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:3065:gst_pad_query_caps:<qgstvideorenderersink1:sink> query returned video/x-raw(memory:GLMemory), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(
int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(in
t)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)RGBx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)ARGB, framerate=(fraction)[ 0
/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]
0:00:00.636340088  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:900:gst_element_get_compatible_pad_template: Looking for a suitable pad template in imxvideoconvert_g2d0 out of 2 templates...
0:00:00.636358712  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:916:gst_element_get_compatible_pad_template: compatible direction: found src pad template "src"
0:00:00.636418960  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:919:gst_element_get_compatible_pad_template: intersecting video/x-raw(memory:GLMemory), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 21
47483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147
483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)RGBx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)ARGB, framerate=(fraction)[ 0/1, 2147483
647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]
0:00:00.636451584  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:921:gst_element_get_compatible_pad_template: ..and video/x-raw, format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR }; video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComp
osition), format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR }
0:00:00.636467584  6874     0x3d9e4e00 DEBUG               GST_CAPS gstutils.c:927:gst_element_get_compatible_pad_template: caps are not compatible
0:00:00.636476708  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:941:gst_element_get_compatible_pad_template: No compatible pad template found
0:00:00.636500332  6874     0x3d9e4e00 INFO        GST_ELEMENT_PADS gstutils.c:1227:gst_element_get_compatible_pad:<imxvideoconvert_g2d0> Could not find a compatible pad to link to qgstvideorenderersink1:sink
0:00:00.636514832  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:1999:gst_element_link_pads_full: we might have request pads on both sides, checking...
0:00:00.636529207  6874     0x3d9e4e00 DEBUG       GST_ELEMENT_PADS gstutils.c:2053:gst_element_link_pads_full: no link possible from imxvideoconvert_g2d0 to qgstvideorenderersink1
0:00:00.636543956  6874     0x3d9e4e00 INFO                 default gstutils.c:2162:gst_element_link_pads_filtered: Could not link pads: imxvideoconvert_g2d0:(null) - qgstvideorenderersink1:(null)
0:00:00.636568955  6874     0x3d9e4e00 ERROR           GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link imxvideoconvert_g2d0 to qgstvideorenderersink1
Error: " filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! queue ! imxvideoconvert_g2d ! qtvideosink" : "could not link imxvideoconvert_g2d0 to qgstvideorenderersink1"
qml: onError: could not link imxvideoconvert_g2d0 to qgstvideorenderersink1

It seems one supports memory:GLMemory, while other memory:SystemMemory..?

0 Kudos
981 Views
VincasD
Contributor III

It seems we need to set QT_GSTREAMER_USE_OPENGL_PLUGIN=1 (see https://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/gsttools/qgstutils.cpp?h=v5.15.4-lts-lgpl#n1573) for your application so that qtvideosink would attempt to use OpenGL rendering. But, it does not work on IMX, because of unsupported format:

https://codereview.qt-project.org/c/qt/qtmultimedia/+/276809

So we need to either some fast VPU-base pixel format conversation, or that vpudec would itself provide RGB format, OR that Qt developers would implement NV12/I420 format in qtvideosink..? I imagine that's doable, because glimagesink works fine...

0 Kudos