Hi,
I have this device (some tablet):
# uname -a
Linux imx8mmevk 4.14.98-2.3.0+ #1 SMP PREEMPT Thu Jan 20 11:24:01 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
# cat /etc/issue
NXP i.MX Release Distro 4.14-sumo
If I use this pipeline as a source for Qt's MediaPlayer QML type in I get really good performance:
source: "gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! fpsdisplaysink video-sink=glimagesink"
fpsdisplaysink shows ~24fps (see glimagesink.png).
Now if I change to use qtimagesink so that video is embedded inside QtQuick interface like this:
source: "gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! fpsdisplaysink video-sink=qtvideosink"
I get only about 8-9 fps (see qtvideosink.png)...
I've tried changing output-format but with no luck.
Am I doing something wrong, or qtvideoisink implementation is inherently incompatible with imx8 HW acceleration?
Thanks!
P.S. I've attached example application too.
Solved! Go to Solution.
I've found solution:
My applicaiton now displays FullHD RTSP stream using only ~8% of CPU!
It seems if I remove fpsdisplaysink as intermediary, I get rather stable 23-25fps with qtvideosink too.
OK so it seems I can reach 25fps using this pipeline:
gst-launch-1.0 rtspsrc "location=rtsp://login:pass@192.168.1.110:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1" protocols=udp latency=100 ! queue max-size-buffers=0 ! rtpjitterbuffer latency=100 ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! vpudec ! queue max-size-buffers=0 ! fpsdisplaysink video-sink=glimagesink
Thanks to: https://imxdev.gitlab.io/tutorial/i.MX_6Q_RTSP_video_and_camera_stream/
Though video stutters (0 - 25fps) when I use qtvideosink in my Qt application
Well, even if CPU usage looks great (small), the FPS is rather low...
If I use (avoiding Qt' qtvideosink)
gst-launch-1.0 filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! fpsdisplaysink video-sink=glimagesink
to play back Big Buck Bunny 30fps Full HD movie I get about 23-25fps, NOT 30fps.
And if I use
gst-launch-1.0 rtspsrc "location=rtsp://login:pass@192.168.1.110:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1" protocols=udp latency=100 ! rtph264depay ! vpudec ! queue ! fpsdisplaysink video-sink=glimagesink
to display RTSP stream from Hikvision IP camera, I get only about 12fps.
Attached screenshots.
Is this limitation of imx8mm decoder, or maybe I have to tune gstreamer pipeline?
I've found solution:
My applicaiton now displays FullHD RTSP stream using only ~8% of CPU!
I've tried glupload & glcolorconvert to convert to the format qtvideosink needs like this:
source: "gst-pipeline: filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! queue ! glupload ! glcolorconvert ! qtvideosink"
but now it's even slower when without QT_GSTREAMER_USE_OPENGL_PLUGIN...
I've tried to use imxvideoconvert_g2d to convert pixels, but it does not connect to qtvideosink:
"gst-pipeline: filesrc location=/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! imxvideoconvert_g2d ! qtvideosink"
0:00:00.635955725 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:1172:gst_element_get_compatible_pad: incompatible pads
0:00:00.635969849 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:1178:gst_element_get_compatible_pad: unreffing pads
0:00:00.635987849 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:1201:gst_element_get_compatible_pad:<imxvideoconvert_g2d0> Could not find a compatible unlinked always pad to link to qgstvideorenderersink1:sink, now checking request pads
0:00:00.636004223 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:3058:gst_pad_query_caps:<qgstvideorenderersink1:sink> get pad caps with filter (NULL)
0:00:00.636020223 6874 0x3d9e4e00 DEBUG query gstquery.c:675:gst_query_new_custom: creating new query 0x3dcf59e0 caps
0:00:00.636039847 6874 0x3d9e4e00 DEBUG GST_PADS gstpad.c:4049:gst_pad_query:<qgstvideorenderersink1:sink> doing query 0x3dcf59e0 (caps)
0:00:00.636066471 6874 0x3d9e4e00 DEBUG GST_PADS gstpad.c:4072:gst_pad_query:<qgstvideorenderersink1:sink> sent query 0x3dcf59e0 (caps), result 1
0:00:00.636258215 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:3065:gst_pad_query_caps:<qgstvideorenderersink1:sink> query returned video/x-raw(memory:GLMemory), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(
int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(in
t)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)RGBx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)ARGB, framerate=(fraction)[ 0
/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]
0:00:00.636340088 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:900:gst_element_get_compatible_pad_template: Looking for a suitable pad template in imxvideoconvert_g2d0 out of 2 templates...
0:00:00.636358712 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:916:gst_element_get_compatible_pad_template: compatible direction: found src pad template "src"
0:00:00.636418960 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:919:gst_element_get_compatible_pad_template: intersecting video/x-raw(memory:GLMemory), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 21
47483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147
483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)RGBx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]; video/x-raw(memory:GLMemory), format=(string)ARGB, framerate=(fraction)[ 0/1, 2147483
647/1 ], width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ]
0:00:00.636451584 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:921:gst_element_get_compatible_pad_template: ..and video/x-raw, format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR }; video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComp
osition), format=(string){ RGB16, RGBx, RGBA, BGRA, BGRx, BGR16, ARGB, ABGR, xRGB, xBGR }
0:00:00.636467584 6874 0x3d9e4e00 DEBUG GST_CAPS gstutils.c:927:gst_element_get_compatible_pad_template: caps are not compatible
0:00:00.636476708 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:941:gst_element_get_compatible_pad_template: No compatible pad template found
0:00:00.636500332 6874 0x3d9e4e00 INFO GST_ELEMENT_PADS gstutils.c:1227:gst_element_get_compatible_pad:<imxvideoconvert_g2d0> Could not find a compatible pad to link to qgstvideorenderersink1:sink
0:00:00.636514832 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:1999:gst_element_link_pads_full: we might have request pads on both sides, checking...
0:00:00.636529207 6874 0x3d9e4e00 DEBUG GST_ELEMENT_PADS gstutils.c:2053:gst_element_link_pads_full: no link possible from imxvideoconvert_g2d0 to qgstvideorenderersink1
0:00:00.636543956 6874 0x3d9e4e00 INFO default gstutils.c:2162:gst_element_link_pads_filtered: Could not link pads: imxvideoconvert_g2d0:(null) - qgstvideorenderersink1:(null)
0:00:00.636568955 6874 0x3d9e4e00 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link imxvideoconvert_g2d0 to qgstvideorenderersink1
Error: " filesrc location=/root/qtprefix/memory/bbb.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! vpudec ! queue ! imxvideoconvert_g2d ! qtvideosink" : "could not link imxvideoconvert_g2d0 to qgstvideorenderersink1"
qml: onError: could not link imxvideoconvert_g2d0 to qgstvideorenderersink1
It seems one supports memory:GLMemory, while other memory:SystemMemory..?
It seems we need to set QT_GSTREAMER_USE_OPENGL_PLUGIN=1 (see https://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/gsttools/qgstutils.cpp?h=v5.15.4-lts-lgpl#n1573) for your application so that qtvideosink would attempt to use OpenGL rendering. But, it does not work on IMX, because of unsupported format:
https://codereview.qt-project.org/c/qt/qtmultimedia/+/276809
So we need to either some fast VPU-base pixel format conversation, or that vpudec would itself provide RGB format, OR that Qt developers would implement NV12/I420 format in qtvideosink..? I imagine that's doable, because glimagesink works fine...