IMX8mp: GStreamer video into Qt6 QWidget

[TerryBarnaby1]

We have a custom IMX8mp based system that has been running an older NXP Yocto with QT5. This is for a video processing system using gstreamer for the video processing and with a video display into a QT5 QWidget. This is on a production system and has been working well for some time (although there were quite a few issues getting it going originally).

We have now doing a similar project and are starting with the NXP imx-linux-mickledore, imx-6.1.55-2.2.0.xml Yocto release. We have this basically running on our cutdown platform but are having problems getting a Video stream to show in a QWidget. We are using Weston/Wayland, with gstreamer and QT6. We use gstreamers waylandsink and set its Wayland surface to that of the QWidget. This works fine if the QWidget is a top level widget (QWidget(0)), but if this widget is nested in another, even just another QWidget the gstreamer pipeline just hangs in gstwaylandsink.c/gst_wayland_sink_show_frame at: {while (self->redraw_pending)
    g_cond_wait (&self->redraw_wait, &self->render_lock);}

I note the https://github.com/nxp-imx/gst-plugins-bad where waylandsink is has not been updated for 2 years.

1. Has anyone managed to get a gstreamer video pipeline to display in a QT6 application recently ?
2. Is there a later waylandsink I should be using ?
3. Anyone any ideas on the best way to do this these days with an IMX8mp ?

Terry

[malik_cisse]

I can understand your arguments regarding qml6glsink.

Note that QVideoSink is not tied to qml. It can as well be used with QWidget:
https://doc.qt.io/qt-6/qvideosink.html

Here is an example (they use qml here but you don't have to):
https://stackoverflow.com/questions/69432427/how-to-use-qvideosink-in-qml-in-qt6

You will probably find equivalent QWidget examples also.

Your gstreamer pipeline could sink into an "apsink". Tha apsink callback can feed the individual frames to QVideoSink

Of course this is less elegant than a solution where you sink directly to wayland.

[TerryBarnaby1]

I don't believe QVideoSync will be useful, it just accepts frames from the QMultimedia internals so you can get at the frame data. I have a program driven gstreamer pipeline.

I do use gstreamer appsinks for things and have tried using one of those to draw into a QWidget the frames, but this was just a simple test with an RGB888 stream. On my system the video will likely be a 16bit YUYV video stream and so will need YUYV to ARGB888 conversion along with scaling for the QWidget's window. This would have to be done in hardware (IMX8mp 2D/3D engine ?). It may be that if I can create a QVideoFrame with the appsink data frame (Might have to memcpy ) and use QPainter to "paint" onto a QWidget it might (by magic as there is no real documentation!) use the MX8mp 2D or 3D hardware to do this. How it will synchronise to video and LCD frame rate is another issue. I will have a play, but really the Waylandsink overlay approach seems best (and did work with older software). If anyone knows why this may not working (something in qtwayland, Gstreamer waylandsink, wayland lib, Weston, IMX G2D, IMX 2D/3D engines etc.) please let me know. It seems like the Wayland surface that the QWidget returned to be used by Waylandsink is "inactive" for some reason. Maybe something in qtwayland has set something or there is some issue in Weston.

Terry

[malik_cisse]

Just some clarifications:

>>I don't believe QVideoSync will be useful, it just accepts frames from the QMultimedia internals so you can get at the frame data
This is what is indeed said in the doc,but you can actually use QVideoSync the other way around as a frame source to feed a sink.  no QMultimedia in the loop at all. This is shown in the link I sent earlier. 
I am using exactly this setup and I am doing already what you are planning to do (with 12MP images by the way) except that I am using qml.

>>On my system the video will likely be a 16bit YUYV video stream and so will need YUYV to ARGB888 conversion along with scaling for the QWidget's window
This is peanuts for the imx8mp. The G2D GPU does that for you at almost no cost. for example:
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=800 ! videocrop top=0 left=120 right=120 bottom=0 ! imxvideoconvert_g2d rotation=3 ! video/x-raw, width=480, height=800 ! waylandsink

Operation such as cropping, rotation, scaling and format conversion is peanuts for the G2D.
YUYV to RGBX conversion is done in the pipeline above. CPU will be at ~1% in the case above.

Let me know which solution worked for you as I am highly interested.

[TerryBarnaby1]

Hi, Yes it should be peanuts for the imx8mp hardware.

We have been using gstreamer pipelines with waylandsink, with H264 encoding, overlaying QTWIdgets for overlay info, taking picture snapshots, de-interlacing, 1920x1080p25 etc etc, and it all works (just) using mainly hardware with low overall CPU usage ( < 15%).

I assume you mean QVideoFrame rather than QVideoSync ? I have tried create one of these from an appsink, but I'm not sure there is a way to render this with hardware doing 16 bit YUYV to ARGB8888 conversion with scaling from within Qt to a QWidget. Is there a method I don't see this in the link you posted ?

With the latest and greatest (never greatest with software updates these days!) NXP Yocto release, and using QT6 I can't with a standard Linux embedded system simply show a gstreamer processed Video in an area on my application!

The state of software today is appalling. Also I would have thought that NXP, with the iMX8 range where a target is Video processing, would have worked on this and would have a method of achieving this even if the underlying Yocto Linux does not provide a standard and simple way of handling this. For the old Yocto with QT5 I did have to modify qtwayland (lost of bugs/issues) and Weston, it looks like the same is needed even now. Rant rant

[malik_cisse]

Hi Terry,

Could you solve this problem yet?
I really meant QVideosink by the way. This really works as a frame source also. I have demonstrated it.

[TerryBarnaby1]

Not as yet. I have discussions in Gstreamer, Qt and Wayland mail lists/groups.

Main thread is in wayland-devel mail list: "Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston".

The current feeling is that it is a Qtwayland bug/issue/feature. The problem also shows with Fedora37 KDE/Plasma/Qt6.5.1/Wayland/Weston. It appears that Qtwayland is not actually setting up and using separate Wayland surfaces for sub Widgets properly.

I have been waylaid with testing a new IMX8mp custom board this last week, hope to get back to looking at this next week.

[TerryBarnaby1]

Thanks for the reply.

We really don't want to use QML unless we really have to. The software stack is getting way way too complex as it is and this will also increase the Yocto image size. Also I'm not sure we can use qml6sink directly from gstreamer without using QMultimedia which would be no use and I have seen notes to say that using qml6sink imposes quite a performance penalty. Does anyone know if we can use qml6sink with a program generated and managed gstreamer pipeline ?

We really just need a "simple" method to getting the waylandsink to work with a QWidget if we can. This appears to use the imx8mp graphics hardware to video stream format convert and scale to size and works quite well (apart from not working inside a non top level QWidget).

[malik_cisse]

There is a solution for this, however it is for qml6 and not for QTWidget:
https://forum.qt.io/topic/151706/qml6glsink

The QVideoSink solution mentioned there would probably work in your case though.

[Bio_TICFSL]

(y)