gstreamer-imx plugins with Qt5 QtQuick/QML scene graph?

[adowdy]

Hey all!

I'm stumped trying to implement a camera live stream inside my Qt5 QtQuick2 QML application. The target is an embedded imx6q running yocto linux, and I'm trying various gstreamer-based video implementations.

I began with QtMultimedia with gstreamer support turned on, but it only implements a generic playbin solution which was extremely unstable and impossible to troubleshoot.

I've moved on to qt-gstreamer approach which lets me control the gstreamer pipeline a bit more, but since my application is QtQuick2 I am currently very limited to using qtquick2videosink as a part of qt-gstreamer. gstreamer-imx provided video sinks don't interact with the Qt application, and either fights for the screen pixels with the application, or rest of the Qt application will take over the screen and I lose my video window. I need the video pixels to 'stay on top' somehow.

So with qtquick2videosink it 'works' inside the QML scene, but I am left with other side effects. I need to use hardware acceleration for video decoding so my pipeline uses imxvpudec (decodes h264) -- but it seems if I use this in conjunction with qtquick2videosink I have creeping performance problems (huge CPU usage and memory bandwidth due to massive copy operations from imxvpu DMA memory to qt scene graph), and also it seems imxvpudec leaves behind cma_alloc() logs in dmesg (with CMA_DEBUG turned on in the linux kernel). This is un-shippable because I need to be able to cleanly take down the pipeline and restart it with different streams, or on disconnect cases.

What is the solution here? Has anyone else out there been able to do a Qt application that integrates a video stream in this way, but can utilize any of the hardware acceleration? Can I implement my own version of qtquick2videosink that could properly handle CMA/DMA cleanup operations? Is it possible for my Qt application to play well with a gstreamer-imx supported video sink, taking full advantage of the 'zero copy' pipeline?

See here for reference. Also, thank you for any and all consideration!

gstreamer-imx/zerocopy.md at master · Freescale/gstreamer-imx · GitHub 

[VincasD]

I have similar issue - I need to use imxvpudec_h264 with qmlglsink but performance is horrible:

https://github.com/Freescale/gstreamer-imx/issues/316

One dev said he had managed to write plugin based on `eglvivsink` for avoiding expensive copies:

https://github.com/Freescale/gstreamer-imx/issues/193#issuecomment-575166901

[joanxie]

did you use qt camera component? if you need to use gstreamer sink in the qt,

how about to put imxv4l2sink as m_videoSink in the function setVideoRenderer(QObject *videoOutput) of qgstreamerplayersession.c

[adowdy]

Joan thank you for your comments. I have a more up to date thread with relevant info here.

QtMultimedia gstreamer1.0 rtsp video + wayland + imx6 gst-imx-plugins - memory leaks? 

I am still working on a solution though. I don't think Qt actually allows me to pass in my own plugin video sink, though I tried.

With my system configuration, it's utilizing a specialized imx6 videosurface in Qt for rendering. Seems to respect zero-copy pipeline, but I still see alloc problems when pipeline restarts (I have to be able to restart cleanly for my usecase).

My latest comment on the other thread has my status. I want to find a way for Qt to clean up DMA buffers shared by imxvpudec, but don't currently know how it should be done.

- AD