AnsweredAssumed Answered

gstreamer-imx plugins with Qt5 QtQuick/QML scene graph?

Question asked by Aaron Dowdy on Feb 21, 2018
Latest reply on Mar 5, 2018 by Aaron Dowdy

Hey all!

 

I'm stumped trying to implement a camera live stream inside my Qt5 QtQuick2 QML application. The target is an embedded imx6q running yocto linux, and I'm trying various gstreamer-based video implementations.

 

I began with QtMultimedia with gstreamer support turned on, but it only implements a generic playbin solution which was extremely unstable and impossible to troubleshoot.

 

I've moved on to qt-gstreamer approach which lets me control the gstreamer pipeline a bit more, but since my application is QtQuick2 I am currently very limited to using qtquick2videosink as a part of qt-gstreamer. gstreamer-imx provided video sinks don't interact with the Qt application, and either fights for the screen pixels with the application, or rest of the Qt application will take over the screen and I lose my video window. I need the video pixels to 'stay on top' somehow.

So with qtquick2videosink it 'works' inside the QML scene, but I am left with other side effects. I need to use hardware acceleration for video decoding so my pipeline uses imxvpudec (decodes h264) -- but it seems if I use this in conjunction with qtquick2videosink I have creeping performance problems (huge CPU usage and memory bandwidth due to massive copy operations from imxvpu DMA memory to qt scene graph), and also it seems imxvpudec leaves behind cma_alloc() logs in dmesg (with CMA_DEBUG turned on in the linux kernel). This is un-shippable because I need to be able to cleanly take down the pipeline and restart it with different streams, or on disconnect cases.

 

What is the solution here? Has anyone else out there been able to do a Qt application that integrates a video stream in this way, but can utilize any of the hardware acceleration? Can I implement my own version of qtquick2videosink that could properly handle CMA/DMA cleanup operations? Is it possible for my Qt application to play well with a gstreamer-imx supported video sink, taking full advantage of the 'zero copy' pipeline?

 

See here for reference. Also, thank you for any and all consideration!

 

gstreamer-imx/zerocopy.md at master · Freescale/gstreamer-imx · GitHub 

Outcomes