I've tried a few times now to phrase this question in a way that can be understood, and I figured an updated version might get some response, since I have new information and a bit more direction.
I need to be able to implement a streaming video player inside my Qt5.6 Qt app using QtQuick2 view. The proposed solution is to use QtMultimedia... the frontend is QML ( import QtMultimedia 5.0 , and use MediaPlayer {} and VideoOutput {} ) provided to supply an rtsp://url to an IP camera stream.
The platform details are as follows:
imx6q target with embedded display
yocto-based embedded linux base
wayland (not XCB, and not EGLFS)
Qt 5.6
QtMultimedia configured to use gstreamer 1.0
OpenEmbedded Layer Index - qtmultimedia
gstreamer 1.0 plugins
OpenEmbedded Layer Index - gstreamer1.0-plugins-imx
To get any form of reasonable performance I need to utilize these gstreamer imx plugins in the pipeline.
My problem is, I must use QtMultimedia to integrate into the QML scenegraph, and it generates its own pipeline through very non-obvious means. I was able to hack QtMultimedia to display their 'playBin' pipeline contents shown here, but still doesn't show the videoSink at the end. (you can open image in its own tab to zoom in and view the details)
I can't trace in the qtmultimedia source code how this video sink comes about, but after enabling debug logging from QtMultimedia it does mention "qgstvideorenderersink0" in a terse way, indicating it's a gstreamer element. qgstvideorenderersink doesn't appear anywhere in gst-inspect-1.0 so Qt is deriving this somehow else, but I don't know from where or how it behaves. It seems to be utilizing some hardware acceleration because the performance looks good while the video stream is playing.
THE PROBLEM --
This pipeline 'works' for a limited amount of time, then fails badly. I absolutely need it to be able to recover cleanly. Firstly, periodically the pipeline is producing some h264 parsing errors:
Error: "Error parsing H.264 stream"
NOTE - every time this happens, we see the video flicker, then resume playing. I also begin noticing that various pipeline elements have new enumerators after them, as seen in the image above (e.g. rtpsession8, etc). If I showed a printout of the original pipeline, it shows 0 indexed instances of every plugin, so it's like we're jumping to new pipeline copies (and maybe not cleaning up properly).
After several of these periodic H.264 errors over a period of say, up to 60 minutes, The application reaches an unrecoverable error:
GLib-ERROR **: ../../glib-2.46.2/glib/gmem.c:100: failed to allocate 65611 bytes
I suspect my memory allocation problem has something to do with so called 'zero-copy operations' not being respected by whatever videoSink QtMultimedia provides... This page is the only shred of information I can come up with.
gstreamer-imx/zerocopy.md at master · Freescale/gstreamer-imx · GitHub
I produced a similar problem testing with a very old (third party) qt-gstreamer implementation which provides the plugin qtquick2videosink instead of qt's own inhouse solution. I could get the pipeline to run for a very long time with this one (no H.264 parsing errors that appear to reallocate a whole new pipeline automatically), but if I try to clean up the pipeline and assemble a different one myself with qtquick2videosink from qt-gstreamer (such as is needed to change the source to a different rtsp URI), I experience leaks which results in a similar memory allocation failure in imxvpudec plugin (CMA/DMA allocs).
If I take the exact same pipeline but direct to imxeglvivsink or imxipuvideosink instead, I can allocate, then clean up the pipeline indefinitely with no issue of vpu running out of CMA/DMA alloc space. It's like it 'knows' how to clean up these CMA/DMA buffers properly only with imx provided video sinks.
Here's a similar working gst-launch test that runs on my platform:
gst-launch-1.0 rtspsrc location=rtsp://admin:admin@10.50.15.254:554/mpeg4cif name=source ! queue ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideosink
NOTE - also works fine with imxeglvivsink
I'd love to be able to just use imxeglvivsink OR imxipuvideosink directly, but I can't figure out how to get it to integrate into my Qt app scene correctly. I've tried forcing QtMultimedia to use this in place of its own weird sink by setting this environment variable:
I can see that Qt appears to recognize this variable in their source code, but the approach isn't working for me. It doesn't appear to solve the memory allocation problem, and the pipeline debug output still reports it's using qgstvideorenderersink0. Is there a way to actually use a supported video sink and have it work inside the Qt scene graph the way qgstvideorenderersink does?
I also have an application where I build the pipeline up manually and direct to an imx videosink, it seems to blit to the screen ok the first time I tell it to play, but stops drawing to the screen when I try to interact at all with my Qt application. Is there a way to force imxeglvivsink 'on top' all the time in wayland? I think if I could do this I could get away with using the 'supported' imx sinks, and just have the video render over the top of the rest of the Qt app.
So there you have it. Only when I introduce QtQuick2 wayland compatible video sinks do I somehow break with the zero-copy pipeline, and then I land in trouble because it leads to unrecoverable leaks. This is my theory, anyway. Even if it proves true, I'm not sure what the best approach to solving this is. I am in depserate need of help at this point, any and all is seriously appreciated! Thank you!
- AD
