AnsweredAssumed Answered

Full HD Video GPU rendering in QT5

Question asked by Olivier Bourgois on Feb 13, 2017
Latest reply on Feb 22, 2017 by Olivier Bourgois

Has anybody managed to get a 1080p h.264 video displaying at full frame at > 30fps in Qt QML or at the very least Qt GUI with a GPU shader backend on IMX6? 


Currently we are doing some tests with a sabredl with dual-lite, but final product will use a dual with the more powerful GPU core.


Our intent is to do some white point adjustments on the entire application video output using the GPU with GLES2. (The IMX6 does not have dedicated per-channel CLUT HW for color profile adjustments)  Our application must integrate web content via the Qt webengine and video via Qt multimedia.  There are certain modes in the application when the video goes full screen and the path from VPU to GPU seems to be the bottleneck limiting us to approximately 10-12fps.


Our application is in many ways similar to the following Qt demo where an openGL shader effect is applied to a video: Qt-5.7/multimedia/video/qmlVideofx example


We are running without X11 or Wayland, just eglfs as our application is standalone and we probably need the GLES backend to do the color adjustments anyways.


Tests done without Qt  but with the gstreamer1.0-imx-plugins seem to show that the path from VPU decoding to GPU processing is feasible at 1080p.  It seems smooth at least visibly although I need to patch the gstreamer plugin code some more to get some hard FPS numbers.  The parameters used are:


gst-launch-1.0 filesrc location=<path_to_1080p_source> ! qtdemux ! h264parse ! imxvpudec ! imxeglvivsink


There is also a demo in the fsl-gpu-sdk-2.2.2 under GLES2/DirectMultiSamplingVideoYUV which uses the imx-gst1.0-plugin instead by changing prefered provider in yocto.  However when I modify the code for 1080p I am once again left with ~10fps and the video frames are extremely jerky.


In all cases the GlTexDirectVIVMap() and GlTexDirectInvalidateVIV() gles2 extensions seem to be the secret sauce to get zero-copy fast communications between the VPU decoded frames and the GPU as all of these codes integrate these functions,  Including a Qt plugin under qtmultimedia/src/plugins/videonode/imx6/.


So if anybody has gotten something like this working, any pointers as to what kernel version, vivante drivers, distros, Qt versions, patches are required to get this working at 1080p >30fps.


The best results up until now have been obtained with:

1) kernel and vivante drivers from meta-fsl-bsp-release at version imx-4.1.15-1.0.0_ga

2) meta-fsl-arm layer at SHA 35b8b9bd9863de208ab60e33b55f10ee43e2619b

3) upgrade of meta-qt5 layer to 5.7 SHA 81fb771c3f31110e50eebcb004809361fdb28194


I currently do not fully yet fully understand how to work to keep things in sync in yocto.  I tried to keep things as close to the meta-fsl-bsp-release to not break anything, except for meta-qt5 which was simply not working properly with qtmultimedia and qtwebengine in eglfs at the given 5.5 version from the BSP.


I've also noticed some strangeness in the Qt video widget when using the gstreamer1.0-imx-plugins instead of the freescale provided imx-gst1.0-plugin.  The widget shows several frames one after another tiled horizontally in the widget.


We really need to get a decent distro together with all the needed pieces for our custom hardware.  The original idea was to let Qt framework and the BSP manage all the hard multimedia / web / opengl integration and focus on our core application.  This is proving to be much more complex than we originally anticipated as this integration does not seem complete as of today.  And I'm still not sure if we should be basing it on community or freescale to get things working properly.


Any help, suggestions or pointers would be greatly appreciated.