Full HD Video GPU rendering in QT5

[olivierbourgois]

Has anybody managed to get a 1080p h.264 video displaying at full frame at > 30fps in Qt QML or at the very least Qt GUI with a GPU shader backend on IMX6? 

Currently we are doing some tests with a sabredl with dual-lite, but final product will use a dual with the more powerful GPU core.

Our intent is to do some white point adjustments on the entire application video output using the GPU with GLES2. (The IMX6 does not have dedicated per-channel CLUT HW for color profile adjustments)  Our application must integrate web content via the Qt webengine and video via Qt multimedia.  There are certain modes in the application when the video goes full screen and the path from VPU to GPU seems to be the bottleneck limiting us to approximately 10-12fps.

Our application is in many ways similar to the following Qt demo where an openGL shader effect is applied to a video: Qt-5.7/multimedia/video/qmlVideofx example

We are running without X11 or Wayland, just eglfs as our application is standalone and we probably need the GLES backend to do the color adjustments anyways.

Tests done without Qt  but with the gstreamer1.0-imx-plugins seem to show that the path from VPU decoding to GPU processing is feasible at 1080p.  It seems smooth at least visibly although I need to patch the gstreamer plugin code some more to get some hard FPS numbers.  The parameters used are:

export FB_MULTI_BUFFER=2
gst-launch-1.0 filesrc location=<path_to_1080p_source> ! qtdemux ! h264parse ! imxvpudec ! imxeglvivsink

There is also a demo in the fsl-gpu-sdk-2.2.2 under GLES2/DirectMultiSamplingVideoYUV which uses the imx-gst1.0-plugin instead by changing prefered provider in yocto.  However when I modify the code for 1080p I am once again left with ~10fps and the video frames are extremely jerky.

In all cases the GlTexDirectVIVMap() and GlTexDirectInvalidateVIV() gles2 extensions seem to be the secret sauce to get zero-copy fast communications between the VPU decoded frames and the GPU as all of these codes integrate these functions,  Including a Qt plugin under qtmultimedia/src/plugins/videonode/imx6/.

So if anybody has gotten something like this working, any pointers as to what kernel version, vivante drivers, distros, Qt versions, patches are required to get this working at 1080p >30fps.

The best results up until now have been obtained with:

1) kernel and vivante drivers from meta-fsl-bsp-release at version imx-4.1.15-1.0.0_ga

2) meta-fsl-arm layer at SHA 35b8b9bd9863de208ab60e33b55f10ee43e2619b

3) upgrade of meta-qt5 layer to 5.7 SHA 81fb771c3f31110e50eebcb004809361fdb28194

I currently do not fully yet fully understand how to work to keep things in sync in yocto.  I tried to keep things as close to the meta-fsl-bsp-release to not break anything, except for meta-qt5 which was simply not working properly with qtmultimedia and qtwebengine in eglfs at the given 5.5 version from the BSP.

I've also noticed some strangeness in the Qt video widget when using the gstreamer1.0-imx-plugins instead of the freescale provided imx-gst1.0-plugin.  The widget shows several frames one after another tiled horizontally in the widget.

We really need to get a decent distro together with all the needed pieces for our custom hardware.  The original idea was to let Qt framework and the BSP manage all the hard multimedia / web / opengl integration and focus on our core application.  This is proving to be much more complex than we originally anticipated as this integration does not seem complete as of today.  And I'm still not sure if we should be basing it on community or freescale to get things working properly.

Any help, suggestions or pointers would be greatly appreciated.

[Bio_TICFSL]

hi

Getting reasonable GPU performance at 720p/1080p can be very challenging mainly because the up layers of the software stack eg QT/GTK etc don't always support EGL and therefore resort to software rendering. Although in reality its a lot more complicated than that. If your intent on getting the best performance then you might find yourself rolling a lot of your own code and not heavily relying on ready to use libraries.  Also in order to get 1080p@60 in i.MX6DL, you should try first the VPU unit test and gstreamer with different VPU clock setting such as the following link show:

 https://community.nxp.com/docs/DOC-93587 

regards

[olivierbourgois]

Thank you Bio_TICFSL,

I will try the unit tests and clock adjustements to better gauge the VPU performance. Our final development is on a custom board and distro with IMX6D and I've allready noticed a performance increase relative to the sabreliteDL from 24fps to 33fps with the cinematicexperience demo QT app running at 1080p. However this app does not integrate any live video.

I'm starting to wonder if you are not right in saying that "rolling our own" may be the required route. When we choose the IMX6D as a multimedia processor it was because it has hardware VPU decoding capable of full HD and GPU rendering. This chip is now several years old and with wayland, Android, Qt and the entire world going towards GPU compositing, you'd think there should be something available today to build on.

I'm just wondering how to go about integrating what I need and being able to get it working better than the pros in Qt. I need to mix HTML5 web content, an RTP stream of 1080p@30 video and on top of that GPU based white balance adjustments of the TFT. Does anyone know what are my options? Are we talking direct integration of the chromium browser into an egl/GLES2 app and custom gstreamer plugins? Any other insights?

Thank you to all those who offer their comments to this discussion.

Regards,