How to use qtmultimedia(QML) with Gstreamer 1.0

[#1]

1. Enable QtMultimedia

1) In “source/meta-qt5/recipes-qt/qt5/qtmultimedia_git.bb”, add “gstreamer” at the end of “PACKAGECONFIG ??=” .

     For example on  L3.14.52  bsp, build  x11  backend:

PACKAGECONFIG ??= "${@bb.utils.contains('DISTRO_FEATURES', 'alsa', 'alsa', '', d)} \

                   ${@bb.utils.contains('DISTRO_FEATURES', 'pulseaudio', 'pulseaudio', '', d)} \

                   gstreamer"

2) Build qtmultimedia:  bitbake qtmultimedia

3) Copy the built package to target file system.

     For example :
cp  -a ./build-x11/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/qtmultimedia/5.5.0+gitAUTOINC+46a83d5b86-r0/image/usr/   /usr  

export  DISPLAY=:0

Video example application located at :
/usr/share/qt5/examples/multimedia/video/qmlvideo

2. Enable Qt Logging Info

export  QT_LOGGING_RULES=<category>[.type]=ture|false

     For example:

Turn on all debug logging:
     export QT_LOGGING_RULES=*.debug=true

Turn on all logging level of all multimedia module:
     export QT_LOGGING_RULES=qt.multimedia.*=true

3. Performance Checking

Gstreamer: overlaysink fps=51.585

     gst-launch-1.0 playbin uri=file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi  video-sink=”overlaysink sync=false”

Gstreamer: glimagesink fps=43.850

     gst-launch-1.0 playbin uri=file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi video-sink="glimagesink sync=false"

Qtmultimedia: fps=45.201

     ./qmlvideo  -url file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi -hide-perf

Current qt5 multimedia has a special video node for imx6 at     "/src/plugins/videonode/imx6" which uses GPU DirectVIV to accelerate the video rendering.

So  the rendering performance is almost same as the glimagesink. For check the free run frame rate, the qt multimedia source need change a little bit to set sink sync as false and calculate the real video rendering rate.

Set sync as false:

In "/src/plugins/gstreamer/mediaplayer/qgstreamerplayersession.cpp"

QGstreamerPlayerSession::playbinNotifySource():

-       g_object_set(G_OBJECT(self->m_videoSink), "sync", !self->m_isLiveSource, NULL);

+       g_object_set(G_OBJECT(self->m_videoSink), "sync", false/!self->m_isLiveSource/, NULL);

Get fps:

In "/src/gsttools/qgstvideorenderersink.cpp"

+#include <QTime>

+gint g_frame_showed;

+QTime g_time;

+

bool QVideoSurfaceGstDelegate::start(GstCaps *caps)

…

+   g_frame_showed = 0;

+   g_time.restart();

…

void QVideoSurfaceGstDelegate::stop()

…

+   printf(">>>>> fps=%f:\n", (gdouble)g_frame_showed*1000/g_time.elapsed());

…

bool  QVideoSurfaceGstDelegate::handleEvent(QMutexLocker *locker)

…

           
const bool rendered = m_activeRenderer->present(m_surface, buffer);

+   g_frame_showed++;

…

Or use the attached patch: qtmultimedia_fps_check.diff

4. Streaming Playing:  HTTP RTP RTSP

HTTP:

./qmlvideo -url http://10.192.225.205/streaming/http/H264_HP13_1280x720_29_2850_AAC_LC_44_64_NobodyMTV.mp4

RTSP:

./qmlvideo -url rtsp://10.192.225.205/MPEG4_SP1_720x576_30fps_282_AMRNB_TVC.mp4

RTP:

QT mediaplayer use gstreamer playbin as the backend, playbin don’t support rtp address as uri. So QT mediaplayer can’t support RTP playing directly.

5. Audio Recording

/usr/share/qt5/examples/multimedia/audiorecorder/audiorecorder

This is a widget application, not a QML application. It use QAudioRecorder as its background worker. By default it will use pulsesrc and lamemp3enc to recorder audio as mkv file at
user home folder with name of clip_0001.mkv, clip_0002.mkv, and so on.

If need special pipeline for recording, example use imxmp3enc to record mp3 files, we need change the function GstElement *QGstreamerCaptureSession::buildEncodeBin
in  “src/plugins/gstreamer/mediacapture/qgstreamercapturesession.cpp” as well as QGstreamerMediaContainerControl(QObject *parent) in “src/plugins/gstreamer/mediacapture/qgstreamermediacontainercontrol.cpp”

6. Camera

Widgets Application:

“/usr/share/qt5/examples/multimediawidgets/camera/camera”

QML Application:

“/usr/share/qt5/examples/multimedia/declarative-camera/declarative-camera”

Qt Multimedia has some bugs on L3.14.52 with QCamera for recording. Use the attached patches:

0001-gstreamer-fix-camerabin-not-negotiated-error.patch

0001-QCamera-can-t-set-recording-container-format-video-a.patch
0002-QtCamera-change-default-encoding-parameters-to-use-N.patch

If you want use overlaysink
as the preview sink, then use following patch:

0003-use-overlaysink-as-the-preview-sink.patch

Both applications need to set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC to use imxv4l2src

If camera device is other than /dev/video0, set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE to the right camera device, such as
"/dev/video1" ,  for example:

export QT_GSTREAMER_CAMERABIN_VIDEOSRC=imxv4l2src
export DISPLAY=:0

export QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE="/dev/video1"

By default, the “post-preview” feature of camerabin was disabled by the patches provided, since the imxv4l2src can’t work with “post-preview” internal “videoconvert” element.

To enable “post-preview”  feature, one need uncomment following line

//g_object_set(G_OBJECT(m_camerabin), POST_PREVIEWS_PROPERTY, TRUE, NULL);

in “/qtmultimedia/src/plugins/gstreamer/camerabin/camerabinsession.cpp”,

Function:

CameraBinSession::CameraBinSession(GstElementFactory *sourceFactory, QObject *parent)

And

Change “videoconvert” to “imxvideoconvert_g2d” or “imxvideoconvert_ipu” or “imxvideoconvert_pxp” in line :

csp = gst_element_factory_make ("videoconvert", "preview-vconv");

of  file “/gstreamer1.0-plugins-bad/gst-libs/gst/basecamerabinsrc/gstcamerabinpreview.c” 

function:

GstCameraBinPreviewPipelineData *gst_camerabin_create_preview_pipeline (GstElement * element,   GstElement * filter)

[xinyu_chen]

I found problems on the current Yocto BSP release with Qt5 on Xwayland backend.

Looks like it's not easy to support Video playback or Camera viewfinder in the Xwayland backend Qt's widget. In the Qt xwayland, looks like we can only chose the waylandsink as the Gstreamer sink for video/camera, but the QtMultimedia's QGstreamerVideoOverlay does not support wayland backend video sink. I tried to hack the src/gsttools/qgstreamervideooverlay.cpp file to add waylandsink support, but the waylandsink plugin in the gst-plugins-bad told me: "waylandsink cannot use an externally-supplied surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext".

Anybody have tried this? Any suggestion?

[xinyu_chen]

Ok, I finally work it out by myself. The Qtmultimedia VideoOverlay does not support wayland platform other than xcb. But it provides a qgstvideorendersink software gstreamer sink for it's QVideoWidget rendering. I try to enable this on i.MX6UL, and add the PXP hardware acceleration for CSC on the preview pipeline. Here're some patches:

1. gstreamer1.0-plugins-bad: hack the camerbin2 to insert the imxvideoconvert_pxp into pipeline for preview: meta-qt-bsp/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad at imx-4.1-krogoth · muddog/meta-q... 
2. qtmultimedia: rework around for the synchronization problem when using PXP as csc videoconvert, which do convert faster than the QVideoWidget can do paintEvent, and cause qgstvideorendersink render failed issue: meta-qt-bsp/recipes-qt/qt5/qtmultimedia at imx-4.1-krogoth · muddog/meta-qt-bsp · GitHub 

Though the camerbin2 viewfiner can be accelerate by PXP on i.MX6UL, but the performance of the camerabin2 and qgstvideorendersink is poor, which would cause around 60% CPU loading when camera example running preview. I would not suggest to use QCamera on the i.MX6UL/ULL platform. But writing own gst appsink to get frame is a better solution.

Thanks.

[maximkuk]

Hello!

Is there a way to use other imx plugins for hardware encoding/decoding and deinterlacing? Or I should make some changes in gstreamer or qtmultimedia sources by myself? 

May be there are patches or other related resources?

[xinyu_chen]

Hi Maxim

Which platform do you want? The gstreamer en/decoding plugins should be already supported by our official release, but you need to check which QTMultimedia widget or component do you want to use, and which backend of QT do you support (wayland or X11? ). Then we can tell what kind of jobs to do.

[maximkuk]

Thanks for your reply!

I use iMX6q sabresd with linux 4.1.36.
Qt 5.7,
gstreamer 1.0.
gstreamer1.0-plugins-imx - 0.12.2-r0
No wayland or X will be used. 
Qt application starts with --platform eglfs.

The first problem - deinterlacing. What should I do? write own qt video surface or filter or qt/gst plugin or anything else? I planned to use QML VideoOutput but can use any other component/class.

Second - encoding to h264. Is hardware encoding supported in QtMultimedia? Can I manipulate encoder parameters from qt?

[xinyu_chen]

Hi Maxim

For this two problems, could you make it more specific?

Deinterlacing - you have a video source, and use some class like QMediaPlayer to play, but need a deinterlace process, right?  Multimedia | Qt Multimedia 5.8 

Encoding - You would recording that from a Camera or a TV-In? QMediaRecodering would search the h264 encoder that gst-plugin has. Not sure it would use the VPU or SW one. If default is SW one, a little hack is needed.

[maximkuk]

My video source is camera - interlaced bt656, UYVY. I need to convert it to progressive RGB to show it on display. For that purpose I use gstreamer pipelien:

gst-launch-1.0 imxv4l2videosrc ! imxipuvideotransform deinterlace=true ! tee name=t ! queue ! imxipuvideosink use-vsync=true t.

On demand, I connect to "tee" other "bins" - for encoding to h264 and write stream to file; for encoding to jpeg and write one frame to file; and streaming h264-encoded video by rtsp-server. Can I do something like this with Qt Multimedia (HW-accelerated)?

I have built qtmultimedia plugins:

├── audio
│   └── libqtmedia_pulse.so
├── mediaservice
│   ├── libgstaudiodecoder.so
│   ├── libgstcamerabin.so
│   ├── libgstmediacapture.so
│   └── libgstmediaplayer.so
├── playlistformats
│   └── libqtmultimedia_m3u.so
└── video
         └── videonode
                  ├── libeglvideonode.so
                  └── libimx6vivantevideonode.so

Am I right, QT_VIDEONODE environment variable should be set to specify default gst video plugin (for hw encoding/decoding)?

I tried to add imxipuvideotransform plugin to "viewfinder-filter" parameter of camerabin in camerabinsession.cpp but get interlaced video with low frame rate on display and high memory usage.

I want to add a deinterlacing process to camerabin to get progressive frames for writing to files and streaming to network.

[xinyu_chen]

Hi Maxim

So by using the gst-launch mentioned above, you can show on display correctly right?

I'm not sure Qtmultimedia really has the component that can be used to streaming a encoded stream to RTSP server, do you find? If not, I really recommend that you use GST API to create pipeline by yourself in the QT application, and preview it by grep buffers from queue.

For the deinterlace, I think you should hack the "src-videoconvert" or "vfbin-videoconvert" in the camerbin2 viewfinder source, but not "viewfinder-filter". camerbin2 is a very complicated bin, recording and viewfinder go different pads, so maybe do deinterlace in the src-xx pad is the best choice.

Thanks

[maximkuk]

Thanks for your advice! I will look at camerabin sources, but I think, GST API is really better way in my case.

For RTSP streaming I use gst-rtsp-server (gstreamer1.0-rtsp-server package in yocto).

[lucdagnon]

Hi

i'm facing the  same issue . Using Qt5.4.2 on imx53  with gstreamer 0.10.

***When i use qt camera component, the video stream is quite acceptable (no interlacing, low latency) but the cpu is loaded at 90%  including 65% for QSGRenderThread

     How can i reduce this cpu charge?

***When i use media player component by setting uri handling to tvsrc or imxv4lsrc , i got bad performances( latency, interlacing) and 66 % of cpu.

      for the latency, i tried to set the property sync to false in qvideosurfacegstsink.cpp(  it's where qt's sink is made apparently) but nothing changes. when i put imxv4l2sink as m_videoSink in the function setVideoRenderer(QObject *videoOutput) of qgstreamerplayersession.cpp, the video stream is displayed on fullscreen and my qml interface is just behind.

      to add deinterlacing, i tried to use freescale plugins mfw_ipucsc combine with mfw_deinterlacer in qgstreamerplayersession.cpp but i got

Error: "Internal data flow error."

what's about this error? concerning mfw_ipucsc, it replaces the ffmpegcolorspace plugin

those plugins work with gst-launch:

gst-launch tvsrc ! mfw_ipucsc ! mfw_deinterlacer ! imxv4l2sink sync=false

[Nian8113]

Hello,

I made a QT application for video play.

I got the problem:

QLibraryPrivate::loadPlugin failed on "/usr/lib/qt5/plugins/mediaservice/libgstmediaplayer.so" : "Cannot load library /usr/lib/qt5/plugins/mediaservice/libgstmediaplayer.so: (libqgsttools_p.so.1: cannot open shared object file: No such file or directory)"

But the library are existed.

I'm using Qt 5.6.2 on imx6q with gstreamer 1.0.

How should I fix this problem?

Thanks.

[weihengliu]

Hello

can you provide the download address of "patchesIn3.14.52.zip" again?