How to use qtmultimedia(QML) with Gstreamer 1.0

Document created by Yuri Muhin Employee on Mar 1, 2016Last modified by Yifang Guo on Mar 22, 2016
Version 12Show Document
  • View in full screen mode

1. Enable QtMultimedia

1) In source/meta-qt5/recipes-qt/qt5/qtmultimedia_git.bb, add “gstreamer” at the end of “PACKAGECONFIG ??=” .

     For example on  L3.14.52  bsp, build  x11  backend:

 

PACKAGECONFIG ??= "${@bb.utils.contains('DISTRO_FEATURES', 'alsa', 'alsa', '', d)} \

                   ${@bb.utils.contains('DISTRO_FEATURES', 'pulseaudio', 'pulseaudio', '', d)} \

                   gstreamer"


2)
Build qtmultimedia:  bitbake qtmultimedia

3) Copy the built package to target file system.

     For example :
cp  -a
./build-x11/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/qtmultimedia/5.5.0+gitAUTOINC+46a83d5b86-r0/image/usr/   /usr  

export  DISPLAY=:0

Video example application located at :
/usr/share/qt5/examples/multimedia/video/qmlvideo

 

2. Enable Qt Logging Info

export  QT_LOGGING_RULES=<category>[.type]=ture|false

     For example:

Turn on all debug logging:
     export QT_LOGGING_RULES=*.debug=true

Turn on all logging level of all multimedia module:
     export QT_LOGGING_RULES=qt.multimedia.*=true

 

3. Performance Checking

Gstreamer: overlaysink fps=51.585

     gst-launch-1.0 playbin uri=file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi  video-sink=”overlaysink sync=false”

Gstreamer: glimagesink fps=43.850

     gst-launch-1.0 playbin uri=file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi video-sink="glimagesink sync=false"

Qtmultimedia: fps=45.201

     ./qmlvideo  -url file:///H264_AVC_1080p_30fps_27Mbps_mp3.avi -hide-perf

Current qt5 multimedia has a special video node for imx6 at     "/src/plugins/videonode/imx6" which uses GPU DirectVIV to accelerate the video rendering.

So  the rendering performance is almost same as the glimagesink. For check the free run frame rate, the qt multimedia source need change a little bit to set sink sync as false and calculate the real video rendering rate.

Set sync as false:

In "/src/plugins/gstreamer/mediaplayer/qgstreamerplayersession.cpp"

QGstreamerPlayerSession::playbinNotifySource():

-       g_object_set(G_OBJECT(self->m_videoSink), "sync", !self->m_isLiveSource, NULL);

+       g_object_set(G_OBJECT(self->m_videoSink), "sync", false/!self->m_isLiveSource/, NULL);

Get fps:

In "/src/gsttools/qgstvideorenderersink.cpp"

+#include <QTime>

+gint g_frame_showed;

+QTime g_time;

+

bool QVideoSurfaceGstDelegate::start(GstCaps *caps)

+   g_frame_showed = 0;

+   g_time.restart();

void QVideoSurfaceGstDelegate::stop()

+   printf(">>>>> fps=%f:\n", (gdouble)g_frame_showed*1000/g_time.elapsed());

bool  QVideoSurfaceGstDelegate::handleEvent(QMutexLocker *locker)

           
const bool rendered = m_activeRenderer->present(m_surface, buffer);

+   g_frame_showed++;

Or use the attached patch: qtmultimedia_fps_check.diff

 

4. Streaming Playing:  HTTP RTP RTSP

HTTP:

./qmlvideo -url http://10.192.225.205/streaming/http/H264_HP13_1280x720_29_2850_AAC_LC_44_64_NobodyMTV.mp4

RTSP:

./qmlvideo -url rtsp://10.192.225.205/MPEG4_SP1_720x576_30fps_282_AMRNB_TVC.mp4

RTP:

QT mediaplayer use gstreamer playbin as the backend, playbin don’t support rtp address as uri. So QT mediaplayer can’t support RTP playing directly.

 

5. Audio Recording

/usr/share/qt5/examples/multimedia/audiorecorder/audiorecorder

This is a widget application, not a QML application. It use QAudioRecorder as its background worker. By default it will use pulsesrc and lamemp3enc to recorder audio as mkv file at
user home folder with name of clip_0001.mkv, clip_0002.mkv, and so on.

If need special pipeline for recording, example use imxmp3enc to record mp3 files, we need change the function GstElement *QGstreamerCaptureSession::buildEncodeBin
in  src/plugins/gstreamer/mediacapture/qgstreamercapturesession.cpp
as well as QGstreamerMediaContainerControl(QObject *parent) in src/plugins/gstreamer/mediacapture/qgstreamermediacontainercontrol.cpp

 

6. Camera

Widgets Application:

/usr/share/qt5/examples/multimediawidgets/camera/camera

QML Application:

/usr/share/qt5/examples/multimedia/declarative-camera/declarative-camera

Qt Multimedia has some bugs on L3.14.52 with QCamera for recording. Use the attached patches:

0001-gstreamer-fix-camerabin-not-negotiated-error.patch

0001-QCamera-can-t-set-recording-container-format-video-a.patch
0002-QtCamera-change-default-encoding-parameters-to-use-N.patch

If you want use overlaysink
as the preview sink, then use following patch:

0003-use-overlaysink-as-the-preview-sink.patch

 

Both applications need to set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC to use imxv4l2src

If camera device is other than /dev/video0, set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE to the right camera device, such as
"/dev/video1" ,  for example:

export QT_GSTREAMER_CAMERABIN_VIDEOSRC=imxv4l2src
export DISPLAY=:0

export QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE="/dev/video1"

 

By default, the “post-preview” feature of camerabin was disabled by the patches provided, since the imxv4l2src can’t work with “post-preview” internal “videoconvert” element.

To enable “post-preview”  feature, one need uncomment following line

//g_object_set(G_OBJECT(m_camerabin), POST_PREVIEWS_PROPERTY, TRUE, NULL);

in /qtmultimedia/src/plugins/gstreamer/camerabin/camerabinsession.cpp,

Function:

CameraBinSession::CameraBinSession(GstElementFactory *sourceFactory, QObject *parent)

And

Change “videoconvert” to “imxvideoconvert_g2d” or “imxvideoconvert_ipu” or “imxvideoconvert_pxp” in line :

csp = gst_element_factory_make ("videoconvert", "preview-vconv");

of  file /gstreamer1.0-plugins-bad/gst-libs/gst/basecamerabinsrc/gstcamerabinpreview.c” 

function:

GstCameraBinPreviewPipelineData *gst_camerabin_create_preview_pipeline (GstElement * element,   GstElement * filter)

 


4 people found this helpful

Attachments

Outcomes