Current qt5 multimedia has a special video node for imx6 at "/src/plugins/videonode/imx6" which uses GPU DirectVIV to accelerate the video rendering.
So the rendering performance is almost same as the glimagesink. For check the free run frame rate, the qt multimedia source need change a little bit to set sink sync as false and calculate the real video rendering rate.
Set sync as false:
In "/src/plugins/gstreamer/mediaplayer/qgstreamerplayersession.cpp"
This is a widget application, not a QML application. It use QAudioRecorder as its background worker. By default it will use pulsesrc and lamemp3enc to recorder audio as mkv file at user home folder with name of clip_0001.mkv, clip_0002.mkv, and so on.
If need special pipeline for recording, example use imxmp3enc to record mp3 files, we need change the function GstElement *QGstreamerCaptureSession::buildEncodeBin in “src/plugins/gstreamer/mediacapture/qgstreamercapturesession.cpp” as well as QGstreamerMediaContainerControl(QObject *parent) in “src/plugins/gstreamer/mediacapture/qgstreamermediacontainercontrol.cpp”
If you want use overlaysink as the preview sink, then use following patch:
0003-use-overlaysink-as-the-preview-sink.patch
Both applications need to set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC to use imxv4l2src
If camera device is other than /dev/video0, set environment variable QT_GSTREAMER_CAMERABIN_VIDEOSRC_DEVICE to the right camera device, such as "/dev/video1" , for example:
By default, the “post-preview” feature of camerabin was disabled by the patches provided, since the imxv4l2src can’t work with “post-preview” internal “videoconvert” element.
To enable “post-preview” feature, one need uncomment following line
I found problems on the current Yocto BSP release with Qt5 on Xwayland backend.
Looks like it's not easy to support Video playback or Camera viewfinder in the Xwayland backend Qt's widget. In the Qt xwayland, looks like we can only chose the waylandsink as the Gstreamer sink for video/camera, but the QtMultimedia's QGstreamerVideoOverlay does not support wayland backend video sink. I tried to hack the src/gsttools/qgstreamervideooverlay.cpp file to add waylandsink support, but the waylandsink plugin in the gst-plugins-bad told me: "waylandsink cannot use an externally-supplied surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext".
Ok, I finally work it out by myself. The Qtmultimedia VideoOverlay does not support wayland platform other than xcb. But it provides a qgstvideorendersink software gstreamer sink for it's QVideoWidget rendering. I try to enable this on i.MX6UL, and add the PXP hardware acceleration for CSC on the preview pipeline. Here're some patches:
Though the camerbin2 viewfiner can be accelerate by PXP on i.MX6UL, but the performance of the camerabin2 and qgstvideorendersink is poor, which would cause around 60% CPU loading when camera example running preview. I would not suggest to use QCamera on the i.MX6UL/ULL platform. But writing own gst appsink to get frame is a better solution.
Is there a way to use other imx plugins for hardware encoding/decoding and deinterlacing? Or I should make some changes in gstreamer or qtmultimedia sources by myself?
May be there are patches or other related resources?
Which platform do you want? The gstreamer en/decoding plugins should be already supported by our official release, but you need to check which QTMultimedia widget or component do you want to use, and which backend of QT do you support (wayland or X11? ). Then we can tell what kind of jobs to do.
I use iMX6q sabresd with linux 4.1.36. Qt 5.7, gstreamer 1.0. gstreamer1.0-plugins-imx - 0.12.2-r0 No wayland or X will be used. Qt application starts with --platform eglfs.
The first problem - deinterlacing. What should I do? write own qt video surface or filter or qt/gst plugin or anything else? I planned to use QML VideoOutput but can use any other component/class.
Second - encoding to h264. Is hardware encoding supported in QtMultimedia? Can I manipulate encoder parameters from qt?
For this two problems, could you make it more specific?
Deinterlacing - you have a video source, and use some class like QMediaPlayer to play, but need a deinterlace process, right? Multimedia | Qt Multimedia 5.8
Encoding - You would recording that from a Camera or a TV-In? QMediaRecodering would search the h264 encoder that gst-plugin has. Not sure it would use the VPU or SW one. If default is SW one, a little hack is needed.
My video source is camera - interlaced bt656, UYVY. I need to convert it to progressive RGB to show it on display. For that purpose I use gstreamer pipelien:
gst-launch-1.0 imxv4l2videosrc ! imxipuvideotransform deinterlace=true ! tee name=t ! queue ! imxipuvideosink use-vsync=true t.
On demand, I connect to "tee" other "bins" - for encoding to h264 and write stream to file; for encoding to jpeg and write one frame to file; and streaming h264-encoded video by rtsp-server. Can I do something like this with Qt Multimedia (HW-accelerated)?
Am I right, QT_VIDEONODE environment variable should be set to specify default gst video plugin (for hw encoding/decoding)?
I tried to add imxipuvideotransform plugin to "viewfinder-filter" parameter of camerabin in camerabinsession.cpp but get interlaced video with low frame rate on display and high memory usage.
I want to add a deinterlacing process to camerabin to get progressive frames for writing to files and streaming to network.
So by using the gst-launch mentioned above, you can show on display correctly right?
I'm not sure Qtmultimedia really has the component that can be used to streaming a encoded stream to RTSP server, do you find? If not, I really recommend that you use GST API to create pipeline by yourself in the QT application, and preview it by grep buffers from queue.
For the deinterlace, I think you should hack the "src-videoconvert" or "vfbin-videoconvert" in the camerbin2 viewfinder source, but not "viewfinder-filter". camerbin2 is a very complicated bin, recording and viewfinder go different pads, so maybe do deinterlace in the src-xx pad is the best choice.
i'm facing the same issue . Using Qt5.4.2 on imx53 with gstreamer 0.10.
***When i use qt camera component, the video stream is quite acceptable (no interlacing, low latency) but the cpu is loaded at 90% including 65% for QSGRenderThread
How can i reduce this cpu charge?
***When i use media player component by setting uri handling to tvsrc or imxv4lsrc , i got bad performances( latency, interlacing) and 66 % of cpu.
for the latency, i tried to set the property sync to false in qvideosurfacegstsink.cpp( it's where qt's sink is made apparently) but nothing changes. when i put imxv4l2sink as m_videoSink in the function setVideoRenderer(QObject *videoOutput) of qgstreamerplayersession.cpp, the video stream is displayed on fullscreen and my qml interface is just behind.
to add deinterlacing, i tried to use freescale plugins mfw_ipucsc combine with mfw_deinterlacer in qgstreamerplayersession.cpp but i got
Error: "Internal data flow error."
what's about this error? concerning mfw_ipucsc, it replaces the ffmpegcolorspace plugin
QLibraryPrivate::loadPlugin failed on "/usr/lib/qt5/plugins/mediaservice/libgstmediaplayer.so" : "Cannot load library /usr/lib/qt5/plugins/mediaservice/libgstmediaplayer.so: (libqgsttools_p.so.1: cannot open shared object file: No such file or directory)"
