i.MX6 QMediaPlayer does qtvideosink use IPU?

[MicMoba]

I want to use qt5 with qml to show a dual-stream.

main.qml:

import QtQuick.Window 2.1
import QtQuick 2.0
import QtMultimedia 5.12

Window {
    id: window
    visible: true
    width: 800
    height: 480
    x: 0
    y: 0

    Row {
        anchors.fill: parent

        MediaPlayer {

              id: mediaplayer_cam1
              source: "gst-pipeline: udpsrc port=10002 caps=application/x-rtp,payload=96 ! rtph264depay ! h264parse ! imxvpudec ! qtvideosink"
              autoPlay: true
         }
         VideoOutput {
            id: videooutput_cam1

            width: 400

            height: 480

            rotation: 0

           fillMode: Image.Stretch
            source: mediaplayer_cam1
         }

       MediaPlayer {

            id: mediaplayer_cam2
            source: "gst-pipeline: udpsrc port=5002 caps=application/x-rtp,payload=96 ! rtph264depay ! h264parse ! imxvpudec ! qtvideosink"
            autoPlay: true
         }
         VideoOutput {
            id: videooutput_cam2

            width: 400

            height: 480

            rotation: 0

           fillMode: Image.Stretch

            source: mediaplayer_cam2
         }

    }

}

In principle it works. I started the qt application out of my serial terminal and then I will see the two streams in a split-screen on my display.

But when I exit the application over ctrl+c I got a lot of warnings and traces like this:

^C------------[ cut here ]------------
WARNING: CPU: 3 PID: 1986 at /home/mylinux/yocto/warrior/my-bsp-platform/build-fb/tmp/work-shared/mx6qdl-module/kernel-source/mm/page_alloc.c:7658 cma_release+0x6c/0x94
375 pages are still in use!
Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bluetooth ecdh_generic mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc ipu_fg_overlay_sdc  v4l2_int_device imx_sdma galcore(O)
CPU: 3 PID: 1986 Comm: udpsrc1:src Tainted: G        W  O    4.14.126 #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[<8010e78c>] (unwind_backtrace) from [<8010acfc>] (show_stack+0x10/0x14)
[<8010acfc>] (show_stack) from [<80948ee8>] (dump_stack+0x78/0x8c)
[<80948ee8>] (dump_stack) from [<8012d52c>] (__warn+0xe4/0x100)
[<8012d52c>] (__warn) from [<8012d1a4>] (warn_slowpath_fmt+0x38/0x48)
[<8012d1a4>] (warn_slowpath_fmt) from [<80215b64>] (cma_release+0x6c/0x94)
[<80215b64>] (cma_release) from [<80113e48>] (__arm_dma_free.constprop.3+0xe4/0x120)
[<80113e48>] (__arm_dma_free.constprop.3) from [<80725658>] (vpu_free_dma_buffer+0xa0/0xd4)
[<80725658>] (vpu_free_dma_buffer) from [<80726cb0>] (vpu_release+0x15c/0x2f0)
[<80726cb0>] (vpu_release) from [<8021b358>] (__fput+0x84/0x1d0)
[<8021b358>] (__fput) from [<80147b40>] (task_work_run+0x90/0xac)
[<80147b40>] (task_work_run) from [<8012feac>] (do_exit+0x490/0xab8)
[<8012feac>] (do_exit) from [<80131140>] (do_group_exit+0x3c/0xbc)
[<80131140>] (do_group_exit) from [<8013a93c>] (get_signal+0xfc/0x76c)
[<8013a93c>] (get_signal) from [<8010a46c>] (do_signal+0x64/0x48c)
[<8010a46c>] (do_signal) from [<8010aa48>] (do_work_pending+0xa4/0xb4)
[<8010aa48>] (do_work_pending) from [<8010788c>] (slow_work_pending+0xc/0x20)
---[ end trace b532011085fbc682 ]---
------------[ cut here ]------------

As longer the qt application runs as more messages were printed out.

I read that qtvideosink is not using the IPU. So I have two questions:

What causes the traces when I cancel the application?

How can I use the hardware acceleration for display my streams?

Thanks

Michael

[VincasD]

It seems you need to set QT_GSTREAMER_USE_OPENGL_PLUGIN=1 for your application so that qtvideosink would attempt to use OpenGL rendering. But, it does not work on IMX, because of unsupported format:

https://codereview.qt-project.org/c/qt/qtmultimedia/+/276809

So we need to either some fast VPU-base pixel format conversation, or that vpudec would itself provide RGB format, OR that Qt developers would implement NV12/I420 format in qtvideosink..? I imagine that's doable, because glimagesink works fine...

[vaudoitlaurent]

Hi,

i'm working on a imx6 DualPlus custom board,

using the meta-fsl-bsp warrior release (kernel 4.19).

On this we apply a nxp patch to allow up to 4 camera on the mipi csi bus.

We plan to develop a Qt5 application, which must be able to display up to 4 camera, in addition to other features (automotive stuff).

We have a first test application, which use gstreamer pipeline like this: gst-pipeline: imxv4l2src device=/dev/videoX ! qtvideosink

This works well, but cpu load seems high.

I've found this post, which ask the question of IPU usage with QT.

If i understand well, using qtvideosink is not the best way to use the imx6 hardware features like IPU.

It is best to take example of the qmlvideo example, which use camerabin.

This leads me to have some other question, to understand well.

With camerabin, we can set up the input plugin with

export QT_GSTREAMER_CAMERABIN_VIDEOsrc=imxv4l2src

but what about the output plugin, does it use the imxv4l2sink or other?

Is it the correct way to take the best performance from the imx6? Are we sure the imx6 hardware is used, instead of doing everything by software?

Could you explain or point on a link who could describe how it works?

Thanks in advance for your support

Best regards

Laurent

[joanxie]

why don't you use qmlvideo which is supported in the current bsp? refer to your log file, I dont think you need to care about this warning

[MicMoba]

Joan,
what do you mean with "qmlvideo wich is supported in the current bsp?"
First I thought qmlvideo is Qt element like Mediaplayer or VideoOutput but it isn't.

[joanxie]

try to add "PACKAGECONFIG_append_pn-qtmultimedia = " gstreamer"  to local.conf" and "IMAGE_INSTALL_append = "\
    qtmultimedia \
    qtmultimedia-plugins \
    qtmultimedia-examples \
    qtmultimedia-qmlplugins""

in the lcoal.conf, then build agian, you can find qmlvideo under /usr/share/qt5/

[MicMoba]

Hi Joan,

thanks for reply. I built it and it ran.

I think I can take the qmlvideo as base for my application.

Thank you

[joanxie]

good !!

[MicMoba]

thanks for your reply.

I don't know about qmlvideo. I will have a look on it.