QT6 Qml embedded video play for weston with Gstreamer Glimagesink on i.MX8MPlus

[#1]

This post shows a workaround about how to embed video on your QT Qml application with high performance on i.MX8MPlus. 

The demo works on Linux BSP L6.1.22_2.0.0 and weston branch weston-imx-11.0.

This demo set the glimagesink pipeline on weston top view and  works for 1080p screen. 

QT MediaPlayer

Since qmlglsink is not supported now on QT6, one way we can display video with QML application is to use MediaPlayer component. However, it has bad performance when playing with high resolution videos. From my test, while playing with 1080p60 video, cpu loading is around 50% and some frames will be lost. 

With pure Gstreamer pipeline, 

gst-launch-1.0 -vvv filesrc location=3.mp4 ! qtdemux  ! vpudec ! video/x-raw,format=NV12 ! imxvideoconvert_g2d ! queue  ! glimagesink

It works good with 1080p60 video.

Therefore, we could make it work with QML application to achieve the best performance on our platform.

Demo explanation 

The demos works with QT6 qml  and the Gstreamer pipeline is

gst_parse_launch("filesrc location=3.mp4 ! qtdemux ! vpudec ! video/x-raw,format=NV12 ! imxvideoconvert_g2d ! video/x-raw,format=BGRx,width=1280,height=720 ! queue ! glimagesink ",NULL);

You can change 3.mp4 to your video. 

if you want to change the size of glimagesink output , you need also change the QML rectangle and weston.ini.

Compile Steps

Weston

git clone https://github.com/nxp-imx/weston-imx.git

git checkout weston-imx-11.0.1

git apply 0001_Weston_Fix_Glimagesink_Pos_and_QML_APP.patch

Follow the below link to compile weston-11 and copy them to the rootfs.

Weston-imx intergration guide - Graphics Technology Engineering Center - NXP Confluence

QML application

source toolchain

cmake .

make

In CMakeLists.txt, you need to modify the INCLUDE_DIRECTORIES to your toolchain path.

Run Steps

Add the following component to the shell section on the /etc/xdg/weston/weston.ini

qtapp is the name of your app.

qt_pos sets the initial x and y position of your app.

glsink_pos sets the initial x and y position of your gstreamer glimagesink pipeline.

These parameters can set your gstreamer video on your application rectangle for playing video, test and choose for your best.

 systemctl restart weston

./FIXVIDEO

Result

          You can see your Gstreamer video is always on top and you can control your pipeline by press the Start, Pause and Stop button.

 Pending to do:

       This is just a simple demo for 1080p screen and shows the video with 1280x720 rectangle. If your screen and show-rectangle is different, then you need to modify not only the qml file, the Gstreamer pipeline, but also the weston.ini to fit your video on the rectangle.

       The video is fixed on the position you set and can not be moved.

 

Reference Link

 i.MX8MPlus :https://www.nxp.com/design/design-center/development-boards/i-mx-evaluation-and-development-boards/e... 

weston: https://github.com/nxp-imx/weston-imx 

[feiyu]

你好！

看到您这篇帖子，可能对我们有所帮助

我们的开发板也是i.mx8mp，内核5.10.9，Qt5，Yocto3.2.0。

开发软件的功能主要是视频播放和控制，需要将视频嵌入显示在widget里面，并且可以改变widget的大小，目前软件在ubuntu上开发是没有问题的，并且也能够正常运行，但是利用i.mx8mp的Qt交叉编译环境编译并运行到i.mx8mp上时，软件会崩溃，并报出Segmentation fault错误，经过问题定位，就是Glimagesink这个插件出现的问题，如果换成waylandsink，软件不会崩溃，但是视频会显示到widget之外

图1 Ubuntu显示效果，是正常想要的效果

图2 i.mx8mp崩溃log

图3 将glimagesink换成waylandsink后i.mx8mp显示效果

非常期待您能回复，如果提供例程或者问题调查方向，不胜感激

[inly_huang]

Hi, @feiyu 

         We can also make a patch for waylandsink  in gstreamer-plugin-bad to fix the specific app waylandsink position.

        For example,  you can use xdg_toplevel_set_title or xdg_toplevel_set_app_id to specify the app name in waylandsink source code.

        Then  the specific app name is known in weston source code. Therefore you can fix the app position in map  and weston_view_set_initial_position function in shell.c for desktop-shell. 

Thanks

[feiyu]

Hi, @inly_huang

Thank you very much for your response.

It may not be exactly the same as your solution.

Our core issue is that the connection of glimagessink to Qt widget causes the program to crash.

Do you have any ideas for solving this problem?

Thanks

[inly_huang]

Hi， @feiyu 

    Could you show the pipeline? And have you tested in other version bsp?

Thanks

[feiyu]

Hi, @inly_huang

Thank you for your response. The issue has been resolved, mainly due to the inability to directly bind Waylandink to Qt Widget under Wayland. The demo code currently used is attached, and the problem has been resolved.

feiyu

int main(int argc, char *argv[]) {
    gst_init(&argc, &argv);
    QApplication app(argc, argv);
    app.connect(&app, SIGNAL(lastWindowClosed()), &app, SLOT(quit()));

    qDebug() << "qt platform name: " << QGuiApplication::platformName();
    // prepare the pipeline
    GstElement *pipeline = gst_parse_launch("videotestsrc ! waylandsink", NULL);

    // prepare the ui
    PlayerWindow *window = new PlayerWindow(pipeline);
    window->resize(640, 480);
    window->show();

    GstElement *vsink = gst_element_factory_make("waylandsink", "wsink");
    g_object_set(GST_OBJECT(pipeline), "video-sink", vsink, NULL);

    // connect to interesting signals
    GstBus *bus = gst_element_get_bus(pipeline);
    gst_bus_add_watch(bus, &PlayerWindow::postGstMessage, window);
    gst_bus_set_sync_handler(bus, bus_sync_handler, window, NULL);
    gst_object_unref(bus);

    // run the pipeline
    GstStateChangeReturn sret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
    if (sret == GST_STATE_CHANGE_FAILURE) {
        gst_element_set_state(pipeline, GST_STATE_NULL);
        gst_object_unref(pipeline);
        // Exit application
        QTimer::singleShot(0, QApplication::activeWindow(), SLOT(quit()));
    }

    int ret = app.exec();

    window->hide();
    delete window;
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);

    return ret;
}