iMX8 gstreamer waylandsink not working when used with gst_video_overlay in NV12 format

[merwin]

I am trying to use waylandsink as a video overlay attached to my wayland surface with gst_video_overlay_set_window_handle() on an iMX8M EVK.  The video fails to render when using NV12 format with the following gstreamer launch string unless I make the video surface big enough (more than around 2560x1920).

"videotestsrc ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink name=sink window-width=320 window-height=240"

When I change the format to I420 it works at all sizes.

I also can't get any video to render (at any size) when using the vpudec decoder with the following launch string when the waylandsink is attached to my wayland surface:

"uridecodebin uri=file:///video.mp4 ! waylandsink name=sink window-width=320 window-height=240"

When not attached to my wayland surface it works.

gstreamer‌ wayland waylandsink nv12 overlay

[joanxie]

how about using glimagesink or kmssink? try to use glimagesink

[merwin]

I can't use kmssink or glimagesink because the video window needs to coexist with a qt window on the same desktop.  

[joanxie]

could you share the reproduce steps? let me reproduce this on my imx8M evk  board.

[merwin]

Here is a stripped down example source code.  You need to run with weston at 3840x2160 and resize the window until the video shows (almost full screen).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <gst/gst.h>
#include <gst/video/videooverlay.h>
#include <wayland-client.h>
#include "viewporter-client-protocol.h"

int done = 0;
struct wl_compositor *compositor;
struct wl_shell *shell;
struct wl_shm *shm;
struct wp_viewporter *viewporter;
struct wp_viewport *viewport;
struct wl_surface *surface;
GstElement *videosink;

void sig_handler(int signum, siginfo_t *siginfo, void *ctx)
{
    done = 1;
}

static void cb_global_registry_handler(void *data, struct wl_registry *registry, uint32_t id, const char *interface, uint32_t version)
{
    //fprintf(stderr, "Got a registry event for %s id %d\n", interface, id);

    if (strcmp(interface, "wl_compositor") == 0)
        compositor = (struct wl_compositor *) wl_registry_bind(registry, id, &wl_compositor_interface, 1);
    else if (strcmp(interface, "wl_shell") == 0)
        shell = (struct wl_shell *) wl_registry_bind(registry, id, &wl_shell_interface, 1);
    else if (strcmp(interface, "wl_shm") == 0)
        shm = (struct wl_shm *) wl_registry_bind(registry, id, &wl_shm_interface, 1);
    else if (strcmp(interface, "wp_viewporter") == 0)
        viewporter = (struct wp_viewporter *) wl_registry_bind(registry, id, &wp_viewporter_interface, 1);
}

static void cb_global_registry_remover(void *data, struct wl_registry *registry, uint32_t id)
{
}

static const struct wl_registry_listener registry_listener =
{
    cb_global_registry_handler,
    cb_global_registry_remover
};

static void cb_shell_surface_ping(void *data, struct wl_shell_surface *shell_surface, uint32_t serial)
{
    wl_shell_surface_pong(shell_surface, serial);
}

static void cb_shell_surface_configure(void *data, struct wl_shell_surface *shell_surface, uint32_t edges, int32_t width, int32_t height)
{
    wp_viewport_set_destination(viewport, width, height);
    wl_surface_commit(surface);
    gst_video_overlay_set_render_rectangle(GST_VIDEO_OVERLAY(videosink), 10, 10, width-20, height-20);
}

static void cb_shell_surface_popup_done(void *data, struct wl_shell_surface *shell_surface)
{
}

static const struct wl_shell_surface_listener shell_surface_listener =
{
    cb_shell_surface_ping,
    cb_shell_surface_configure,
    cb_shell_surface_popup_done
};

int main(int argc, char *argv[])
{
    struct sigaction sa;
    sa.sa_sigaction = sig_handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART | SA_SIGINFO;
    sigaction(SIGINT, &sa, NULL);

    fprintf(stderr, "init\n");
    gst_init(&argc, &argv);
    GError *e;
    GstElement *pipeline = gst_parse_launch("videotestsrc ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink name=sink window-width=320 window-height=240", &e);

    fprintf(stderr, "create display\n");
    struct wl_display *display = wl_display_connect(NULL);
    fprintf(stderr, "display=%p\n", display);
    struct wl_registry *registry = wl_display_get_registry(display);
    wl_registry_add_listener(registry, &registry_listener, NULL);
    wl_display_dispatch(display);
    wl_display_roundtrip(display);
    fprintf(stderr, "compositor=%p\n", compositor);
    fprintf(stderr, "shell=%p\n", shell);
    fprintf(stderr, "shm=%p\n", shm);
    fprintf(stderr, "viewporter=%p\n", viewporter);

    fprintf(stderr, "create surface\n");
    surface = wl_compositor_create_surface(compositor);
    fprintf(stderr, "surface=%p\n", surface);
    struct wl_shell_surface *shell_surface = wl_shell_get_shell_surface(shell, surface);
    fprintf(stderr, "shell_surface=%p\n", shell_surface);
    wl_shell_surface_set_toplevel(shell_surface);
    wl_shell_surface_add_listener(shell_surface, &shell_surface_listener, NULL);
    wl_surface_commit(surface);

    fprintf(stderr, "create buffer\n");
    int buffer_width = 1;
    int buffer_height = 1;
    int buffer_stride = buffer_width * sizeof(uint32_t);
    int buffer_size = buffer_stride * buffer_height;
    fprintf(stderr, "buffer_size=%d\n", buffer_size);
    char filename[] = "/tmp/gst-wayland-test-XXXXXX";
    int fd = mkostemp(filename, O_CLOEXEC);
    fprintf(stderr, "fd=%d, filename=%s\n", fd, filename);
    ftruncate(fd, buffer_size);
    uint32_t *data = (uint32_t *) mmap(NULL, buffer_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    fprintf(stderr, "data=%p\n", data);
    uint32_t *p = data;
    for(int y=0; y<buffer_height; y++)
        for(int x=0; x<buffer_width; x++)
            *p++ = 0xFF00FF00;
    struct wl_shm_pool *pool = wl_shm_create_pool(shm, (int32_t) fd, buffer_size);
    fprintf(stderr, "pool=%p\n", pool);
    struct wl_buffer *buffer = wl_shm_pool_create_buffer(pool, 0, buffer_width, buffer_height, buffer_stride, WL_SHM_FORMAT_ARGB8888);
    fprintf(stderr, "buffer=%p\n", buffer);
    wl_shm_pool_destroy(pool);
    wl_surface_attach(surface, buffer, 0, 0);

    fprintf(stderr, "create viewport\n");
    int window_width=320;
    int window_height=240;
    viewport = wp_viewporter_get_viewport(viewporter, surface);
    fprintf(stderr, "viewport=%p\n", viewport);
    wp_viewport_set_source(viewport, 0, 0, buffer_width, buffer_height);
    wp_viewport_set_destination(viewport, window_width, window_height);
    wl_surface_commit(surface);

    fprintf(stderr, "set gstreamer overlay\n");
    GstContext *context = gst_context_new("GstWaylandDisplayHandleContextType", TRUE);
    fprintf(stderr, "context=%p\n", context);
    gst_structure_set(gst_context_writable_structure(context), "handle", G_TYPE_POINTER, display, NULL);
    gst_element_set_context(pipeline, context);
    videosink = gst_bin_get_by_name((GstBin *)pipeline, "sink");
    fprintf(stderr, "videosink=%p\n", videosink);
    gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(videosink), (guintptr) surface);
    gst_video_overlay_set_render_rectangle(GST_VIDEO_OVERLAY(videosink), 10, 10, window_width-20, window_height-20);
    wl_surface_commit(surface);

    fprintf(stderr, "start pipeline\n");
    gst_element_set_state(pipeline, GST_STATE_PLAYING);

    while (!done) wl_display_dispatch_pending(display);

    fprintf(stderr, "stop pipeline\n");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);
    gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(videosink), (guintptr) NULL);
    gst_deinit();

    fprintf(stderr, "cleanup\n");
    wl_surface_destroy(surface);
    munmap(data, buffer_size);
    close(fd);
    wl_display_disconnect(display);

    fprintf(stderr, "done\n");
    return 0;
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[joanxie]

do you use the latest bsp version L4.14.78? could you find the same issue on the latest bsp version?

[merwin]

I am using L4.14.78.

[joanxie]

"

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink  window-width=320 window-height=240 is working well.

I can compile your file. Where do you get  #include "viewporter-client-protocol.h" ? I can find it in the Yocto build folder but not on the target.

"

[merwin]

viewporter-client-protocol.h comes from the weston sources.

[joanxie]

I got reply from the expert team:

Here is the application compiled (with the Makefile that generates the missing header) in I420, NV12 and "NV12 bigger buffer".

It includes the waylandsink debug trace of each application.

I am still analyzing the issue. My guess is has something to do with the memory type used for the buffer.

I420 is only supported in SHM (share memory). All the other formats are using (DMABuf). 

gst_wayland_sink_get_caps:<sink> [00m display caps: video/x-raw, format=(string){ BGRA, BGRx, RGB16, I420, NV12, YUY2, NV12_10LE }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw(memory:DMABuf), format=(string){ RGB16, BGRx, RGBx, xBGR, ARGB, BGRA, RGBA, ABGR, xRGB, YUY2, NV12, NV12_10LE }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]

remark: it doesn't work with any formats except I420.

From the log:

Creating wl_buffer from SHM of size 115200 (320 x 240, stride 320), format I420

Creating wl_buffer from DMABuf of size 115200 (320 x 240), format NV12 modifier 0x0000000000000000

However, I can't explain why this is working with 2560x1920 buffer in NV12.

[merwin]

I was only using videotestsrc in the example to show the problem with NV12 format.  I actually want to use the vpudec as the source, which outputs NV12 and also does not render. 

[joanxie]

"

This should be an alpha blend issue. 8mq display control has scale ratio range from (1/8, 7), When video buffer is less than (width=2650,height=1920), video will go to overlay plane. On 8mq platform, overlay plane is under graphic plane which has GUI buffer. You need set your wayland surface alpha to 0 to make video can be seen. Please refer below function in gstreamer
https://bitbucket.sw.nxp.com/projects/MMCSH/repos/gst-plugins-bad/browse/ext/wayland/wlwindow.c?at=r...

void
gst_wl_window_set_alpha (GstWlWindow * window, gfloat alpha)
{
if (window && window->blend_func)

{ zwp_blending_v1_set_alpha(window->blend_func, wl_fixed_from_double(alpha)); if(alpha < 1.0) zwp_blending_v1_set_blending(window->blend_func, ZWP_BLENDING_V1_BLENDING_EQUATION_FROMSOURCE); else zwp_blending_v1_set_blending(window->blend_func, ZWP_BLENDING_V1_BLENDING_EQUATION_PREMULTIPLIED); }

}

when your buffer is bigger than 2560*1920, the down scale ratio is bigger than 8, so video will go to GUI plane and you don't need set alpha. Also I420 is not accept for overlay plane, so it is on graphic plane.

Indeed this is an Alpha issue:

It is not enough to add the parameter to the  "waylandsink alpha=0.0".

The alpha is set here:

gstwaylandsink.c\wayland\ext - gst-plugins-bad - i.MX Gstreamer Bad Fork 

But in the case of the customer application, the "sink->window" already exist the first time this function is called. In case of the customer application, the alpha is never set.

"

[merwin]

I was able to get it working after modifying the waylandsink code.  Thanks.

[NagendraB]

Hello  @merwin   

i am trying to test the gstreamer api gst_video_overlay_set_window_handle and gst_element_set_context to render the video test source on user created wayland/egl surface , but nothing works as expected (platform used is imx8mp 5.4.70) 

any leads in this contexts is much appreciated , target is to use eglsurface to render video using gstreamer 

[merwin]

I would like to help but we haven't been using the iMX8 for a while now and I don't remember the details.

[NagendraB]

Hello Merwin

the alpha property of waylandsink is not working for me, is there any limitation ?  any inputs to fix this ?

(platform - imx8m-plus 5.4.70-imx8mp)