iMX8 gstreamer waylandsink not working when used with gst_video_overlay in NV12 format

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

iMX8 gstreamer waylandsink not working when used with gst_video_overlay in NV12 format

10,178 Views
merwin
Contributor II

I am trying to use waylandsink as a video overlay attached to my wayland surface with gst_video_overlay_set_window_handle() on an iMX8M EVK.  The video fails to render when using NV12 format with the following gstreamer launch string unless I make the video surface big enough (more than around 2560x1920).

"videotestsrc ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink name=sink window-width=320 window-height=240"

When I change the format to I420 it works at all sizes.

I also can't get any video to render (at any size) when using the vpudec decoder with the following launch string when the waylandsink is attached to my wayland surface:

"uridecodebin uri=file:///video.mp4 ! waylandsink name=sink window-width=320 window-height=240"

When not attached to my wayland surface it works.

gstreamer‌ wayland waylandsink nv12 overlay

0 Kudos
15 Replies

8,075 Views
joanxie
NXP TechSupport
NXP TechSupport

how about using glimagesink or kmssink? try to use glimagesink

0 Kudos

8,075 Views
merwin
Contributor II

I can't use kmssink or glimagesink because the video window needs to coexist with a qt window on the same desktop.  

0 Kudos

8,075 Views
joanxie
NXP TechSupport
NXP TechSupport

could you share the reproduce steps? let me reproduce this on my imx8M evk  board.

0 Kudos

8,075 Views
merwin
Contributor II

Here is a stripped down example source code.  You need to run with weston at 3840x2160 and resize the window until the video shows (almost full screen).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <gst/gst.h>
#include <gst/video/videooverlay.h>
#include <wayland-client.h>
#include "viewporter-client-protocol.h"

int done = 0;
struct wl_compositor *compositor;
struct wl_shell *shell;
struct wl_shm *shm;
struct wp_viewporter *viewporter;
struct wp_viewport *viewport;
struct wl_surface *surface;
GstElement *videosink;

void sig_handler(int signum, siginfo_t *siginfo, void *ctx)
{
    done = 1;
}

static void cb_global_registry_handler(void *data, struct wl_registry *registry, uint32_t id, const char *interface, uint32_t version)
{
    //fprintf(stderr, "Got a registry event for %s id %d\n", interface, id);

    if (strcmp(interface, "wl_compositor") == 0)
        compositor = (struct wl_compositor *) wl_registry_bind(registry, id, &wl_compositor_interface, 1);
    else if (strcmp(interface, "wl_shell") == 0)
        shell = (struct wl_shell *) wl_registry_bind(registry, id, &wl_shell_interface, 1);
    else if (strcmp(interface, "wl_shm") == 0)
        shm = (struct wl_shm *) wl_registry_bind(registry, id, &wl_shm_interface, 1);
    else if (strcmp(interface, "wp_viewporter") == 0)
        viewporter = (struct wp_viewporter *) wl_registry_bind(registry, id, &wp_viewporter_interface, 1);
}

static void cb_global_registry_remover(void *data, struct wl_registry *registry, uint32_t id)
{
}

static const struct wl_registry_listener registry_listener =
{
    cb_global_registry_handler,
    cb_global_registry_remover
};

static void cb_shell_surface_ping(void *data, struct wl_shell_surface *shell_surface, uint32_t serial)
{
    wl_shell_surface_pong(shell_surface, serial);
}

static void cb_shell_surface_configure(void *data, struct wl_shell_surface *shell_surface, uint32_t edges, int32_t width, int32_t height)
{
    wp_viewport_set_destination(viewport, width, height);
    wl_surface_commit(surface);
    gst_video_overlay_set_render_rectangle(GST_VIDEO_OVERLAY(videosink), 10, 10, width-20, height-20);
}

static void cb_shell_surface_popup_done(void *data, struct wl_shell_surface *shell_surface)
{
}

static const struct wl_shell_surface_listener shell_surface_listener =
{
    cb_shell_surface_ping,
    cb_shell_surface_configure,
    cb_shell_surface_popup_done
};

int main(int argc, char *argv[])
{
    struct sigaction sa;
    sa.sa_sigaction = sig_handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART | SA_SIGINFO;
    sigaction(SIGINT, &sa, NULL);

    fprintf(stderr, "init\n");
    gst_init(&argc, &argv);
    GError *e;
    GstElement *pipeline = gst_parse_launch("videotestsrc ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink name=sink window-width=320 window-height=240", &e);

    fprintf(stderr, "create display\n");
    struct wl_display *display = wl_display_connect(NULL);
    fprintf(stderr, "display=%p\n", display);
    struct wl_registry *registry = wl_display_get_registry(display);
    wl_registry_add_listener(registry, &registry_listener, NULL);
    wl_display_dispatch(display);
    wl_display_roundtrip(display);
    fprintf(stderr, "compositor=%p\n", compositor);
    fprintf(stderr, "shell=%p\n", shell);
    fprintf(stderr, "shm=%p\n", shm);
    fprintf(stderr, "viewporter=%p\n", viewporter);

    fprintf(stderr, "create surface\n");
    surface = wl_compositor_create_surface(compositor);
    fprintf(stderr, "surface=%p\n", surface);
    struct wl_shell_surface *shell_surface = wl_shell_get_shell_surface(shell, surface);
    fprintf(stderr, "shell_surface=%p\n", shell_surface);
    wl_shell_surface_set_toplevel(shell_surface);
    wl_shell_surface_add_listener(shell_surface, &shell_surface_listener, NULL);
    wl_surface_commit(surface);

    fprintf(stderr, "create buffer\n");
    int buffer_width = 1;
    int buffer_height = 1;
    int buffer_stride = buffer_width * sizeof(uint32_t);
    int buffer_size = buffer_stride * buffer_height;
    fprintf(stderr, "buffer_size=%d\n", buffer_size);
    char filename[] = "/tmp/gst-wayland-test-XXXXXX";
    int fd = mkostemp(filename, O_CLOEXEC);
    fprintf(stderr, "fd=%d, filename=%s\n", fd, filename);
    ftruncate(fd, buffer_size);
    uint32_t *data = (uint32_t *) mmap(NULL, buffer_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    fprintf(stderr, "data=%p\n", data);
    uint32_t *p = data;
    for(int y=0; y<buffer_height; y++)
        for(int x=0; x<buffer_width; x++)
            *p++ = 0xFF00FF00;
    struct wl_shm_pool *pool = wl_shm_create_pool(shm, (int32_t) fd, buffer_size);
    fprintf(stderr, "pool=%p\n", pool);
    struct wl_buffer *buffer = wl_shm_pool_create_buffer(pool, 0, buffer_width, buffer_height, buffer_stride, WL_SHM_FORMAT_ARGB8888);
    fprintf(stderr, "buffer=%p\n", buffer);
    wl_shm_pool_destroy(pool);
    wl_surface_attach(surface, buffer, 0, 0);

    fprintf(stderr, "create viewport\n");
    int window_width=320;
    int window_height=240;
    viewport = wp_viewporter_get_viewport(viewporter, surface);
    fprintf(stderr, "viewport=%p\n", viewport);
    wp_viewport_set_source(viewport, 0, 0, buffer_width, buffer_height);
    wp_viewport_set_destination(viewport, window_width, window_height);
    wl_surface_commit(surface);

    fprintf(stderr, "set gstreamer overlay\n");
    GstContext *context = gst_context_new("GstWaylandDisplayHandleContextType", TRUE);
    fprintf(stderr, "context=%p\n", context);
    gst_structure_set(gst_context_writable_structure(context), "handle", G_TYPE_POINTER, display, NULL);
    gst_element_set_context(pipeline, context);
    videosink = gst_bin_get_by_name((GstBin *)pipeline, "sink");
    fprintf(stderr, "videosink=%p\n", videosink);
    gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(videosink), (guintptr) surface);
    gst_video_overlay_set_render_rectangle(GST_VIDEO_OVERLAY(videosink), 10, 10, window_width-20, window_height-20);
    wl_surface_commit(surface);

    fprintf(stderr, "start pipeline\n");
    gst_element_set_state(pipeline, GST_STATE_PLAYING);

    while (!done) wl_display_dispatch_pending(display);

    fprintf(stderr, "stop pipeline\n");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);
    gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(videosink), (guintptr) NULL);
    gst_deinit();

    fprintf(stderr, "cleanup\n");
    wl_surface_destroy(surface);
    munmap(data, buffer_size);
    close(fd);
    wl_display_disconnect(display);

    fprintf(stderr, "done\n");
    return 0;
}
0 Kudos

8,075 Views
joanxie
NXP TechSupport
NXP TechSupport

do you use the latest bsp version L4.14.78? could you find the same issue on the latest bsp version?

0 Kudos

8,075 Views
merwin
Contributor II

I am using L4.14.78.

0 Kudos

8,075 Views
joanxie
NXP TechSupport
NXP TechSupport

"

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw,format=NV12,width=320,height=240 ! waylandsink  window-width=320 window-height=240 is working well.

 

I can compile your file. Where do you get  #include "viewporter-client-protocol.h" ? I can find it in the Yocto build folder but not on the target.

"

0 Kudos

8,075 Views
merwin
Contributor II

viewporter-client-protocol.h comes from the weston sources.

0 Kudos

8,075 Views
joanxie
NXP TechSupport
NXP TechSupport

I got reply from the expert team:

Here is the application compiled (with the Makefile that generates the missing header) in I420, NV12 and "NV12 bigger buffer".

It includes the waylandsink debug trace of each application.

I am still analyzing the issue. My guess is has something to do with the memory type used for the buffer.

I420 is only supported in SHM (share memory). All the other formats are using (DMABuf). 

 

gst_wayland_sink_get_caps:<sink> [00m display caps: video/x-raw, format=(string){ BGRA, BGRx, RGB16, I420, NV12, YUY2, NV12_10LE }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw(memory:DMABuf), format=(string){ RGB16, BGRx, RGBx, xBGR, ARGB, BGRA, RGBA, ABGR, xRGB, YUY2, NV12, NV12_10LE }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]

 

remark: it doesn't work with any formats except I420.

 

From the log:

Creating wl_buffer from SHM of size 115200 (320 x 240, stride 320), format I420

Creating wl_buffer from DMABuf of size 115200 (320 x 240), format NV12 modifier 0x0000000000000000

 

However, I can't explain why this is working with 2560x1920 buffer in NV12.

0 Kudos

8,076 Views
merwin
Contributor II

I was only using videotestsrc in the example to show the problem with NV12 format.  I actually want to use the vpudec as the source, which outputs NV12 and also does not render. 

0 Kudos

8,076 Views
joanxie
NXP TechSupport
NXP TechSupport

"

This should be an alpha blend issue. 8mq display control has scale ratio range from (1/8, 7), When video buffer is less than (width=2650,height=1920), video will go to overlay plane. On 8mq platform, overlay plane is under graphic plane which has GUI buffer. You need set your wayland surface alpha to 0 to make video can be seen. Please refer below function in gstreamer
https://bitbucket.sw.nxp.com/projects/MMCSH/repos/gst-plugins-bad/browse/ext/wayland/wlwindow.c?at=r...

void
gst_wl_window_set_alpha (GstWlWindow * window, gfloat alpha)
{
if (window && window->blend_func)

{ zwp_blending_v1_set_alpha(window->blend_func, wl_fixed_from_double(alpha)); if(alpha < 1.0) zwp_blending_v1_set_blending(window->blend_func, ZWP_BLENDING_V1_BLENDING_EQUATION_FROMSOURCE); else zwp_blending_v1_set_blending(window->blend_func, ZWP_BLENDING_V1_BLENDING_EQUATION_PREMULTIPLIED); }

}

when your buffer is bigger than 2560*1920, the down scale ratio is bigger than 8, so video will go to GUI plane and you don't need set alpha. Also I420 is not accept for overlay plane, so it is on graphic plane.

Indeed this is an Alpha issue:

It is not enough to add the parameter to the  "waylandsink alpha=0.0".

The alpha is set here:

gstwaylandsink.c\wayland\ext - gst-plugins-bad - i.MX Gstreamer Bad Fork 

 

But in the case of the customer application, the "sink->window" already exist the first time this function is called. In case of the customer application, the alpha is never set.

"

8,076 Views
merwin
Contributor II

I was able to get it working after modifying the waylandsink code.  Thanks.

0 Kudos

5,880 Views
NagendraB
Contributor II

Hello  @merwin   


i am trying to test the gstreamer api gst_video_overlay_set_window_handle and gst_element_set_context to render the video test source on user created wayland/egl surface , but nothing works as expected (platform used is imx8mp 5.4.70) 

any leads in this contexts is much appreciated , target is to use eglsurface to render video using gstreamer 

0 Kudos

5,864 Views
merwin
Contributor II

I would like to help but we haven't been using the iMX8 for a while now and I don't remember the details.

0 Kudos

6,016 Views
NagendraB
Contributor II

Hello Merwin

the alpha property of waylandsink is not working for me, is there any limitation ?  any inputs to fix this ?

(platform - imx8m-plus 5.4.70-imx8mp) 

0 Kudos