Sabre i.MX6DL Gstreamer single frame stepping

[steveevers]

When implementing a GStreamer 1.0 mp4 decode pipeline on a Sabre i.MX6DL platform with the fsl-image-qt5, forward and reverse playing at various speeds works fine. When implementing single frame stepping as detailed in:

Basic tutorial 13: Playback speed - GStreamer SDK documentation - GStreamer SDK documentation

the vpudec is not rendering the decoded image for the next frame. I've modified this GStreamer example to setup up a simple decode pipe, starts decoding an mp4 file, and issue gst_event_new_step() events to the pipeline. The pipeline does appear to advance and reverse the media as reported by gst_element_query_position() calls, but the rendering is not updated (it is stuck at the point it was paused).

Thanks for any tips or suggestions on getting the vpu to render new frames on gst_event_new_step() calls.

HW/SW description:

Dev board = Sabre i.MX6DL

Freescale release = fsl-yocto-3.14.28-1.0.0

graphics backend = eglfs

bitback image = fsl-image-qt5

[alejandrolozan1]

Hi,

Did you modify something to the code you mentioned? Is there anything else I have to do to reproduce the problem?

Best Regards,

Alejandro

[alejandrolozan1]

Hi,

I have tried the code you mentioned with gstreamer-1.0 but if fails to compile. It compiles correctly with gstreamer-0.10.

/Alejandro

[steveevers]

Hi Alejandro,

See below for the gstreamer basic_tutorial-13.c modified for streamer 1.0. Differences are:

1) gst_element_query_position() API modification

2) I am using a local file-system mp4 container instead of network uri stream.

2) gst_event_new_step() can only take non-negative values in gst 1.0. A seek event, send_seek_event(), needs to be sent with a reversed data rate to change direction and then gst_event_new_step() event can be sent with position step values.

Note: this build works on my HOST machine. It does not work on the IMX6 target. On some of the code I have instrumented it appears the parsing is working (timestamps advancing or reversing, but the display is not being updated).

// host build

// gcc basic-tutorial-13.c -o host-tutorial-13 `pkg-config --cflags --libs gstreamer-1.0`

// IMX6 target build and deploy

// $CC -DIMX6 -Wall basic-tutorial-13.c -o target-tutorial-13 $(pkg-config --cflags --libs gstreamer-1.0)

// cp ~/projects/gstSDK/tutorialSource/myModifications/basic_tutorial-13/target-tutorial-13 /media/nfs/revd_4/home/root/deploy/

#include <string.h>

#include <stdio.h>

#include <gst/gst.h>

typedef struct _CustomData {

  GstElement *pipeline;

  GstElement *video_sink;

  GMainLoop *loop;

  gboolean playing;  /* Playing or Paused */

  gdouble rate;      /* Current playback rate (can be negative) */

} CustomData;

/* Send seek event to change rate */

static void send_seek_event (CustomData *data) {

  gint64 position;

  // GstFormat format = GST_FORMAT_TIME;

  GstEvent *seek_event;

  /* Obtain the current position, needed for the seek event */

  // if (!gst_element_query_position (data->pipeline, &format, &position)) {

  if (!gst_element_query_position (data->pipeline, GST_FORMAT_TIME, &position)) {

    g_printerr ("Unable to retrieve current position.\n");

    return;

  }

  /* Create the seek event */

  if (data->rate > 0) {

    seek_event = gst_event_new_seek (data->rate, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE,

        GST_SEEK_TYPE_SET, position, GST_SEEK_TYPE_SET, -1);

  } else {

    seek_event = gst_event_new_seek (data->rate, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE,

        GST_SEEK_TYPE_SET, 0, GST_SEEK_TYPE_SET, position);

  }

  if (data->video_sink == NULL) {

    /* If we have not done so, obtain the sink through which we will send the seek events */

    g_object_get (data->pipeline, "video-sink", &data->video_sink, NULL);

  }

  /* Send the event */

  gst_element_send_event (data->video_sink, seek_event);

  g_print ("Current rate: %g\n", data->rate);

}

// TODO no negative rates in steps 

// https://developer.gnome.org/gstreamer/stable/gstreamer-GstEvent.html#gst-event-new-step

/* Process keyboard input */

static gboolean handle_keyboard (GIOChannel *source, GIOCondition cond, CustomData *data) {

  gchar *str = NULL;

  if (g_io_channel_read_line (source, &str, NULL, NULL, NULL) != G_IO_STATUS_NORMAL) {

    return TRUE;

  }

  switch (g_ascii_tolower (str[0])) {

  case 'p':

    data->playing = !data->playing;

    gst_element_set_state (data->pipeline, data->playing ? GST_STATE_PLAYING : GST_STATE_PAUSED);

    g_print ("Setting state to %s\n", data->playing ? "PLAYING" : "PAUSE");

    break;

  case 's':

    if (g_ascii_isupper (str[0])) {

      data->rate *= 2.0;

    } else {

      data->rate /= 2.0;

    }

    send_seek_event (data);

    break;

  case 'd':

    data->rate *= -1.0;

    send_seek_event (data);

    break;

  case 'n':

    if (data->video_sink == NULL) {

      /* If we have not done so, obtain the sink through which we will send the step events */

      g_object_get (data->pipeline, "video-sink", &data->video_sink, NULL);

    }

    gst_element_send_event (data->video_sink,

        gst_event_new_step (GST_FORMAT_BUFFERS, 1, 1/*data->rate*/, TRUE, FALSE));

    g_print ("Stepping one frame\n");

    break;

  case 'q':

    g_main_loop_quit (data->loop);

    break;

  default:

    break;

  }

  g_free (str);

  return TRUE;

}

int main(int argc, char *argv[]) {

  CustomData data;

  GstStateChangeReturn ret;

  GIOChannel *io_stdin;

  /* Initialize GStreamer */

  gst_init (&argc, &argv);

  /* Initialize our data structure */

  memset (&data, 0, sizeof (data));

  /* Print usage map */

  g_print (

    "USAGE: Choose one of the following options, then press enter:\n"

    " 'P' to toggle between PAUSE and PLAY\n"

    " 'S' to increase playback speed, 's' to decrease playback speed\n"

    " 'D' to toggle playback direction\n"

    " 'N' to move to next frame (in the current direction, better in PAUSE)\n"

    " 'Q' to quit\n");

  /* Build the pipeline */

#if defined(IMX6)

  data.pipeline = gst_parse_launch ("playbin uri=file:////home/root/media/br_0_gop_15_q_neg1.mp4", NULL);

#else

  data.pipeline = gst_parse_launch ("playbin uri=file:////home/steve/media/br_0_gop_15_q_neg1.mp4", NULL);

#endif

  /* Add a keyboard watch so we get notified of keystrokes */

#ifdef _WIN32

  io_stdin = g_io_channel_win32_new_fd (fileno (stdin));

#else

  io_stdin = g_io_channel_unix_new (fileno (stdin));

#endif

  g_io_add_watch (io_stdin, G_IO_IN, (GIOFunc)handle_keyboard, &data);

  /* Start playing */

  ret = gst_element_set_state (data.pipeline, GST_STATE_PLAYING);

  if (ret == GST_STATE_CHANGE_FAILURE) {

    g_printerr ("Unable to set the pipeline to the playing state.\n");

    gst_object_unref (data.pipeline);

    return -1;

  }

  data.playing = TRUE;

  data.rate = 1.0;

  /* Create a GLib Main Loop and set it to run */

  data.loop = g_main_loop_new (NULL, FALSE);

  g_main_loop_run (data.loop);

  /* Free resources */

  g_main_loop_unref (data.loop);

  g_io_channel_unref (io_stdin);

  gst_element_set_state (data.pipeline, GST_STATE_NULL);

  if (data.video_sink != NULL)

    gst_object_unref (data.video_sink);

  gst_object_unref (data.pipeline);

  return 0;

}

[alejandrolozan1]

Hi,

I just changed this line

gst_event_new_step (GST_FORMAT_BUFFERS, 1, 1/*data->rate*/, TRUE, FALSE));

to

gst_event_new_step (GST_FORMAT_BUFFERS, 1, 1/*data->rate*/, TRUE, TRUE));

And it seems that is working.

Best Regards,

Alejandro

[steveevers]

Thanks for looking at this Alejandro.

I made the change gst_event_new_step (GST_FORMAT_BUFFERS, 1, 1/*data->rate*/, TRUE, TRUE));. When I pause the pipeline and issue the 'n' step command from the console, the pipeline just starts playing again. What I am trying to do step the pipeline one frame at a time.

The demo works (frame steps) when built for a PC, but does not appear to work when targeting the iMX6 for me. Where you targeting an iMX.6 device, pausing it and then trying to frame step it?

Thanks

[alejandrolozan1]

Hi,

That is weird, on my side I type 'p' to pause the video and then 'n'. Everytime I hit 'n' a step frame is reproduced.

The difference is that I am using x11 backend.

/Alejandro

[alejandrolozan1]

Hi,

Sorry for the delay.

I was just able to reproduce the problem. Let me keep delving into this. I will get back to you as soon as I get something useful.

Best Regards,

Alejandro