Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

[dv1]

Hello,

I have been using the GStreamer plugins for i.MX6 video decoding using a Yocto-based rootfs. I used the recipes from Freescale/meta-fsl-arm · GitHub for building (I used the danny branch). I used "imx6qsabresd" as the machine in Yocto's local.conf.

This gave me a kernel 3.0.35, and the following package versions:

firmware-imx-vpu-imx6q - 12.09.01-r4.0

gst-fsl-plugin - 3.0.1-r9.1

imx-audio - 0.1-r4

imx-lib - 12.09.01-r4.0

libfslparser - 3.0.1-r1

libfslvpuwrap3 - 1.0.17-r0

libgal-common-mx6 - 12.09.01-r1.0

When I tried to play a h264 video with GStreamer's playbin2 , it worked fine:

gst-launch-0.10 playbin2 uri=file:///media/mmcblk1p3/sintel_trailer-1080p.mp4

However, when I use an appsink as videosink, it breaks down:

gst-launch-0.10 playbin2 uri=file:///media/mmcblk1p3/sintel_trailer-1080p.mp4 video-sink="appsink"

I get errors like:

0:00:00.531415996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.560362996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.561086330  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.601098330  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.601788996  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.601995329  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.602423663  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.602734329  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.603029996  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.603659329  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.604238663  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.604373663  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.604993996  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.605017996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.605644663  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.605856996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.606362329  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.606614329  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.606981329  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.607351996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.607581663  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.608115996  1015 0x2b825d80 WARN          mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.608310330  1015 0x2b825c60 WARN                  vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

I never see these errors when using another sink, like mfw_v4lsink, fakesink, or fbdevsink.

This is relevant to me, because WebKit internally uses a derivative of the appsink for the video frames. In other words, the VPU GStreamer plugins will not work with WebKit.

Are the errors above known? Is there a newer version of the Freescale video decoder plugins which fixes them?

Original Attachment has been moved to: vpu-errors.log.zip

[frsc]

I would also be interested in any new development concerning HTML5 videos in WebKit on i.MX6.

I tried a yocto system with WebKit-GTK-2.4.5 and gstreamer1.0 and the gstreamer-imx plugins from dv1.

I'm using a local webm video file and I can see the progress-bar moving as the video plays, but only get a blank image.

[stevengreenberg]

Does anyone know if this ever been resolved? I see it's almost a year since the original post and there's no significant status change in the related bugzilla bug. I'm currently trying to use appsink with vpudec and I'm getting this same error.

If it hasn't been resolved, are there any suggested workarounds?

[daiane_angolini]

Well, there is a newer BSP (1.1.0), but I´m not sure it fixes this issues, because I even understand the issue (I don´t know aapsink)

Are you able to test one gstreamer command line without playbin, only to display video on appsink? ( I mean, create your own command line with decoder, demux and appsink, without audio)

Please let me know the results

[dv1]

Okay, I now tried it with a manually created pipeline:

gst-launch-0.10 filesrc location=sintel_trailer-1080p.mp4 ! "video/quicktime, variant=(string)iso" ! aiurdemux name=d     "d.video_0000" ! vpudec ! appsink

I attached the log (produced with GST_DEBUG=2,*aiur*:5,*vpudec*:5 ) .

[daiane_angolini]

I was not able to see anything special on your log, sorry.

What would be the behavior if you use qtdemux instead of aiurdemux

[dv1]

I used this pipeline for trying out qtdemux:

GST_DEBUG=2,*aiur*:5,*vpudec*:5 gst-launch-0.10 filesrc location=sintel_trailer-1080p.mp4 ! "video/quicktime, variant=(string)iso" ! qtdemux name=d     "d.video_00" ! vpudec ! appsink

And it didn't work either, but the errors are different now. I attached the new log.

[daiane_angolini]

So, I suppose it´s an error on the decoder-sink interface

I suppose you have tested the same video on you linux PC and it worked.

And I suppose you have tried a non vpu accelerated decoder on the board, and it worked.

How could I install appsink in my rootfs? It was not installed by default. Are you using prebuilt LTIB image? Please, let me know

[dv1]

Your assumptions are correct. The videos work fine on the PC, and they also work fine with other decoders.

I did not use LTIB, I used Yocto instead. Yocto 1.3 (with poky 8.0 "danny"), to be exact. I also added Freescale's meta-fsl-arm layer, to get the VPU plugins.

The appsink is part of the GStreamer base plugins package. The name in the LTIB package list is "gstreamer-plugins-base" : https://support.bluetechnix.at/wiki/Qt_Embedded_on_i.MX53#LTIB_configuration_.28Qt_Embedded.29

[daiane_angolini]

Ow! you´re using yocto!

appsink is default installed on it? I have fsl-image-gui and fsl-image-test already built for sabre sd (I suppose the board will not be a problem)

Do you think I could use some of those images?

[dv1]

These images should be OK, but you should modify their recipes to include "gst-plugins-base-meta" to the EXTRA_IMAGE_FEATURES list (for example, look at the fsl-image-test recipe: meta-fsl-demos/recipes-fsl/images/fsl-image-test.bb at master · Freescale/meta-fsl-demos · GitHub ) . This installs all plugins from the base package (including appsink). By default, these images do not install it.

To be able to test other decoders etc. it is probably best to just add all of GStreamer, so I add these entries to the list:

	gst-plugins-base-meta
	gst-plugins-good-meta
	gst-plugins-bad-meta
	gst-plugins-ugly-meta
	gst-ffmpeg

[daiane_angolini]

Ok!

Let me bitbake it and I update you as soon as I have tested

(sorry, it can take 1 or 2 days until I´m able to effectively test it, but I´m here)

[dv1]

It also happens with newer versions:

firmware-imx-vpu-imx6q : 1:1.1.0-r4.0

gst-fsl-plugin : 3.0.5-r9.3

imx-audio : 0.1-r4

imx-lib : 1:1.1.0-r4.0

libfslparser : 3.0.5-r1

libfslvpuwrap3 : 1.0.17-r0

libgal-common-mx6 : 12.09.01-r2.0

libgstfsl-0.10-0 : 3.0.5-r9.3

[daiane_angolini]

I tested a lot of variations and I was not able to get appsink in my rootfs. Can you send me a patch?

I don´t think it´s a fsl-plugins *bug*. vpudec is not supposed to be tested using appsink as video output, although there is a gstreamer api. (ofcourse, we all expects that everything just works)

Have you tried to figure out what would be the mismatch on the communication between vpudec and appsink? What one gives and other one expects? I´m not a gstreamer expert, I don´t know to code for it, and maybe the best place to you to ask help on this is gstreamer community itself.

[dv1]

Hello, sorry for the delay, had other things to attend to. You should install the -meta versions of the gst-plugins. To do a quick test, in your local meta-fsl-arm copy, create a "recipes-image" directory, and copy the attached file into it. Then build with bitbake the "fsl-gst-test-image" image. This should give you an image with all of the GStreamer plugins added (minus the ones in -ugly, since they must be explicitely whitelisted), including the appsink and all of the Freescale plugins.

As for your questions about the mismatch, I unfortunately did not have time do look further into that yet :smileysad:

[daiane_angolini]

I think someone else reported this on meta-freescale.

Did you enter meta-freescale?

And I saw some patches was included on master in order to "fix" the -meta for gstreamer. Do you think it could be a missing plugin?

[dv1]

Hi,

if you mean https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 , then yes, I added these bugreports.

The -meta fixes on master are unrelated. They fix a problems when trying to install -meta packages on images with Yocto. Before this patch, -meta contains dependencies to non-existing packages.

I am pretty sure the cause lies in the Aiur demuxer. I have been able to write a VPU decoder plugin for GStreamer 1.0 (unfinished; will be on github in the next few days), and it worked perfectly with appsink. It works with the GStreamer-provided demuxers and base class instead of using custom ones.

[varsmolta]

When you are writing the VPU decoder plugin for Gstreamer 1.0, what did you use as your base code? The reason I ask is the freescale released gst plugins (gst-fsl-plugins) are based off of 0.10 version. So did you rewrite the VPU plugins from scratch or port them from the 0.10 version? Could you please provide a link to the github if you have this working? Are you testing this in yocto master?

[dv1]

It is written from scratch. See my answer to Daiane Angolini above for the Github link.

[daiane_angolini]

I see

Do you know that there is a new MM version on master-next waiting for some test to be merged?

It´s 3.0.8

If you have a chance to test, please, let me know.

Nice to know you´ve been coding a VPU decoder, I would like to see it when you place in github, if possible.

[dv1]

Sorry for the long delay, I was very busy.

The plugins are available on Github, here: dv1/gst-fsl · GitHub .

They are in heavy development at the moment, so compatibility can become broken anytime. The VPU decoder is running pretty well, with the exception of VC-1 and WMV3 content, which is still a TODO. There are also plugins for transforming video frames by using the IPU, and for displaying video streams to the framebuffer with the IPU. A second sink using Vivante GLES direct textures will follow.