topic Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit) in i.MX Processors

Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Fri, 22 Feb 2013 15:11:00 GMT

Hello,

I have been using the GStreamer plugins for i.MX6 video decoding using a Yocto-based rootfs. I used the recipes from Freescale/meta-fsl-arm · GitHub for building (I used the danny branch). I used "imx6qsabresd" as the machine in Yocto's local.conf.

This gave me a kernel 3.0.35, and the following package versions:

firmware-imx-vpu-imx6q - 12.09.01-r4.0

gst-fsl-plugin - 3.0.1-r9.1

imx-audio - 0.1-r4

imx-lib - 12.09.01-r4.0

libfslparser - 3.0.1-r1

libfslvpuwrap3 - 1.0.17-r0

libgal-common-mx6 - 12.09.01-r1.0

When I tried to play a h264 video with GStreamer's playbin2 , it worked fine:

gst-launch-0.10 playbin2 uri=file:///media/mmcblk1p3/sintel_trailer-1080p.mp4

However, when I use an appsink as videosink, it breaks down:

gst-launch-0.10 playbin2 uri=file:///media/mmcblk1p3/sintel_trailer-1080p.mp4 video-sink="appsink"

I get errors like:

0:00:00.531415996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.560362996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.561086330 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.601098330 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.601788996 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.601995329 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.602423663 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.602734329 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.603029996 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.603659329 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.604238663 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.604373663 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.604993996 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.605017996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.605644663 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.605856996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.606362329 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.606614329 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.606981329 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.607351996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.607581663 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

0:00:00.608115996 1015 0x2b825d80 WARN mfw_aacdecoder mfw_gst_aacdec.c:1724:mfw_gst_aacdec_data: not able to push the data

0:00:00.608310330 1015 0x2b825c60 WARN vpudec vpudec.c:1541:gst_vpudec_chain: Got no frame buffer message, return 0x89, 8 frames in displaying queue!!

I never see these errors when using another sink, like mfw_v4lsink, fakesink, or fbdevsink.

This is relevant to me, because WebKit internally uses a derivative of the appsink for the video frames. In other words, the VPU GStreamer plugins will not work with WebKit.

Are the errors above known? Is there a newer version of the Freescale video decoder plugins which fixes them?

Original Attachment has been moved to: vpu-errors.log.zip

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Tue, 26 Feb 2013 14:21:16 GMT

Well, there is a newer BSP (1.1.0), but I´m not sure it fixes this issues, because I even understand the issue (I don´t know aapsink)

Are you able to test one gstreamer command line without playbin, only to display video on appsink? ( I mean, create your own command line with decoder, demux and appsink, without audio)

Please let me know the results

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Wed, 27 Feb 2013 13:29:07 GMT

Okay, I now tried it with a manually created pipeline:

gst-launch-0.10 filesrc location=sintel_trailer-1080p.mp4 ! "video/quicktime, variant=(string)iso" ! aiurdemux name=d "d.video_0000" ! vpudec ! appsink

I attached the log (produced with GST_DEBUG=2,*aiur*:5,*vpudec*:5 ) .

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Wed, 27 Feb 2013 18:44:29 GMT

I was not able to see anything special on your log, sorry.

What would be the behavior if you use qtdemux instead of aiurdemux

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Thu, 28 Feb 2013 09:24:26 GMT

I used this pipeline for trying out qtdemux:

GST_DEBUG=2,*aiur*:5,*vpudec*:5 gst-launch-0.10 filesrc location=sintel_trailer-1080p.mp4 ! "video/quicktime, variant=(string)iso" ! qtdemux name=d "d.video_00" ! vpudec ! appsink

And it didn't work either, but the errors are different now. I attached the new log.

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Thu, 28 Feb 2013 16:16:37 GMT

So, I suppose it´s an error on the decoder-sink interface

I suppose you have tested the same video on you linux PC and it worked.

And I suppose you have tried a non vpu accelerated decoder on the board, and it worked.

How could I install appsink in my rootfs? It was not installed by default. Are you using prebuilt LTIB image? Please, let me know

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Thu, 28 Feb 2013 16:22:41 GMT

Your assumptions are correct. The videos work fine on the PC, and they also work fine with other decoders.

I did not use LTIB, I used Yocto instead. Yocto 1.3 (with poky 8.0 "danny"), to be exact. I also added Freescale's meta-fsl-arm layer, to get the VPU plugins.

The appsink is part of the GStreamer base plugins package. The name in the LTIB package list is "gstreamer-plugins-base" : https://support.bluetechnix.at/wiki/Qt_Embedded_on_i.MX53#LTIB_configuration_.28Qt_Embedded.29

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Thu, 28 Feb 2013 16:27:45 GMT

Ow! you´re using yocto!

appsink is default installed on it? I have fsl-image-gui and fsl-image-test already built for sabre sd (I suppose the board will not be a problem)

Do you think I could use some of those images?

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Thu, 28 Feb 2013 16:48:16 GMT

These images should be OK, but you should modify their recipes to include "gst-plugins-base-meta" to the EXTRA_IMAGE_FEATURES list (for example, look at the fsl-image-test recipe: meta-fsl-demos/recipes-fsl/images/fsl-image-test.bb at master · Freescale/meta-fsl-demos · GitHub ) . This installs all plugins from the base package (including appsink). By default, these images do not install it.

To be able to test other decoders etc. it is probably best to just add all of GStreamer, so I add these entries to the list:

	gst-plugins-base-meta
	gst-plugins-good-meta
	gst-plugins-bad-meta
	gst-plugins-ugly-meta
	gst-ffmpeg

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Thu, 28 Feb 2013 17:06:21 GMT

Ok!

Let me bitbake it and I update you as soon as I have tested

(sorry, it can take 1 or 2 days until I´m able to effectively test it, but I´m here)

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Tue, 05 Mar 2013 14:52:15 GMT

It also happens with newer versions:

firmware-imx-vpu-imx6q : 1:1.1.0-r4.0

gst-fsl-plugin : 3.0.5-r9.3

imx-audio : 0.1-r4

imx-lib : 1:1.1.0-r4.0

libfslparser : 3.0.5-r1

libfslvpuwrap3 : 1.0.17-r0

libgal-common-mx6 : 12.09.01-r2.0

libgstfsl-0.10-0 : 3.0.5-r9.3

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Tue, 05 Mar 2013 16:16:46 GMT

ok thanks. I think I had already bitbaked the image, I´m about to test it. asap I let you know :smileywink:

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Wed, 06 Mar 2013 19:38:35 GMT

I tested a lot of variations and I was not able to get appsink in my rootfs. Can you send me a patch?

I don´t think it´s a fsl-plugins *bug*. vpudec is not supposed to be tested using appsink as video output, although there is a gstreamer api. (ofcourse, we all expects that everything just works)

Have you tried to figure out what would be the mismatch on the communication between vpudec and appsink? What one gives and other one expects? I´m not a gstreamer expert, I don´t know to code for it, and maybe the best place to you to ask help on this is gstreamer community itself.

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Sat, 30 Mar 2013 15:44:19 GMT

Hello, sorry for the delay, had other things to attend to. You should install the -meta versions of the gst-plugins. To do a quick test, in your local meta-fsl-arm copy, create a "recipes-image" directory, and copy the attached file into it. Then build with bitbake the "fsl-gst-test-image" image. This should give you an image with all of the GStreamer plugins added (minus the ones in -ugly, since they must be explicitely whitelisted), including the appsink and all of the Freescale plugins.

As for your questions about the mismatch, I unfortunately did not have time do look further into that yet :smileysad:

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Mon, 01 Apr 2013 16:23:26 GMT

I´m really busy this and next week, so please, bring this issue up again when you re-start to work on this

Thanks a lot in advance.

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

varsmolta — Sat, 27 Jul 2013 16:41:19 GMT

Were you able to get appsink working with the gstreamer VPU plugins? I am looking to do something similar and would like to have an idea about its feasibility before I get started. Thanks

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Tue, 30 Jul 2013 13:23:17 GMT

I think someone else reported this on meta-freescale.

Did you enter meta-freescale?

And I saw some patches was included on master in order to "fix" the -meta for gstreamer. Do you think it could be a missing plugin?

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Tue, 30 Jul 2013 13:37:39 GMT

No, it has not been fixed yet. I also added bug reports to the meta-fsl-arm bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and https://bugzilla.yoctoproject.org/show_bug.cgi?id=4156 .

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

dv1 — Tue, 30 Jul 2013 13:40:53 GMT

Hi,

if you mean https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 , then yes, I added these bugreports.

The -meta fixes on master are unrelated. They fix a problems when trying to install -meta packages on images with Yocto. Before this patch, -meta contains dependencies to non-existing packages.

I am pretty sure the cause lies in the Aiur demuxer. I have been able to write a VPU decoder plugin for GStreamer 1.0 (unfinished; will be on github in the next few days), and it worked perfectly with appsink. It works with the GStreamer-provided demuxers and base class instead of using custom ones.

Re: Problems with GStreamer 0.10 VPU plugins and Appsink (and WebKit)

daiane_angolini — Tue, 30 Jul 2013 19:02:33 GMT

I see

Do you know that there is a new MM version on master-next waiting for some test to be merged?

It´s 3.0.8

If you have a chance to test, please, let me know.

Nice to know you´ve been coding a VPU decoder, I would like to see it when you place in github, if possible.