Freescale's hardware accelerated h264 codec

[rostislav]

I installed the Freescale's codecs following i.MX Linux Mulimedia Framework User's Guide.

While trying to use a h264 codec we get the following error:

0:11.053 PExternalT...0x2f2be420 GStreamer mfw_gst_h264_getbuffer: >>DECODER: Error -4 in allocating the Framebuffer[0]

Could anybody point to a document that may help to interpret codec errors? What the  "Error -4" means?

iMX53 VPU usage

I printed that question in my old thread, but I have doubts because the old thread status assumed answered. That is a reason I reprinted the question starting new thread.

[timothybean]

Hi Robert.

What version gst-fsl plugins are you using? What BSP?

I am not sure I could help much more without being able to test it myself.

Where are you located?

Tim

[rostislav]

We use Ubuntu 10.4.

Plugins: IMX_MMCODECS_11.09 downloaded from the Freescale's website.

Root file system is taken from the Ubuntu 10.4 SD card that came with QSB.

Kernel from Freescale LTIB: 2.6.35.3-1129-g691c08a

CPU is iMX53.

We are located in Toronto.

[rjongbloed]

The appsrc/appsink is very straightforward. In essence, I start a thread that reads from a socket and "pushes" the RTP packet into the appsrc. Similarly, in another thread, I pull from the appsink and write to the socket. Of course, it is not really as simple as that, but logging tells me that thread is operating as it should. And I cannot imagine how that could cause an IPU contention.

I will try moving the queue, as soon as I get access the the machines again. Though, a question: does using a "queue" element automatically create a thread? I have in my head there is some weird syntax involving {}'s that makes threads in a pipeline.

[LeonardoSandova]

Hi Rostislav,

Seems that error comes when trying to allocate memory. Could you post the gstreamer pipeline? Also attached the produced log when the pipeline is run with the parameter '--gst-debug=*mfw*:5'

BTW, I do not know about the error number (-4), not really informative :smileysad:

Leo

[rjongbloed]

This is a test script which pretty much does exactly what the application does. Substituting x264 and running on a normal PC, this pipeline does work. On the embedded system, bang!

#!/bin/sh

ENC="mfw_vpuencoder codec-type=2"

DEC="mfw_h264decoder"

#ENC="x264enc"

#DEC="ffdec_h264"

gst-launch \

  "videotestsrc" ! \

  "ffmpegcolorspace" ! \

  "video/x-raw-yuv,format=(fourcc)I420,width=352,height=288,framerate=(fraction)90000/3000" ! \

  "$ENC" ! \

  "rtph264pay" "mtu=1444" "pt=107" ! \

  "application/x-rtp,media=video,payload=107,clock-rate=90000,encoding-name=H264" ! \

  "rtph264depay" ! \

  "$DEC" ! \

  "ffmpegcolorspace" ! \

  "ximagesink" "display=:0.0"

and this is the output:

H264D_ARM11_02.06.01  build on Jul 12 2011 13:38:27.

MFW_GST_H264_DECODER_PLUGIN 2.0.3-1-179-e630aa8d build on Dec 26 2011 16:04:09.

Setting pipeline to PAUSED ...

[INFO]  Product Info: i.MX53

VPU Version: firmware 1.4.41; libvpu: 5.3.2

MFW_GST_VPU_ENCODER_PLUGIN 2.0.3-1-179-e630aa8d build on Dec 26 2011 15:58:19.

Pipeline is PREROLLING ...

Caught SIGSEGV accessing address 0x180f

#0  0x2ad1d976 in ?? ()

#1  0x2ad31722 in ?? ()

Spinning.  Please run 'gdb gst-launch 16089' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

^CCaught interrupt -- handling interrupt.

Interrupt: Stopping pipeline ...

ERROR: pipeline doesn't want to preroll.

Setting pipeline to NULL ...

^C

the SEGV is disappointing!

[LeonardoSandova]

Try:

+ ENC="vpuenc"

+ DEC="vpudec"

+ mfw_ipucsc instead of ffmpegcolorspace

+ mfw_v4lsink instead of ximagesink

[rjongbloed]

Unfortunately, I get

WARNING: erroneous pipeline: no element "vpuenc"

and when I do a gst-inspect, I get:

lucid@lucid-desktop:~/projects$ gst-inspect | grep vpu
mfw_vpuencoder:  mfw_vpuencoder: Freescale: Hardware (VPU) Encoder
mfw_vpudecoder:  mfw_vpudecoder: Freescale: Hardware (VPU) Decoder

is there something else we should install?

[LeonardoSandova]

Oh Sorry, you are on MX53. Can you try those codecs (mfw_*) ? There is nothing more you need to install. You got the HW accelerated gstreamer elements you need to use.

Leo

[rjongbloed]

I have dug into this further. Going back to the gst-launch script, there is nothing appearing on the screen and there are occasional

[WARN]  VPU mutex couldn't be locked before timeout expired

messages appearing. And when I do the ^C, I get

ERROR: pipeline doesn't want to preroll.

which implies something is stuck. While I am not really an expert in gstreamer, the pipeline looks OK and does work on a normal PC, with changes to the codecs etc. So, it has to be something subtle with the mfw_* stuff. I have attached a full trace log file.

Any ideas?

[LeonardoSandova]

Robert, you issue will be escalated. I ran out of ideas.

Leo

[rostislav]

Leonardo,

Could you please involve anybody who is related to the freescale gstreamer elements development. We need to find a solution for the problem.

[LeonardoSandova]

Your issue has been pushed to an internal query and it is currently being investigate.

Some observations from my side:

1. Do you observe the same issue without the (de)pay elements? Please construct a pipeline with just filesrc, enc, dec and filesink.

2. Why do you encode then decode at the same pipeline? your board should be streaming ( encoding + rtppaying) and your host should be doing the playback (decoding + depaying) but you have everything on a single pipeline. I know on PC works fine, but I do not understand the purpose of that pipeline.

Leo

[rjongbloed]

1. Yes that works.But why does the (de)pay elements work fine when using codecs on a normal x86 PC?

2. This is for test purposes only. Imagine that it is really two pipelines on two different machines. As in:

gst-launch \

  "videotestsrc" ! "mfw_ipucsc" ! \

  "video/x-raw-yuv,format=(fourcc)I420,width=352,height=288,framerate=(fraction)90000/3000" ! \

  "mfw_vpuencoder codec-type=2" ! \

  "rtph264pay" "mtu=1444" "pt=107" ! \

packets are sent via RTP on a UDP socket to another machine

gst-launch \

  "application/x-rtp,media=video,payload=107,clock-rate=90000,encoding-name=H264" ! \

  "rtph264depay" ! \

  "mfw_vpudecoder" ! \

  "mfw_ipucsc" ! \

  "mfw_v4lsink"

All I did for testing was not do the packets over the wire bit.

[LeonardoSandova]

For point 2, have you run those two pipelines (on separate machines)?

Leo

[rjongbloed]

I only have one machine to play with, so I have not tried that mode.

However, the principle of having an encoder and and a decoder running in a consecutive pipeline rather than two parallel pipelines should be a valid test.

As it works without the (de)pay stages, I am guessing there is an issue with getting the NALU's in a format that mfw_decoder likes. Is there any information on that?

Do you actually know of anyone that has a working pipeline for receiving RTP packets?

[juangutierrez]

I have tried next pipelines in iMx53 and they are working fine.

These are based on Leo's examples. So, the rtppay/depay is working correctly

# SOURCE FROM PC

gst-launch-0.10 gstrtpbin name=rtpbin \

            videotestsrc ! ffmpegcolorspace ! "video/x-raw-yuv,format=(fourcc)I420,width=352,height=288,framerate=(fraction)90000/3000" ! x264enc byte-stream=true ! queue ! rtph264pay ! rtpbin.send_rtp_sink_0 \

                  rtpbin.send_rtp_src_0 ! udpsink port=5000 host=10.112.98.90                           \

                  rtpbin.send_rtcp_src_0 ! udpsink port=5001 sync=false async=false host=10.112.98.90   \

                  udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0

#SINK FROM IMX53

gst-launch -v gstrtpbin name=rtpbin                                          \

    udpsrc caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=H264" buffer-size=500000 \

            port=5000 ! rtpbin.recv_rtp_sink_0                                \

        rtpbin. ! queue max-size-buffers=1000000 ! rtph264depay ! queue max-size-buffers=1000000 ! mfw_vpudecoder ! "mfw_v4lsink"                    \

     udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                               \

     rtpbin.send_rtcp_src_0 ! udpsink port=5005 sync=false async=false

[woutervh]

Hello Juan Antonio,

I have tried the exact same pipelines you provided, but on my IMX53, I get warnings that a lot of buffers are being dropped and my pc is to slow.

The screen itself is also almost standing still (one frame every second or so)

Did you had fluent image with these?

[rjongbloed]

Thank you very much, this gives me some hints to work with.

It is a mystery why the single pipeline doe snot work, perhaps some strange contention issue with accessing the VPU that needs the encoder and decoder in different threads. As you say, not worth expending effort to fix as that is not how the "real world" apps work anyway.

[rostislav]

Robert tried running test on separate machines following his request for help:

What I have changed is only make one call per embedded host. I am guessing that the hardware can only support one encode and decoder at a time. Which is why my test pipeline using gst-launch did not work either.

Now I have a problem in that it appears to work, for a while. After a minute or so I then get continuous errors, like:

TSM:Too many timestamps recieved!! (cnt=127)

any idea what this means?


Also, is there any word on the second issue I raised:

Finally, the "half" a problem which is that the output of mfw_v4lsink is translucent, how does one alter the "alpha" for it to be opaque? It is not obvious from looking at the "gst-inspect mfw_v4lsink" command.

[LeonardoSandova]

Hi,

for the second issue, check this https://community.freescale.com/docs/DOC-1518

and set the value to 0, which is max opacity value. In the other hand, you can use mfw_isink, and you get the same result, just make sure to VSALPHA=1 before running your pipe.

Leo

[rjongbloed]

What do you mean by "make sure to VSALPHA=1" ? Is this an environment variable? What does it do?