mfw_vpudecoder failed to work on MX27

[RobbieJiang]

Hi all,

I'm testing mfw_vpudeocder plugin on MX27 platform. But it failed to work.

VPU Version: firmware 2.2.5; libvpu: 3.3.2.

The linux kernel is 2.6.19.2. And the mfw_vpuencoder works well.

Following is the my test pipeline:

gst-launch-0.10 -v --gst-debug=mfw_vpudecoder:5 videotestsrc is-live=1 pattern=snow ! 'video/x-raw-yuv,format=(fourcc)I

420,width=352,height=288,framerate=25/1' ! mfw_vpuencoder codec-type=std_avc width=352 height=288 framerate=25 ! mfw_vpudecod

er codec-type=std_avc  !  fakesink

Here I use fakesink instead of mfw_v4lsink, since I have no LCD on hand.

Or mfw_vpudecoder MUST to work with mfw_v4lsink?

There are two error message reported by the debug outputs :

1. mfw_vpudecoder mfw_gst_vpu_decoder.c:882:mfw_gst_vpudec_copy_data: >>VPU_DEC: vpu_DecGetBitstreamBuffer returned zero space do a reset

2. mfw_vpudecoder mfw_gst_vpu_decoder.c:1748:mfw_gst_vpudec_vpu_init: >>VPU_DEC: vpu_DecGetInitialInfo failed. Error code is -10 handle 104624

Anyone know what's the reason? Or how can I fix it and make mfw_vpudecoder works? Or is there anything wrong with my "libvpu.so"?

Note that if I set mfw_vpudecoder's property "loopback" to "true", no such error message is reported.

For your information, the detailed debug outputs are enclosed as follows.

0:00:01.417778948  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:180:mfw_gst_vpudec_set_property: >>VPU_DEC: codec=2

Setting pipeline to PAUSED ...

0:00:01.501541654  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3346:mfw_gst_vpudec_change_state:

>>VPU_DEC: State: Null to Ready

Product Info: i.MX27

VPU Version: firmware 2.2.5; libvpu: 3.3.2

MFW_GST_VPU_DECODER_PLUGIN  build on Feb  2 2012 15:20:13.

0:00:01.550218572  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0xa for VPU returned 1

0:00:01.551200677  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3571:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled next 0xa

Product Info: i.MX27

VPU Version: firmware 2.2.5; libvpu: 3.3.2

MFW_GST_VPU_ENCODER_PLUGIN  build on Feb  2 2012 15:20:16.

0:00:01.582081278  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3376:mfw_gst_vpudec_change_state:

>>VPU_DEC: State: Ready to Paused

0:00:01.582774887  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0x13 for VPU returned 1

0:00:01.583197217  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3571:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled next 0x13

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)25/1, color-matrix=(string)sdtv, chroma-site=(string)mpeg2

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

0:00:01.630560076  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3459:mfw_gst_vpudec_change_state:

>>VPU_DEC: State: Transition to Playing new clock 2458872

0:00:01.631409023  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0x1c for VPU returned 1

0:00:01.632360451  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3571:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled next 0x1c

New clock: GstSystemClock

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)25/1, color-matrix=(string)sdtv, chroma-site=(string)mpeg2

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)25/1, color-matrix=(string)sdtv, chroma-site=(string)mpeg2

/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)25/1, color-matrix=(string)sdtv, chroma-site=(string)mpeg2

/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)25/1, color-matrix=(string)sdtv, chroma-site=(string)mpeg2

/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:src: caps = video/x-h264, height=(int)288, width=(int)352, framerate=(fraction)25/1

0:00:01.773515940  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:759:mfw_gst_vpudec_vpu_open: >>VPU_DEC: codec=2

0:00:01.818350677  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:830:mfw_gst_vpudec_vpu_open: >>VPU_DEC: Setting file play mode to 0

0:00:01.820493910  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:1752:mfw_gst_vpudec_vpu_init: >>VPU_DEC: min buffer count= 3

0:00:01.821309324  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:1755:mfw_gst_vpudec_vpu_init: >>VPU_DEC: InitialInfo picWidth: 352, picHeight: 288, frameRate: 0

/GstPipeline:pipeline0/MfwGstVPU_Dec:mfwgstvpu_dec0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)352, height=(int)288, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, crop-top-by-pixel=(int)0, crop-left-by-pixel=(int)0, crop-right-by-pixel=(int)0, crop-bottom-by-pixel=(int)0, num-buffers-required=(int)5

0:00:01.919564285  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:732:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: Use Non-DR mode.

0:00:01.920372405  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:735:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: buffer:0, type:FB_TYPE_HW

0:00:01.921341503  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:735:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: buffer:1, type:FB_TYPE_HW

0:00:01.922318947  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:735:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: buffer:2, type:FB_TYPE_HW

0:00:01.923294811  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:735:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: buffer:3, type:FB_TYPE_HW

0:00:01.924258721  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:735:mfw_gst_vpudec_FrameBufferInit: >>VPU_DEC: buffer:4, type:FB_TYPE_HW

0:00:01.925261653  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:1997:mfw_gst_vpudec_vpu_init: >>VPU_DEC: VPU Initialization complete

0:00:01.998142781  1601   0x1194d0 DEBUG	mfw_vpudecoder mfw_gst_vpu_decoder.c:3707:mfw_gst_vpudec_setcaps: >>VPU_DEC: set framerate nu 25 de 1

0:00:01.999006014  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3715:mfw_gst_vpudec_setcaps: >>VPU_DEC: Time per frame 40000000

0:00:01.999956992  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3718:mfw_gst_vpudec_setcaps: >>VPU_DEC: Frame Rate = 25

0:00:02.000909473  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3720:mfw_gst_vpudec_setcaps: >>VPU_DEC: Input Width is 352

0:00:02.001874736  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3722:mfw_gst_vpudec_setcaps: >>VPU_DEC: Input Height is 288

0:00:02.002798195  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3741:mfw_gst_vpudec_setcaps: >>VPU_DEC: Assuming parser input using file play mode

/GstPipeline:pipeline0/MfwGstVPU_Dec:mfwgstvpu_dec0.GstPad:sink: caps = video/x-h264, height=(int)288, width=(int)352, framerate=(fraction)25/1

0:00:02.890958722  1601   0x1194d0 ERROR

mfw_vpudecoder mfw_gst_vpu_decoder.c:882:mfw_gst_vpudec_copy_data: >>VPU_DEC: vpu_DecGetBitstreamBuffer returned zero space do a reset

0:00:02.891782256  1601   0x1194d0 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:1354:mfw_gst_vpudec_reset: >>VPU_DEC: Reset close and reopen using same previously allocated memory

0:00:02.892745339  1601   0x1194d0 ERROR

mfw_vpudecoder mfw_gst_vpu_decoder.c:1748:mfw_gst_vpudec_vpu_init: >>VPU_DEC: vpu_DecGetInitialInfo failed. Error code is -10 handle 104624

ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data flow error.

Additional debug info:

gstbasesrc.c(2582): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:

streaming task paused, reason error (-5)

Execution ended after 1262809399 ns.

Setting pipeline to PAUSED ...

0:00:02.899099474  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3465:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled 0x23

0:00:02.899534587  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0x23 for VPU returned 1

0:00:02.899934436  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3479:mfw_gst_vpudec_change_state:

>>VPU_DEC: State: Playing to Paused

Setting pipeline to READY ...

0:00:02.903812782  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3465:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled 0x1a

/GstPipeline:pipeline0/MfwGstVPU_Dec:mfwgstvpu_dec0.GstPad:src: caps = NULL

/GstPipeline:pipeline0/MfwGstVPU_Dec:mfwgstvpu_dec0.GstPad:sink: caps = NULL

0:00:02.907580226  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0x1a for VPU returned 1

0:00:02.908171730  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3487:mfw_gst_vpudec_change_state:

>>VPU_DEC: State: Paused to Ready

0:00:02.908536166  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3190:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: Cleanup frame started 0

(gst-launch-0.10:1601): GStreamer-CRITICAL **: gst_mini_object_unref: assertion `GST_IS_MINI_OBJECT (mini_object)' failed

0:00:02.941636842  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3240:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: Before close frame_started=0 vpu busy 0

0:00:02.942492106  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3248:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: After close

0:00:02.962042256  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3310:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: End Cleanup

/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:src: caps = NULL

/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = NULL

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = NULL

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = NULL

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = NULL

Setting pipeline to NULL ...

0:00:03.003003835  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3465:mfw_gst_vpudec_change_state:

>>VPU_DEC: State unhandled 0x11

0:00:03.004044587  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3470:mfw_gst_vpudec_change_state:

>>VPU_DEC: State Change 0x11 for VPU returned 1

>>VPU_DEC: State: Ready to Null

0:00:03.005817970  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:381:vpu_mutex_lock: >>VPU_DEC: lock mutex is NULL cnt=0

0:00:03.006836918  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3190:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: Cleanup frame started 0

0:00:03.007787820  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:3310:mfw_gst_vpudec_cleanup: >>>>VPU_DEC: End Cleanup

vpu_uninit

Freeing pipeline ...

0:00:03.025310301  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:4029:mfw_gst_vpudec_finalize: >>VPU_DEC: class finalized.

0:00:03.026125264  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:4029:mfw_gst_vpudec_finalize: >>VPU_DEC: class finalized.

0:00:03.111815414  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:4329:mfw_gst_vpudec_vpu_finalize: >>VPU_DEC: Destructor - final cleanup

0:00:03.112613910  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:4338:mfw_gst_vpudec_vpu_finalize: >>VPU_DEC: Cleanup already done before destructor time

0:00:03.113565188  1601

0x170b8 DEBUG

mfw_vpudecoder mfw_gst_vpu_decoder.c:4346:mfw_gst_vpudec_vpu_finalize: >>VPU_DEC: vpu instance 0x10f820 destroyed.

Regards,

Jie

[jimmychan · Accepted solution]

seems you are using an old BSP that with the kernel is 2.6.19.2. Please try the i.MX27PDK BSP. You can download it from here: LDK_IMX27_R1_1

[RobbieJiang]

Thanks for your advice.

I have managed to make mfw_vpudecoder work well with LDK_IMX27_R1_1.

Now both mfw_vpuencoder and mfw_vpudecoer work well separately.

However, when I tried to run two gstreamer pipelines (one for encoding, and the other for decoding) simultaneously, the kernel outputs "VPU has already been opened." for the pipeline which is run a bit later.

In kernel, I find that the vpu driver (mxc_vpu.c) limits that only one open is allowed by keeping "open_count". Does this mean that multiple simultaneous encoder and decoder pipelines are impossible due to the kernel driver limit?

And if so, is it possible to fix it by modifying vpu driver?  And how?

Any suggestion is appreciated.

[daiane_angolini]

As I remember you can have one encoder and one decoder instance simultaneously.

If you gst-inspect mfw_vpudecoder you will find a "loopback" parameter

You should use loopback=true

Please, let me if you was able to run the loopback test

[RobbieJiang]

The loopback test is OK.

But I need to run encoder and decoder simultaneously in two different threads or processes.

When I tried to do this, I got such errors.

Anyone succeeds to do this?

[daiane_angolini]

yes, but when you set you your decoder tread you will use loopback= true in order to tell decoder to not block VPU.

So, loopback=true is only to not block VPU for decoding. You can use it with 2 separated encoder & decoder scenario.

(as I remember I was able to do that sometime ago...)

[RobbieJiang]

Hi Daiane,

Although the loopback test is ok, I still got errors and failed to set up two different gstreamer pipelines (one is mfw_vpuencoder, the othrer is mfw_vpudecoder), even if I set "loopback=true" for mfw_vpudecoder.

Here is the pipeline I used to test loopback:

gst-launch-0.10 -v videotestsrc is-live=1 ! 'video/x-raw-yuv,format=(fourcc)I420,width=352,height=288,framerate=25/1' ! mfw_vpuencoder codec-type=std_avc width=352 height=288 framerate=25 ! mfw_vpudecoder codec_type=std_avc loopback=true  min_latency=true ! mfw_v4lsink sync=false

Here the "loopback=true" is set and the loopback pipeline works well.

Then I tried to test two encoder/decoder instances with a PC (192.168.1.222) and a MX27 board (192.168.1.254). Here are the pipelines.

On PC (192.168.1.222):

Sender:

gst-launch -v filesrc location=./BEYOND-AMANI.mp4  ! qtdemux ! rtph264pay ! udpsink host=192.168.1.254 port=5434

Receiver:

udpsrc port=5434 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)MP4V-ES, profile-level-id=(string)2, config=(string)000001b002000001b59113000001000000012000c888800cd0b042414103, payload=(int)96, ssrc=(uint)3686574436, clock-base=(uint)1422970018, seqnum-base=(uint)63140' ! rtpmp4vdepay ! decodebin ! xvimagesink sync=false

And on MX27 (192.168.1.254)

Receiver (run in one terminal):

gst-launch-0.10 -v udpsrc port=5434 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name= (string)H264, sprop-parameter-sets=(string)\"Z0LAM6tAWh7QgAAAAwCAAAAZR4wZUA\\=\\=\\,aM48gA\\=\\=\", payload=(int)96, ssrc=(uint)2316533906, clock-base=(uint)2197594482, seqnum-base=(uint)15288' ! rtph264depay ! mfw_vpudecoder codec-type=std_avc min_latency=true loopback=true ! mfw_v4lsink

Sender (run in another terminal):

gst-launch-0.10 -v videotestsrc is-live=1 ! 'video/x-raw-yuv,format=(fourcc)I420,width=352,height=288,framerate=25/1' !  mfw_vpuencoder codec-type=0 width=352 height=288 framerate=25 loopback=false ! queue ! rtpmp4vpay ! udpsink host=192.168.1.222 port=5434

Then when I started the sender pipeline on the MX27 board after the receiver had been started, I got the following outputs:

Setting pipeline to PAUSED ...

Can't open /dev/mxc_vpu

IOSystemInit() failure.

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

And as you see, the sender pipeline on MX27 failed to work although the "loopback=true" is set in the receiver pipeline.

Actually, in the vpu driver (kernel/driver/mxc/vpu/mxc_vpu.c), the vpu_open() method is defined as follows (both in MX27 ADS ltib and PDK ltib):

static int vpu_open(struct inode *inode, struct file *filp)

{

    if (open_count++ == 0) {

         filp->private_data = (void *)(&vpu_data);

         vpu_hardware_enable();

    } else {

        printk(KERN_ERR "VPU has already been opened.\n");

        return -EACCES;

    }

    return 0;

}

It seems that the vpu device (/dev/mxc_vpu) ONLY allows one open by keeping the open count with the "open_count" variable. So if the receiver has already opened the vpu device, there is no more chance for the sender pipeline to open the same vpu device for sending video stream, because the kernel will simply return "-EACCES" in this case. And this situation can't be solved by simply setting "loopback=true" property of mfw_vpudecoder.

And also on page 3-9 of the file "i.MX27 Linux Multimedia Framework User's Guide" (Document Number: 924-76396,by Freescale, 2009-11), it says that "The current i.MX27 BSP implementation does not support multiple encoder or decoder instances running at the same time. This is not a limitation in the VPU hardware but in the VPU driver."

So, both for ADS and PDK ltib for MX27, it seems that it is not possible to set up two vpu (mfw_vpuencoder and mfw_vpudecoder) pipelines without modifying the kernel vpu driver. How could it be possible to achieve it by simply setting "loopback=true" property of mfw_vpudecoder?  Would you please try it on a real MX27 platform and see if it really works?

FYI, my VPU firmware version is 2.2.5 and the vpu library (libvpu) version is 3.3.2, and the kernel is 2.6.19.2. (Note the vpu_open() mtethod in kernel vpu driver of LTIB PDK is the same as the one in my kernel.)

[daiane_angolini]

I´m not sure I still have imx27 at office, if i find it I can try. But i will back to office only next Wednesday.

Do you have the VPU api doc? You can enter a SR and ask for this doc, i may help you to implement multi instance encoder/decoder

[RobbieJiang]

Daianae,

I do have the VPU API doc, but have no idea how to implement multi encoder/decoder instances by modifying VPU library.

Would you be more detailed about this?

[daiane_angolini]

sorry. I don´t have.

I would suggest you to start a new discussion asking for help on implementing this.

Maybe someone could help you, and this discussion was already marked as "answered"

Message was edited by: Daiane Angolini