gstreamer transformation plugin: buffer issue

[smspatrick]

Hi everyone,

since a week I've got trouble with a plugin with fixed sink and source pad. It should recieve a NV12 buffer and forward a Y444 one. The recieving is no Problem at all. The fixed formats are set in filter_set_caps and checked after plug them together (v4l2src -> myfilter -> vpuencoder) The problem seems to be in the chain function:

In the beginning it was a NV12 to NV12 forwarding and worked insitu as well as a new buffer with gst_buffer_copy_metadata( outbuf, inbuf, GST_BUFFER_COPY_ALL ) and a memcopy for the Data and a unref of the old buffer in the end.

Now with the new format I need to build a complete new buffer (many versions from "allocate and set" to all-in-one-APIs are available but there all produce the same error I believe I've done something very wrong a few steps before)

This buffer with fresh memory and new Metadata is made as in many source-plugins:

outbuf = gst_buffer_new_and_alloc(outsize);

gst_buffer_set_caps (outbuf, GST_PAD_CAPS (my_filter->srcpad));

as good (or bad) as:

gst_pad_alloc_buffer( filter->srcpad,

            GST_BUFFER_OFFSET_NONE,

            outsize,

            GST_PAD_CAPS (my_filter->srcpad),

            &outbuf);

after that and writing or not in the buffers data I do a

return gst_pad_push (my_filter->srcpad, outbuf);

finally, the plugin compiles and all is pluged together and begins to run, and the first frame produce:

GStreamer-CRITICAL **: gst_mini_object_unref: assertion `mini_object->refcount > 0' failed

and the framework drops Error: Internal data flow error.

just before i checked buffers and caps configuration and all is set as mentioned.

Even with a NV12 setup for the sourcepad the error occurs. But more suspicious to me is, when I copy the old buffers Meta with GST_BUFFER_COPY_ALL even on an Y444 configured Pad, the pad reconfigures without warning to NV12 and runs normally.

Tried gst_buffer_ref(outbuf),  no change. Tried gst_buffer_make_writeable(outbuf) but refcount was allready 1, no change.

Since the caps-class has it's own warning of refcount, this seems for me to came from the vpuencoder trying to remove my buffer.

Has anyone an Idea of whats going on? How can I force the encoder to accept this buffer?

Thanks

gstreamer 0.10.28

mfw_vpuencoder 2.0.3

[LeonardoSandova]

1. try using a S/W encoder and check if you see the same error (in yes, then vpuenc is not unref your buffer)

2. add debug flags so you can see the exact line where the assertion is happening (--gst-debug=your-filter-name:5)

[smspatrick]

Hi and thanks,

step two brought me to the real issue. I discovered the totally undiscribed gstBufferMeta class, which is not used in any example at all, but nearly in all fsl plugins (I think its an meta attachment but I can't detect any improvements of it exept a private-marker, so why this extension?).

But this is the reason for the refcount error, since there is no typical header at the buffers. The direct connection or through-pushing of the gst-fsl v4l source plugin to the gst-fls-vpu encoder works, since both use this gstBufferMeta class.

There is no API how to use them and bind them to buffers. I just tried to use the gstbufmeta.h and c. The look at the v4l-source of freescale use a multiple buffer system with mmap so I can't see when the beuffer meets its meta (or header) and I can't find the point where the data are leaving the source plugin (pad_push).

Would be great if there is a solution to this issue.

Thanks

[karina_valencia]

JianLi May 8, 2013 12:32 AM (in response to Karina Valencia Aguilar)

Actually I'm not quit clear about the issue you described.

Do you mean you want vpuenc accept a Y444 buffer? the vpuenc plugin can only accept YUV with I420/NV12 format, can't accept Y444 format.

Re: gstreamer transformation plugin: buffer issue

[smspatrick]

No, here's  a part of the inspect:

Factory Details:

  Long name:    Freescale: Hardware (VPU) Encoder

  Class:    Codec/Encoder/Video

  Description:    Encodes raw YUV 4:2:0 data to MPEG4 SP, H.264 BP or H.263 (Annex J, K (RS=0 and ASO=0) and T) elementary data;Encodes raw YUV 4:2:0, 4:2:2 horizontal, 4:2:2 vertical or 4:0:0 data into MJPEG elementary data;

  Author(s):    Multimedia Team <shmmmw@freescale.com>

  Rank:        primary (256)

Plugin Details:

  Name:            mfw_vpuencoder

  Description:        Encodes Raw YUV Data to MPEG4 SP, H.264 BP or H.263 data. For H.263 P0, the source frame rate must be 30000/1001 fps

  Filename:        /usr/lib/gstreamer-0.10/libmfw_gst_vpu_enc.so

  Version:        2.0.3

  License:        unknown

  Source module:    gst-fsl-plugins

  Binary package:    Gstreamer Multimedia Plugins (Freescale)

  Origin URL:        http://www.freescale.com

GObject

+----GstObject

       +----GstElement

             +----MfwGstVPU_Enc

Pad Templates:

  SRC template: 'src'

    Availability: Always

    Capabilities:

      video/mpeg

                  width: [ 48, 1280 ]

                 height: [ 32, 720 ]

            mpegversion: 4

           systemstream: false

      video/x-h263

                  width: [ 48, 1280 ]

                 height: [ 32, 720 ]

      video/x-h264

                  width: [ 48, 1280 ]

                 height: [ 32, 720 ]

      image/jpeg

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

  SINK template: 'sink'

    Availability: Always

    Capabilities:

      video/x-raw-yuv

                 format: I420

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

      video/x-raw-yuv

                 format: YV12

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

      video/x-raw-yuv

                 format: NV12

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

      video/x-raw-yuv

                 format: Y42B

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

      video/x-raw-yuv

                 format: Y444

                  width: [ 48, 8192 ]

                 height: [ 32, 8192 ]

And since it accept the Y444 as fourcc during the caps negotiation it clearly does support y444. Since it just read any 4th U and V pixel theres no need for me to rearrange it by myself. There is clearly a selection function in the plugin, so doing this before would just produce unnecessary load. And My data is just 4:4:4. That the encoded H264 data contains just an 4:2:smileylaugh: is not the issue. The issue is, that the encoder isn't able to unref a standard initialized data block.

[karina_valencia · Accepted solution]

Re: gstreamer transformation plugin: buffer issue

JianLi May 21, 2013 7:09 PM (in response to Karina Valencia Aguilar)

1. YUV444 format only for MJPEG encoding, you can find below code segment for this:

    if (vpu_enc->codec != STD_MJPG) {

      if (vpu_enc->yuv_proportion != MFW_GST_VPUENC_YUV_420) {

        /* the source image format is not 4:2:0, post error message */

        GST_ERROR

            (">>VPU_ENC: Wrong YUV format of source image for non-MJPEG encoder.");

        vpu_ret = RETCODE_INVALID_PARAM;

        break;

      }

2. that the encoder isn't able to unref a standard initialized data block.

    So you case is you have a filter plugin, this plugin allocates buffers from v4lsrc as input, and also allocates buffers from vpu encoder plugin as  output, then the filter plugin doing some transformation from the input to the output. Each time done the transformation, the filter will unref the buffer from v4lsrc, and push the vpu buffer to vpu encoder.

Is this the correct scenario?

   Then your problem is, the ref count can't be reduced to 0 for v4lsrc buffer or vpu allocated buffer?

[smspatrick]

Point 1 makes everything much clearer for me.

And Point two: the scenario is correct, the problem is: I can unref the v4lscr buffer, the vpu-encoder isn't able to unref the one that I have created.

But I Allready knew that they both (v4lscr and vpuenc) share a common gpu-mem-circle to reduce cpu load. That seems OK, since I know now that I can only work insitu on the buffer. Only thing bothered me is that freescale wasn't able to tell that much earlier, what could have save me weeks of unnecessary work.

Thanks a lot