gstglupload: early rejection of non-RGBA output format in DirectVIV uploader

[zdobersek]

I am running on iMX6, using the 5.4.47_2.2.0 BSP releases.

I am handling a GStreamer pipeline that is pretty similar to a pipeline constructed by invoking `gst-play-1.0` and using `glimagesink` as the output sink. The main difference is that instead of glimagesink, I am using a custom appsink that can handle RGBA but also multiple YUV formats.

The multitude of supported formats in the sink is causing the DirectVIV uploader in the glupload element to reject the buffer, moving over to the next uploader, which is not usable.

In short, initially the negotiated caps on the glupload's src element are I420, due to that being the format that's coming from the VPU decoder down the pipeline into glupload, and because the sink can support that. Problem is that once the first upload is started, internally glupload has yet to pick an uploader, which in this case would preferably be DirectVIV.

Because of the current I420 caps, the DirectVIV upload implementation rejects the buffer early, here:

https://source.codeaurora.org/external/imx/gst-plugins-base/tree/gst-libs/gst/gl/gstglupload.c?h=MM_...

Due to this rejection I end up using the raw uploader, which doesn't work as expected once I want to sample from the buffer.

If that check is removed, the DirectVIV uploader accepts the buffer with no problems, and the caps on the glupload and through the pipeline down to the custom sink are reconfigured to RGBA. At that point, I am able to map and sample from such uploaded buffer without a problem.

I would like to understand whether this is an acceptable fix, or whether there are different ways to address this.

[joanxie]

do you mean the code like this?

if (fmt != GST_VIDEO_FORMAT_RGBA) return FALSE;

why don't you use glimagesink? because you want to use DirectVIV  which isn't supported by glimadesink? I found a patch about add DirectVIV  in the glimadesink, I attached, maybe you can refer to it

[zdobersek]

Yes, this if-statement is where the upload is rejected.

We use a generic cross-platform video sink that supports sampling from different RGBA and YUV formats and enables compositing video content into more complex GL scenes. That's why glimagesink is not usable for us.

I don't think moving the DirectVIV upload functionality into the sink is the proper approach, since this functionality already resides in glupload, and it functions well -- but only for glimagesink, because of the specific RGBA-only caps.

[joanxie]

I searched again and  found another patch for yuv to rgb by DirectVIV , maybe you can refer to it