How to correctly sample dmabuf with AMPHION_TILED modifier?

[zdobersek]

Relevant details of the iMX8QM setup I am running, based on Yocto dunfell branches:

* iMX Kernel 5.4.47

* imx-gpu-viv 6.4.0.p2.4

* GStreamer stack uses branches MM_04.05.05_2005_L5.4.24, based on upstream 1.16.1

Working on an iMX8QM with the Amphion Malone codec, the v4l2 decoder element in a GStreamer pipeline outputs a NV12 dmabuf object with the DRM_FORMAT_MOD_AMPHION_TILED modifier. I would like to understand whether it's possible to use OpenGL ES to correctly sample from such dmabufs.

Currently, the following playbin invocation incorrectly samples from the decoded dmabufs, producing corrupt/incorrect output (for any H264 or VP8 video stream):

$ gst-launch-1.0 playbin uri=file:///home/root/video_512x512.mp4 video-sink=glimagesink`

Shortly, what happens in the pipeline is that the direct-dmabuf uploader is used in glupload, and that creates an EGLImage wrapping that dmabuf. The same uploader also transforms the caps to the RGBA format, even if the underlying dmabuf is NV12. The EGLImage is then bound to a GL texture through which it is sampled in the glimagesink, as if it were an RGBA texture. All problems bundled together lead to incorrect sampling.

I would like to understand whether the GL drivers actually support sampling from EGLImages that wrap these kinds of dmabufs. If there are existing examples of how this functions, I would love to inspect those so that the GStreamer elements (as well as our custom GStreamer sink, which works similarly to glimagesink) can appropriately handle this data in the most efficient way possible.

[joanxie]

if you use the command failed

$ gst-launch-1.0 playbin uri=file:///home/root/video_512x512.mp4 video-sink=glimagesink

do you mind give the logfile to check?

[zdobersek]

I used this command:

$ GST_DEBUG_DUMP_DOT_DIR=/tmp GST_DEBUG_NO_COLOR=1 GST_DEBUG="3,gl*:6,v4l2*:6" gst-launch-1.0 playbin uri=file:///home/root/video_512x512.mp4 video-sink=glimagesink 2>gst_log.txt

Attached is the whole log, screenshot showing the wrong output, and a tarball containing the origianl dot file dump for the GStreamer pipeline as well as the SVG representation of it.

[brandon_shibley]

@joanxie can you share an update on this issue?

[joanxie]

sorry for my delay, could you send the mp4 test file to me, let me reproduce this on my board

[zdobersek]

Attached is the file used for this test, but the issue is reproducible for any H264 or VP8 video file.

[joanxie]

For imx8, I checked that vpu output tiled format, can’t be support GPU, so you couldn’t test vpu with g2d or GPU3D, if you change to the waylandsink , which should be ok, but I haven’t tested them, because I worked from home, I forgot to take the type C, if you are urgent and need to test again, I can go back to office earlier to test for you. If you disable vpu, you can fix this issue, which means what I said

[zdobersek]

Thanks for testing.

FWIW waylandsink is able to present this correctly, as long as the compositor can push the dmabuf into a KMS plane able to present this data, or draws it with G2D.

Since it's not possible to correctly sample this data through GL shaders, I'll revert to using a G2D converter element.

[joanxie]

yes, the imx8qm is different from imx8mq, the vpu output is tiled format, I still want you know that To decode H.264 bit stream with resolution equal or bigger than your current monitor resolution, take gstreamer as example, you may manually add imxvideoconvert_g2d though you don't ask for this