I.MX8MP VPU H.264 decode using v4l2

[jheaffey]

Hi,

I am trying to use the VPU to decode an H.264 stream on an I.MX8MP processor using the v4l2 API.

I can successfully use the VPU to encode YUYV to H.264, but if try to decode  H.264 (to NV12, as YUYV is not supported by the VPU when decoding) it doesn't work.

I believe I am setting up v4l2 correctly, i.e queueing output buffers, setting controls etc. No errors when doing so.

The H.264 produced by the encoder is being fed into the decoder. No matter how many H.264 buffers I provide to the decoder (using v4l2 qbuf), the v4l2 API never provides an output buffer for me to deque.

I have enabled the debug logs of the imx-vpu-hantro-daemon and it seems that the VPU could be waiting for the H.264 stream to contain a NAL unit or start code or similar, although this is present in the encoder H.264 output.

I have tried to run the VPU decoder unit test app (imx-test/test/mxc_v4l2_vpu_test at lf-6.6.3_1.0.0 · nxp-imx/imx-test · GitHub) but it seems that its not supported on the I.MX8MP, whereas the encoding unit test is supported.

Has anyone verified that decoding H.264 using the v4l2 API on the I.MX8MP works?

Any help would be greatly appreciated.

[Bio_TICFSL]

Hello,

i.MX 8M Hantro VPU with a VPU library and no firmware. This VPU has user space libraries that prepare IOCTL calls to the kernel VPU Hantro Driver. Besides the VPU wrapper API (deprecated), the V4L2-based framework on top of the user library is designed to make the integration more standardized. Check:

https://www.nxp.com/docs/en/reference-manual/IMX_VPU_APPLICATION_PROGRAMMING_INTERFACE_LINUX_REFEREN...

regards

[jheaffey · Accepted solution]

Thanks for the response.

Turns out the issue was that my application was not following the v4l2 decoder state machine. Specifically, my application did not wait for the V4L2_EVENT_SOURCE_CHANGE  event and then send the V4L2_DEC_CMD_START  command. See 4.5.1. Memory-to-Memory Stateful Video Decoder Interface — The Linux Kernel documentation

I also had to enable V4L2_CID_MPEG_VIDEO_DEC_DISPLAY_DELAY_ENABLE otherwise the decoder would require a large number of buffers of compressed video before it would output a decoded frame