IMX8-MP GStreamer mp4 + vpu

[cbar001]

Hi,

I have an embedded linux with 5.10 kernel version running on IMX8 MP. I want to play some .mp4 videos in my LVDS panel via KMS. I have installed gstreamer packages (or plugins) via buildroot but i guess there is still something missing. When i try to play them i got this error:

#gst-launch-1.0 filesrc location=AD4.mp4 ! decode
bin name=dec ! videoconvert ! kmssink

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Missing element: H.264 (Main Profile) decoder
Missing element: MPEG-4 AAC decoder
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:dec: Your GStreamer installation is missing a plug-in.
Additional debug info:
../gst/playback/gstdecodebin2.c(4719): gst_decode_bin_expose (): /GstPipeline:pipeline0/GstDecodeBin:dec:
no suitable plugins found:
Missing decoder: MPEG-4 AAC (audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, level=(string)2, base-profile=(string)lc, profile=(string)lc, codec_data=(buffer)1190, rate=(int)48000, channels=(int)2)
Missing decoder: H.264 (Main Profile) (video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.1, profile=(string)main, codec_data=(buffer)014d4029ffe10015674d4029959005005bb01100000303e80000c3508401000468eb8f20, width=(int)1280, height=(int)720, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true)

ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:dec/GstQTDemux:qtdemux0: Internal data stream error.
Additional debug info:
../gst/isomp4/qtdemux.c(6545): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstDecodeBin:dec/GstQTDemux:qtdemux0:
streaming stopped, reason not-linked (-1)
ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...

I have tried to use vpudec as well and it says it is not found. ¿Any help?

#gst-launch-1.0 filesrc location=/stuff/video/horizontal.mp4 ! qtdemux name=d d.video_o ! queue ! 264parse ! vpudec ! queue ! kmssink
WARNING: erroneous pipeline: no element "vpudec"

I am able to play the video demo in frame buffer and kms with:

1-framebuffer

----------------

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1920,height=1080" ! fbdevsink

2-kms:

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1920,height=1080" ! kmssink driver-name=imx-drm

I would like to play mp4 video with and without VPU. ¿Any ideas what i am missing? avdec_h264 not appearing when i use gst-inspect.

Greetings

[cbar001]

As it is possible we share some valuable information I will open a private ticket to keep on with the conversation. Everything will be summarized there, please mark this topic as solved.

Thanks

[joanxie]

ok, I will close this one and follow your new case

[joanxie]

as I known newer bsp doesn't use kmssink anymore, use waylandsink, why do you insist on using kmssink?

[cbar001]

As we only want to play a videostream, we decide not to add a compositor to our embedded device. Therefore, we have to send the video directly using framebuffer interface or KMS.

We managed to play mp4 video using kmssink and fbdevsink with gst-launch pipelines, so i guess we almost got it. I just would like to know how to add VPU functionality to our IMX8M-Plus device to make "vpudec" pipeline work. Then we will see if lag is solved.

¿Any idea if it is possible or how can I add VPU functionality to IMX8M-Plus?

Greetings

[cbar001]

Hi again,

I achieved to play mp4 video without vpu. I had to add ffmpeg decoders. They are located in libav gstreamer plugins (BR2_PACKAGE_GST1_LIBAV). The commands to play mp4 via frame buffer / KMS are:

Framebuffer

------------------

gst-launch-1.0 -v filesrc location=video2.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! avdec_h264 ! videoconvert ! fbdevsink

kms

------------

gst-launch-1.0 -v filesrc location=video2.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! avdec_h264 ! videoconvert ! kmssink

They both play horizontal videos. However, i need to play vertical format video. It is possible to play it with fbdevsink, but no with kmssink (and im interested in the last one). These are the logs that appears when kmssink is used with vertical mp4 video.

# gst-launch-1.0 -v filesrc location=/stuff/video/vertical.m
p4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! avdec_h264 ! videoconvert !
kmssink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 1920
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 1080
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8bfdf8f800, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8bfdf8f800, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8b, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
Redistribute latency...
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8b, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8bfdf8f800, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8b, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.2, profile=(string)high, codec_data=(buffer)0164002affe1001d6764002aacd9804403c7979a808080a0000003002000000641e306334001000568e97b2c8b, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/avdec_h264:avdec_h264-0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1080, height=(int)1920, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, format=(string)BGRx, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstKMSSink:kmssink0.GstPad:sink: caps = video/x-raw, format=(string)BGRx, width=(int)1080, height=(int)1920, framerate=(fraction)25/1, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1080, height=(int)1920, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1
Redistribute latency...
ERROR: from element /GstPipeline:pipeline0/GstKMSSink:kmssink0: GStreamer encountered a general resource error.
Additional debug info:
../sys/kms/gstkmssink.c(1711): gst_kms_sink_show_frame (): /GstPipeline:pipeline0/GstKMSSink:kmssink0:
drmModeSetPlane failed: Invalid argument (22)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstQTDemux:d: Internal data stream error.
Additional debug info:
../gst/isomp4/qtdemux.c(6545): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:d:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstQueue:queue0: Internal data stream error.
Additional debug info:
../plugins/elements/gstqueue.c(990): gst_queue_handle_sink_event (): /GstPipeline:pipeline0/GstQueue:queue0:
streaming stopped, reason error (-5)
ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...

All mp4 videos are lagging. I expect to fix that enabling vpudec...

¿Any idea of why is kmssink crashing when plays vertical mp4 video?

¿Any idea of how to enable vpu in imx8m-plus?

Greetings

[joanxie]

what bsp version do you test? pls send your vertical format video to me, let me double check in my board