i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

[ryanerb]

The Freescale i.MX6 has many video capabilities that are best accessed through GStreamer.

Gateworks, the leading supplier of Powerful ARM based Single Board Computer solutions using the Freescale i.MX6, has invested countless engineering hours researching and mastering GStreamer for the i.MX series of processors. Gateworks would like to share this GStreamer research with the rest of the i.MX community of developers! Visit the Gateworks Software Wiki GStreamer Page.

There are two main versions of GStreamer: 0.10 and 1.0. Version 1.0 is now the latest standard. gstreamer-1.0​

The i.MX6 processor has hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these hardware blocks is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc.

The GStreamer app works with 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the imxvpudec is a VPU based decoder plugin.

This post is specifically about the plugins. There are different versions and sets of plugins available.

Gateworks has chosen to use the GStreamer-imx plugins for the following reasons:

• Open Source Development model: The project is on github and is very active
• The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
• The source is very well documented and easy to follow
• Things are done in a very standard GStreamer way

Plugin List

For a thorough description of each plugin, and why to use it, please visit the Gateworks Software Wiki GStreamer Page

Type	Plugin(s)	Element(s)	Comments
Audio Decoder	imxaudio	imxuniaudiodec	Uses i.MX uniaudio codecs for decoding
Audio Encoder	imxaudio	imxmp3audioenc	Uses i.MX for MP3 audio encoding
Device Sources	imxv4l2video	imxv4l2videosrc	Get camera source via v4l2
Video Decoder	imxvpu	imxvpudec	VPU Based decoder
Video Encoder	imxvpu	imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263	VPU Based encoders
Video Render (sink)	imxg2d; imxpxp; imxeglvivsink; imxipu	imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink	g2d1, ipu1, pxp2, and egl (overlay) video sinks
Video Converter	imxg2d; imxpxp; imxipu	imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform	g2d, pxp, egl and ipu video filter/converter/scalars3
Video Compositing	imxg2d; imxipu	imxg2dcompositor, imxipucompositor	gpu/ipu accelerated compositing

1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.

2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.

3. Please see note 1 above.

Plugin Example Pipeline

For example, to encode a video from a camera on /dev/video2 into h.264 and save it to a file:

#Take camera input /dev/video2, encode it to h264 at a bitrate of 10mbit/s (CBR) and save to a file.
gst-launch-1.0 imxv4l2videosrc device=/dev/video2 ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

Many more pipeline examples are described and listed on the Gateworks Software Wiki GStreamer Pipelines page

Summary

Using GStreamer 1.0 with the GStreamer-imx plugins is a powerful way to access and apply the multimedia capabilities of the Freescale i.MX6 processors!

If there are other examples you would like to see, please add to the discussion!

arpan_chakravarty​

KevinWong​

[tutran]

Dear Tim,

Oh, I am sorry for lacking of describing more details.

I am using OV5640 camera sensor MIPI with iMX6sabreqsd board.

From imxv4l2src, I saw that it can support higher resolution up to 1920x1080p 15fps.

I also tried with a simpler pipeline:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, framerate=(fraction)15/1' ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

But the same error happened. I can not link the pipeline, the debug log said that the "caps is incompatible".

In my environment, there is no plugin imxv4l2videosrc, I used the native one imxv4l2src instead.

When I used gst-inspect-1.0 for imxv4l2src, I saw a list of capabilities:

Capabilities:

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 640

                 height: 480

              framerate: { 30/1, 15/1 }

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 320

                 height: 240

              framerate: { 30/1, 15/1 }

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 720

                 height: 480

              framerate: { 30/1, 15/1 }

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 720

                 height: 576

              framerate: { 30/1, 15/1 }

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 1280

                 height: 720

              framerate: { 30/1, 15/1 }

      video/x-raw

                 format: { I420, NV12, YUY2, UYVY }

                  width: 1920

                 height: 1080

              framerate: 15/1

And also for the imxvpuenc_h264 plugins.

As my understanding, I can create a capsfilter for choosing the format linking between imxv4l2src and imxvpuenc_h264. Is it correct? Or there is any other constraint when working with a specific camera device.

Please help me to make it clear.

Thanks a lot for your support Tim

[timharvey]

Tu Tran,

You must be using a different/patched version of gstreamer-imx due to the

naming difference of the source plugin but still your issue is that not

everything you are specifying in your capsfilter can be accomplished by the

source. As I mentioned I needed to drop the framerate from the capsfilter

and instead alter the framerate with the videorate plugin.

Try:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw,

format=(string)I420, width=(int)1920, height=(int)1080' ! imxvpuenc_h264

bitrate=10000 ! filesink location=/tmp/file.mp4

If this doesn't work then I would suggest its something to do with the

ov5640 slave driver and/or your version of gstreamer-imx.

The pipelines I suggested work on the Gateworks Ventana boards with our

Yocto 1.8 BSP. I don't have your hardware and can't test your situation.

Regards,

Tim

On Tue, Mar 15, 2016 at 8:37 PM, tutran <admin@community.freescale.com>

[tutran]

Dear Tim,

I got one more problem with the imxvpuenc_h264 plugin that need your support.

When I list all the encoder plugins for iMX I see there two

- vpuenc

- imxvpuenc_h264

In my practice my these two plugins on iMX6DL board (kernel 3.14.28), gstreamer version 1.4.1. I see some stranges

1/. I tried to captured the image using camera and save them to MP4 file

- With vpuenc:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! vpuenc gop-size=30 bitrate=512 ! filesink location=/home/root/test_vpu.mp4

- With imxvpuenc_h264:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! imxvpuenc_h264 gop-size=30 bitrate=512 ! filesink location=/home/root/test_encode.mp4

In two cases, VLC player cannot render the mp4 files and only output from imxvpuenc_h264 can be played using default video app of Ubuntu, or Mplayer. The one output from vpuenc cannot be played by any decoder

2/. For streaming using RTP/UDP protocol

- With vpuenc:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! vpuenc gop-size=30 bitrate=512 ! h264parse ! rtph264pay name=pay0 pt=96 timestamp-offset=0 ! udpsink host=192.168.0.100 port=5000&

- With imxvpuenc_h264:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! video/x-raw, format=I420,width=1920, height=1080 ! imxvpuenc_h264 gop-size=30 bitrate=512 ! h264parse ! rtph264pay name=pay0 pt=96 timestamp-offset=0 ! udpsink host=192.168.0.100 port=5000 &

I use VLC client instead of gstreamer, and with imxvpuenc_h264, I cannot display the video data streaming from the pipeline although wireshark can catch the UDP packets. But it can work perfectly with the pipeline using vpuenc to encode data.

Please give me any hint or statement for the phenomenon. The issues really made me confused about the root cause.

[ryanerb]

Regarding VLC, you likely will need the mp4mux and possibly the -e for termination:

gst-launch-1.0 -e videotestsrc pattern=0 ! imxvpuenc_h264 bitrate=10000 ! h264parse ! mp4mux ! filesink location=/tmp/file3.mp4

[tutran]

Hi Ryan,

Thanks for your suggestion. It still not work with me. VLC cannot play the video either.

[wjmazepas]

Does anyone had the same problem than me?
I installed as say in the git page of the gstreamer-imx, but when i run gst-inspect-1.0 | grep imx, it can only find the imxaudio, the others packages he say that he cant find, but everything is there, i checked a multiple times and it keeps showing the same problem

[wjmazepas]

Got it. Had to do it like this guy did it
Tiny Devices: I.MX6 - gstreamer-imx

[rans]

Hi, 

We are using yocto morti.

Must we add this plugin using by manually building from git ?

Can't they be added with yocto ?

Regards,

Ran

[timharvey]

Welmo,

Glad you got it figured out. We do have some documentation on how to build gstreamer-imx with IPU/VPU/GPU support for Ubuntu ( ventana/ubuntu – Gateworks​ ) and Debian ( ventana/debian – Gateworks ).

Tim

[wjmazepas]

Thanks Tim. Sorry to trouble you again, but i am facing a new problem with the imx package. I posted this thread: Streaming webcam video with the gstreamer-imx
If you could help me, i would be verry happy mate. I'm trying a lot of crazy stuff here but nothing works

[jotes]

Hello!

Is it possible to make colorspace conversion for format 1920x1080 using imxipuvideotransform? I get the following error:

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

I guess it is because of the limitations in IPU (max 1024x1024), but why is it able to convert 1280x720 but not 1920x1080?

Regards!

[timharvey]

Jotes,

There are indeed some limitations when working with the IPU because of its need to have things byte aligned. The 1024x1024 resolution limit of the IPU is dealt with by the IPU driver by splitting the image conversion into stripes yet I've found that the Freescale IPU driver has an issue in scaling formulas that causes things that should be split evenly to not be so, resulting in alignment issues [1] - you may want to try that patch out.

If you can work with the fact that the GPU isn't as flexible in the formats it supports you can try imxg2dvideotransform as it does not have any byte alignment requirements. See Yocto/gstreamer/video – Gateworks  for more details.

Also, because gstreamer-imx is a community based project, you can discuss issues on its Github page: Issues · Freescale/gstreamer-imx · GitHub

Tim

[1] - ipu3: fix stripe calculation · Gateworks/linux-imx6@0d54f90 · GitHub

[jotes]

Tim,

Unfortunately, this patch did not help us. It seems that our problem is not related to any alignment issues. After displaying more details I get:

inw 2, onw 480, ilw 8, ilc 0, olw 1440, irw 8, irc 2, orw 1440, orc 480, difwr 18446744069533073408, lirr 34

mxc_ipu mxc_ipu: split mode output width overflow

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

If I understand it correctly, 1920 image is splitted into 480/1440 (why is so?). So it is still too large to be served by the IPU. Should I split it into more pieces?

I have also noticed, that the Y42B to I420 conversion works if the width is not bigger then 1360 pixels. Here is the pipeline that works:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1360, height=1080, framerate=30/1' ! imxipuvideosink sync=false

and the one that does not work:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1400, height=1080, framerate=30/1' ! imxipuvideosink sync=false

[timharvey]

Sorry, I don't have an answer for you. I'm not even sure what board, kernel, libs etc your using. I had to heavily instrument the Freescale IPUV3 driver in their downstream vendor kernel before I was able to find and fix the tiling issue that I ran into which I patched on the kernel we support for our boards. I would suggest posting a separate thread in IMX community starting off with all the details of what you are using (hardware/software) and see if you get any hits there.

Tim

[angeloaresi]

Hello,

Gstreamer-imx is a great set of plugins: it allows to create very complex pipelines (I could not create the same pipelines with gst-fsl-plugins
) and it is stable.

Now I have found an issue that I can't solve: there is a socket leakage using imxipuvideotransform and imxvpuenc at the same time.

I have posted this issue on github  imxvpuenc_h264 does't close all sockets on shut down operation · Issue #59 · Freescale/gstreamer-imx...

Has someone already solved this issue ? Can someone help to solve it ?

Thanks in advance,

Angelo

[vikaspatil]

Thanks for the useful information and posting it.

I am interested in imxeglvivsink on wayland/weston. Could you provide the example pipeline?

Are gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 latest and stable available version?  Is there any known limitation and bugs documented somewhere with wayland/weston?

Thanks,

Vikash

[timharvey]

Vikash,

We have not personally used the imxeglvivsink on wayland/weston yet.

We do feel that the gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 are very stable and this is what drove our decision to support that combination instead of fsl-gst-plugins (which does not have a community developed codebase and is full of incompatibilities with other gstreamer elements such as v4l2src).

Regards,

Tim

[vikaspatil]

Thanks for the information. So far I am able to run the following pipeline successfully till now. I am using

gstreamer1.0_1.4.5

gstreamer1.0-plugins-imx_0.11.1

Linux 3.14.28

Weston 1.8.0

#gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink

#gst-launch-1.0  imxv4l2videosrc ! imxg2dvideosink

#gst-launch-1.0 videotestsrc pattern=00 ! imxg2dvideosink framebuffer=/dev/fb0

However, it would be good if anyone can help me to run with "imxeglvivsink". I am getting following error

##gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxeglvivsink

Setting pipeline to PAUSED ...

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

Also not able to verify video encode and decode. Could you point me to some standard video stream for decode I can test with (may be h264)  and from where can I download?

##gst-launch-1.0 filesrc location=/home/root/song.mp4 ! h264parse ! imxvpudec ! imxipuvideotransform ! imxg2dvideosink

Setting pipeline to PAUSED ...

[INFO]  Product Info: i.MX6Q/D/S

Pipeline is PREROLLING ...

[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: GStreamer encountered a general stream error.

Additional debug info:

/data/work/vipatil/ELINA_081015/elina-distro/build-orinoco/tmp/work/cortexa9hf-vfp-neon-elina-linux-gnueabi/gstreamer1.0/1.4.5-r0/gstreamer-1.4.5/libs/gst/base/gstbaseparse.c(3264): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH264Parse:h264parse0:

streaming stopped, reason error

ERROR: pipeline doesn't want to preroll.

Setting pipeline to NULL ...

Freeing pipeline ...

Regards,

Vikash

[vikaspatil]

Hi,

I got it working now. Need to export following before using imxeglvivsink.

export XDG_RUNTIME_DIR=/var/run/root/1000

Do you know what are the advantages and disadvantages using gstreamer1.0-plugins-imx_0.11.1 over gst1.0-fsl-plugin_4.0.3?

Regards,

Vikash

[timharvey]

Vikash,

We discuss pros/cons of each in our page at http://trac.gateworks.com/wiki/Yocto/gstreamer

Basically the benefits of gstreamer-imx that we see are:

• Open Source Development model: The project is on github and is very active (gst-fsl-plugins is open-source but not community developed, there is no public git tree, and no ability to get fixes upstream)
• The main developer has been a GStreamer contributor for some time now and is very active in the GStreamer community
• The source is very well documented and easy to follow
• Things are done in a very standard GStreamer way (you can for example use the vpu encoder with a webcam via v4l2src, where-as this is not possible with gst-fsl-plugins)

Tim