topic Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines in i.MX Processors

i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

ryanerb — Wed, 23 Sep 2015 15:39:25 GMT

The Freescale i.MX6 has many video capabilities that are best accessed through GStreamer.

Gateworks, the leading supplier of Powerful ARM based Single Board Computer solutions using the Freescale i.MX6, has invested countless engineering hours researching and mastering GStreamer for the i.MX series of processors. Gateworks would like to share this GStreamer research with the rest of the i.MX community of developers! Visit the Gateworks Software Wiki GStreamer Page.

There are two main versions of GStreamer: 0.10 and 1.0. Version 1.0 is now the latest standard. gstreamer-1.0

The i.MX6 processor has hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these hardware blocks is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc.

The GStreamer app works with 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the imxvpudec is a VPU based decoder plugin.

This post is specifically about the plugins. There are different versions and sets of plugins available.

Gateworks has chosen to use the GStreamer-imx plugins for the following reasons:

Open Source Development model: The project is on github and is very active
The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
The source is very well documented and easy to follow
Things are done in a very standard GStreamer way

Plugin List

For a thorough description of each plugin, and why to use it, please visit the Gateworks Software Wiki GStreamer Page

Type	Plugin(s)	Element(s)	Comments
Audio Decoder	imxaudio	imxuniaudiodec	Uses i.MX uniaudio codecs for decoding
Audio Encoder	imxaudio	imxmp3audioenc	Uses i.MX for MP3 audio encoding
Device Sources	imxv4l2video	imxv4l2videosrc	Get camera source via v4l2
Video Decoder	imxvpu	imxvpudec	VPU Based decoder
Video Encoder	imxvpu	imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263	VPU Based encoders
Video Render (sink)	imxg2d; imxpxp; imxeglvivsink; imxipu	imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink	g2d¹, ipu¹, pxp², and egl (overlay) video sinks
Video Converter	imxg2d; imxpxp; imxipu	imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform	g2d, pxp, egl and ipu video filter/converter/scalars³
Video Compositing	imxg2d; imxipu	imxg2dcompositor, imxipucompositor	gpu/ipu accelerated compositing

1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.

2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.

3. Please see note 1 above.

Plugin Example Pipeline

For example, to encode a video from a camera on /dev/video2 into h.264 and save it to a file:

#Take camera input /dev/video2, encode it to h264 at a bitrate of 10mbit/s (CBR) and save to a file.
gst-launch-1.0 imxv4l2videosrc device=/dev/video2 ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

Many more pipeline examples are described and listed on the Gateworks Software Wiki GStreamer Pipelines page

Summary

Using GStreamer 1.0 with the GStreamer-imx plugins is a powerful way to access and apply the multimedia capabilities of the Freescale i.MX6 processors!

If there are other examples you would like to see, please add to the discussion!

arpan_chakravarty

KevinWong

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

vikaspatil — Wed, 07 Oct 2015 13:13:37 GMT

Thanks for the useful information and posting it.

I am interested in imxeglvivsink on wayland/weston. Could you provide the example pipeline?

Are gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 latest and stable available version? Is there any known limitation and bugs documented somewhere with wayland/weston?

Thanks,

Vikash

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Thu, 08 Oct 2015 19:40:18 GMT

Vikash,

We have not personally used the imxeglvivsink on wayland/weston yet.

We do feel that the gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 are very stable and this is what drove our decision to support that combination instead of fsl-gst-plugins (which does not have a community developed codebase and is full of incompatibilities with other gstreamer elements such as v4l2src).

Regards,

Tim

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

vikaspatil — Wed, 14 Oct 2015 13:28:19 GMT

Thanks for the information. So far I am able to run the following pipeline successfully till now. I am using

gstreamer1.0_1.4.5

gstreamer1.0-plugins-imx_0.11.1

Linux 3.14.28

Weston 1.8.0

#gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink

#gst-launch-1.0 imxv4l2videosrc ! imxg2dvideosink

#gst-launch-1.0 videotestsrc pattern=00 ! imxg2dvideosink framebuffer=/dev/fb0

However, it would be good if anyone can help me to run with "imxeglvivsink". I am getting following error

##gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxeglvivsink

Setting pipeline to PAUSED ...

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

Also not able to verify video encode and decode. Could you point me to some standard video stream for decode I can test with (may be h264) and from where can I download?

##gst-launch-1.0 filesrc location=/home/root/song.mp4 ! h264parse ! imxvpudec ! imxipuvideotransform ! imxg2dvideosink

Setting pipeline to PAUSED ...

[INFO] Product Info: i.MX6Q/D/S

Pipeline is PREROLLING ...

[INFO] bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: GStreamer encountered a general stream error.

Additional debug info:

/data/work/vipatil/ELINA_081015/elina-distro/build-orinoco/tmp/work/cortexa9hf-vfp-neon-elina-linux-gnueabi/gstreamer1.0/1.4.5-r0/gstreamer-1.4.5/libs/gst/base/gstbaseparse.c(3264): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH264Parse:h264parse0:

streaming stopped, reason error

ERROR: pipeline doesn't want to preroll.

Setting pipeline to NULL ...

Freeing pipeline ...

Regards,

Vikash

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

vikaspatil — Fri, 16 Oct 2015 07:53:45 GMT

Hi,

I got it working now. Need to export following before using imxeglvivsink.

export XDG_RUNTIME_DIR=/var/run/root/1000

Do you know what are the advantages and disadvantages using gstreamer1.0-plugins-imx_0.11.1 over gst1.0-fsl-plugin_4.0.3?

Regards,

Vikash

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

jotes — Fri, 16 Oct 2015 08:25:25 GMT

Hello!

Is it possible to make colorspace conversion for format 1920x1080 using imxipuvideotransform? I get the following error:

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

I guess it is because of the limitations in IPU (max 1024x1024), but why is it able to convert 1280x720 but not 1920x1080?

Regards!

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Thu, 22 Oct 2015 16:59:23 GMT

Vikash,

We discuss pros/cons of each in our page at http://trac.gateworks.com/wiki/Yocto/gstreamer

Basically the benefits of gstreamer-imx that we see are:

Open Source Development model: The project is on github and is very active (gst-fsl-plugins is open-source but not community developed, there is no public git tree, and no ability to get fixes upstream)
The main developer has been a GStreamer contributor for some time now and is very active in the GStreamer community
The source is very well documented and easy to follow
Things are done in a very standard GStreamer way (you can for example use the vpu encoder with a webcam via v4l2src, where-as this is not possible with gst-fsl-plugins)

Tim

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Thu, 22 Oct 2015 19:13:43 GMT

Jotes,

There are indeed some limitations when working with the IPU because of its need to have things byte aligned. The 1024x1024 resolution limit of the IPU is dealt with by the IPU driver by splitting the image conversion into stripes yet I've found that the Freescale IPU driver has an issue in scaling formulas that causes things that should be split evenly to not be so, resulting in alignment issues [1] - you may want to try that patch out.

If you can work with the fact that the GPU isn't as flexible in the formats it supports you can try imxg2dvideotransform as it does not have any byte alignment requirements. See Yocto/gstreamer/video – Gateworks for more details.

Also, because gstreamer-imx is a community based project, you can discuss issues on its Github page: Issues · Freescale/gstreamer-imx · GitHub

Tim

[1] - ipu3: fix stripe calculation · Gateworks/linux-imx6@0d54f90 · GitHub

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

angeloaresi — Fri, 23 Oct 2015 06:59:22 GMT

Hello,

Gstreamer-imx is a great set of plugins: it allows to create very complex pipelines (I could not create the same pipelines with gst-fsl-plugins
) and it is stable.

Now I have found an issue that I can't solve: there is a socket leakage using imxipuvideotransform and imxvpuenc at the same time.

I have posted this issue on github imxvpuenc_h264 does't close all sockets on shut down operation · Issue #59 · Freescale/gstreamer-imx · GitHub

Has someone already solved this issue ? Can someone help to solve it ?

Thanks in advance,

Angelo

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

jotes — Tue, 27 Oct 2015 14:41:57 GMT

Tim,

Unfortunately, this patch did not help us. It seems that our problem is not related to any alignment issues. After displaying more details I get:

inw 2, onw 480, ilw 8, ilc 0, olw 1440, irw 8, irc 2, orw 1440, orc 480, difwr 18446744069533073408, lirr 34
mxc_ipu mxc_ipu: split mode output width overflow
mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

If I understand it correctly, 1920 image is splitted into 480/1440 (why is so?). So it is still too large to be served by the IPU. Should I split it into more pieces?

I have also noticed, that the Y42B to I420 conversion works if the width is not bigger then 1360 pixels. Here is the pipeline that works:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1360, height=1080, framerate=30/1' ! imxipuvideosink sync=false

and the one that does not work:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1400, height=1080, framerate=30/1' ! imxipuvideosink sync=false

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

wjmazepas — Tue, 17 Nov 2015 19:17:07 GMT

Does anyone had the same problem than me?
I installed as say in the git page of the gstreamer-imx, but when i run gst-inspect-1.0 | grep imx, it can only find the imxaudio, the others packages he say that he cant find, but everything is there, i checked a multiple times and it keeps showing the same problem

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

wjmazepas — Tue, 17 Nov 2015 20:40:49 GMT

Got it. Had to do it like this guy did it
Tiny Devices: I.MX6 - gstreamer-imx

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Tue, 17 Nov 2015 20:59:52 GMT

Welmo,

Glad you got it figured out. We do have some documentation on how to build gstreamer-imx with IPU/VPU/GPU support for Ubuntu ( ventana/ubuntu – Gateworks ) and Debian ( ventana/debian – Gateworks ).

Tim

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Tue, 17 Nov 2015 21:02:05 GMT

Sorry, I don't have an answer for you. I'm not even sure what board, kernel, libs etc your using. I had to heavily instrument the Freescale IPUV3 driver in their downstream vendor kernel before I was able to find and fix the tiling issue that I ran into which I patched on the kernel we support for our boards. I would suggest posting a separate thread in IMX community starting off with all the details of what you are using (hardware/software) and see if you get any hits there.

Tim

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

wjmazepas — Wed, 18 Nov 2015 00:28:07 GMT

Thanks Tim. Sorry to trouble you again, but i am facing a new problem with the imx package. I posted this thread: Streaming webcam video with the gstreamer-imx
If you could help me, i would be verry happy mate. I'm trying a lot of crazy stuff here but nothing works

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

tutran — Tue, 15 Mar 2016 11:04:28 GMT

Dear Tim,

I have tried your pipeline with imxv4l2src plugin as below:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

Then I viewed the file and see that the frame size is 640x480.

I want to capture with bigger frame size, for example 1280x800

I modified the pipeline like this:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw, format=(string)I420, width=(int)1280, height=(int)800, interlaced=(boolean)false, framerate=(fraction)10/1' ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

But I got the error message:

====== IMXV4L2SRC: 4.0.3 build on Mar 4 2016 15:53:15. ======

WARNING: erroneous pipeline: could not link imxv4l2src0 to imxvpuh264enc0

Could you help me with this?

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Tue, 15 Mar 2016 14:47:58 GMT

Tu Tran,

You don't mention what board you have, but I'm going to guess its a

Gateworks GW5400 with digital HDMI video in on /dev/video0 and Analog CVBS

in on /dev/video1. In this case if you inspect capabilities of /dev/video0

via 'v4l2-ctl --deviceo=/dev/video1 --all' you will see that the only video

capture size available is 720x480 (NTSC video) (You say 640x480 but I

assume you mean 720x480).

Therefore you need to scale the video at some point in your pipeline and in

your case you would want to do that before it enters the encoder element

because it appears your intention is to create a 1280x800 H264 encoded

video. Note that you would likely be way better off encoding the image at

its native resolution and up-scaling it while playing it back as otherwise

your just using up bits on up-scaled video.

You are on the right track by using a capsfilter however your pipeline is

telling the source plugin to capture at 1280x800 which it quite simply

can't do. You need to place something in between the source element and the

encoder that is capable of image scaling and in the case of the IMX6 you

would want to use hardware-scaling provided by either the IMX6 ipu

(imxipuvideotransform), the IMX6 GPU (imxg2dvideotransform), or the IMX6

PXP (imxpxpvideotransform). Your best bet is likely imxipuvideotransform as

it supports the video format needed by the VPU encoders. Note also that if

you specify any parameter that can't be met you will get the infamous

'could not link' elements error. It's best to provide only the parameters

you absolutely need and let Gstreamer try to figure out the rest and if you

run into issues start removing features from your capsfilter until you get

a successful pipeline.

If you remove the 'interlaced=false' and 'framerate=10/1' which the

imxv4l2videosrc element doesn't support you can capture successfully:

gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! videorate !

imxipuvideotransform ! 'video/x-raw, format=(string)I420, width=(int)1280,

height=(int)800' ! imxvpuenc_h264 bitrate=10000 ! filesink

location=/tmp/file.mp4

Although the 'imxipuvideotransform' can't de-interlace via a caps-filter a

'gst-inspect-1.0 imxipuvideotransform' tells you that it indeed can

de-interlace for you by setting the 'deinterlace' property on the element.

Thus you can get closer to your original capsfilter via:

gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! videorate !

imxipuvideotransform deinterlace=true ! 'video/x-raw, format=(string)I420,

width=(int)1280, height=(int)800' ! imxvpuenc_h264 bitrate=10000 ! filesink

location=/tmp/file.mp4

Because imxv4l2videosrc does not appear to allow framerate adjustments, you

can do this via the standard gstreamer 'videorate' element but do note this

may cause some timestamp/playback issues with the h264 encoder:

gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! imxipuvideotransform !

'video/x-raw, format=(string)I420, width=(int)1280, height=(int)800' !

videorate ! 'video/x-raw, framerate=10/1' ! imxvpuenc_h264 bitrate=10000 !

filesink location=/tmp/file.mp4

For more information on the gstreamer-imx plugins you can refer to:

- http://trac.gateworks.com/wiki/Yocto/gstreamer/video - examples and

descriptions

- https://github.com/Freescale/gstreamer-imx sourcecode

Regards,

Tim

Tim Harvey - Principal Software Engineer

Gateworks Corporation - http://www.gateworks.com/

3026 S. Higuera St. San Luis Obispo CA 93401

805-781-2000

On Tue, Mar 15, 2016 at 4:05 AM, tutran <admin@community.freescale.com>

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

tutran — Wed, 16 Mar 2016 03:36:17 GMT

Dear Tim,

Oh, I am sorry for lacking of describing more details.

I am using OV5640 camera sensor MIPI with iMX6sabreqsd board.

From imxv4l2src, I saw that it can support higher resolution up to 1920x1080p 15fps.

I also tried with a simpler pipeline:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, framerate=(fraction)15/1' ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

But the same error happened. I can not link the pipeline, the debug log said that the "caps is incompatible".

In my environment, there is no plugin imxv4l2videosrc, I used the native one imxv4l2src instead.

When I used gst-inspect-1.0 for imxv4l2src, I saw a list of capabilities:

Capabilities:
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 640
                 height: 480
              framerate: { 30/1, 15/1 }
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 320
                 height: 240
              framerate: { 30/1, 15/1 }
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 720
                 height: 480
              framerate: { 30/1, 15/1 }
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 720
                 height: 576
              framerate: { 30/1, 15/1 }
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 1280
                 height: 720
              framerate: { 30/1, 15/1 }
      video/x-raw
                 format: { I420, NV12, YUY2, UYVY }
                  width: 1920
                 height: 1080
              framerate: 15/1

And also for the imxvpuenc_h264 plugins.

As my understanding, I can create a capsfilter for choosing the format linking between imxv4l2src and imxvpuenc_h264. Is it correct? Or there is any other constraint when working with a specific camera device.

Please help me to make it clear.

Thanks a lot for your support Tim

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

timharvey — Wed, 16 Mar 2016 13:31:49 GMT

Tu Tran,

You must be using a different/patched version of gstreamer-imx due to the

naming difference of the source plugin but still your issue is that not

everything you are specifying in your capsfilter can be accomplished by the

source. As I mentioned I needed to drop the framerate from the capsfilter

and instead alter the framerate with the videorate plugin.

Try:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw,

format=(string)I420, width=(int)1920, height=(int)1080' ! imxvpuenc_h264

bitrate=10000 ! filesink location=/tmp/file.mp4

If this doesn't work then I would suggest its something to do with the

ov5640 slave driver and/or your version of gstreamer-imx.

The pipelines I suggested work on the Gateworks Ventana boards with our

Yocto 1.8 BSP. I don't have your hardware and can't test your situation.

Regards,

Tim

On Tue, Mar 15, 2016 at 8:37 PM, tutran <admin@community.freescale.com>

Re: i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

tutran — Fri, 18 Mar 2016 04:05:26 GMT

Dear Tim,

Thanks for your support.

With the new pipeline suggestion, I can do streaming from the camera now.