i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

cancel
Showing results for 
Search instead for 
Did you mean: 

i.MX6 GStreamer-imx Plugins - Tutorial & Example Pipelines

74,244 Views
ryanerb
Contributor III

The Freescale i.MX6 has many video capabilities that are best accessed through GStreamer.

Gateworks, the leading supplier of Powerful ARM based Single Board Computer solutions using the Freescale i.MX6, has invested countless engineering hours researching and mastering GStreamer for the i.MX series of processors. Gateworks would like to share this GStreamer research with the rest of the i.MX community of developers! Visit the Gateworks Software Wiki GStreamer Page.

There are two main versions of GStreamer: 0.10 and 1.0. Version 1.0 is now the latest standard. gstreamer-1.0​

The i.MX6 processor has hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these hardware blocks is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc.

The GStreamer app works with 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the imxvpudec is a VPU based decoder plugin.

This post is specifically about the plugins. There are different versions and sets of plugins available.

Gateworks has chosen to use the GStreamer-imx plugins for the following reasons:

  • Open Source Development model: The project is on github and is very active
  • The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
  • The source is very well documented and easy to follow
  • Things are done in a very standard GStreamer way

Plugin List

For a thorough description of each plugin, and why to use it, please visit the Gateworks Software Wiki GStreamer Page

TypePlugin(s)Element(s)Comments
Audio DecoderimxaudioimxuniaudiodecUses i.MX uniaudio codecs for decoding
Audio Encoderimxaudioimxmp3audioencUses i.MX for MP3 audio encoding
Device Sourcesimxv4l2videoimxv4l2videosrcGet camera source via v4l2
Video DecoderimxvpuimxvpudecVPU Based decoder
Video Encoderimxvpuimxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263VPU Based encoders
Video Render (sink)imxg2d; imxpxp; imxeglvivsink; imxipuimxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosinkg2d1, ipu1, pxp2, and egl (overlay) video sinks
Video Converterimxg2d; imxpxp; imxipuimxg2dvideotransform; imxpxpvideotransform; imxipuvideotransformg2d, pxp, egl and ipu video filter/converter/scalars3
Video Compositingimxg2d; imxipuimxg2dcompositor, imxipucompositorgpu/ipu accelerated compositing

1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.

2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.

3. Please see note 1 above.

Plugin Example Pipeline

For example, to encode a video from a camera on /dev/video2 into h.264 and save it to a file:

#Take camera input /dev/video2, encode it to h264 at a bitrate of 10mbit/s (CBR) and save to a file.
gst-launch-1.0 imxv4l2videosrc device=/dev/video2 ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

Many more pipeline examples are described and listed on the Gateworks Software Wiki GStreamer Pipelines page

Summary

Using GStreamer 1.0 with the GStreamer-imx plugins is a powerful way to access and apply the multimedia capabilities of the Freescale i.MX6 processors!

If there are other examples you would like to see, please add to the discussion!

arpan_chakravarty

KevinWong

39 Replies

879 Views
timharvey
Contributor IV

Tu Tran,

You must be using a different/patched version of gstreamer-imx due to the

naming difference of the source plugin but still your issue is that not

everything you are specifying in your capsfilter can be accomplished by the

source. As I mentioned I needed to drop the framerate from the capsfilter

and instead alter the framerate with the videorate plugin.

Try:

gst-launch-1.0 imxv4l2src device=/dev/video1 ! 'video/x-raw,

format=(string)I420, width=(int)1920, height=(int)1080' ! imxvpuenc_h264

bitrate=10000 ! filesink location=/tmp/file.mp4

If this doesn't work then I would suggest its something to do with the

ov5640 slave driver and/or your version of gstreamer-imx.

The pipelines I suggested work on the Gateworks Ventana boards with our

Yocto 1.8 BSP. I don't have your hardware and can't test your situation.

Regards,

Tim

On Tue, Mar 15, 2016 at 8:37 PM, tutran <admin@community.freescale.com>

830 Views
tutran
Contributor I

Dear Tim,

I got one more problem with the imxvpuenc_h264 plugin that need your support.

When I list all the encoder plugins for iMX I see there two

- vpuenc

- imxvpuenc_h264

In my practice my these two plugins on iMX6DL board (kernel 3.14.28), gstreamer version 1.4.1. I see some stranges

1/. I tried to captured the image using camera and save them to MP4 file

- With vpuenc:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! vpuenc gop-size=30 bitrate=512 ! filesink location=/home/root/test_vpu.mp4

- With imxvpuenc_h264:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! imxvpuenc_h264 gop-size=30 bitrate=512 ! filesink location=/home/root/test_encode.mp4

In two cases, VLC player cannot render the mp4 files and only output from imxvpuenc_h264 can be played using default video app of Ubuntu, or Mplayer. The one output from vpuenc cannot be played by any decoder

2/. For streaming using RTP/UDP protocol

- With vpuenc:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! vpuenc gop-size=30 bitrate=512 ! h264parse ! rtph264pay name=pay0 pt=96 timestamp-offset=0 ! udpsink host=192.168.0.100 port=5000&

- With imxvpuenc_h264:

gst-launch-1.0 -v imxv4l2src device=/dev/video1 ! video/x-raw, format=I420,width=1920, height=1080 ! imxvpuenc_h264 gop-size=30 bitrate=512 ! h264parse ! rtph264pay name=pay0 pt=96 timestamp-offset=0 ! udpsink host=192.168.0.100 port=5000 &

I use VLC client instead of gstreamer, and with imxvpuenc_h264, I cannot display the video data streaming from the pipeline although wireshark can catch the UDP packets. But it can work perfectly with the pipeline using vpuenc to encode data.

Please give me any hint or statement for the phenomenon. The issues really made me confused about the root cause.

0 Kudos

812 Views
ryanerb
Contributor III

Regarding VLC, you likely will need the mp4mux and possibly the -e for termination:

gst-launch-1.0 -e videotestsrc pattern=0 ! imxvpuenc_h264 bitrate=10000 ! h264parse ! mp4mux ! filesink location=/tmp/file3.mp4

0 Kudos

812 Views
tutran
Contributor I

Hi Ryan,

Thanks for your suggestion. It still not work with me. VLC cannot play the video either.

0 Kudos

879 Views
wjmazepas
Contributor II

Does anyone had the same problem than me?
I installed as say in the git page of the gstreamer-imx, but when i run gst-inspect-1.0 | grep imx, it can only find the imxaudio, the others packages he say that he cant find, but everything is there, i checked a multiple times and it keeps showing the same problem

879 Views
wjmazepas
Contributor II

Got it. Had to do it like this guy did it
Tiny Devices: I.MX6 - gstreamer-imx

0 Kudos

876 Views
ranshalit
Senior Contributor I

Hi, 

We are using yocto morti.

Must we add this plugin using by manually building from git ?

Can't they be added with yocto ?

Regards,

Ran

0 Kudos

879 Views
timharvey
Contributor IV

Welmo,

Glad you got it figured out. We do have some documentation on how to build gstreamer-imx with IPU/VPU/GPU support for Ubuntu ( ventana/ubuntu – Gateworks​ ) and Debian ( ventana/debian – Gateworks ).

Tim

879 Views
wjmazepas
Contributor II

Thanks Tim. Sorry to trouble you again, but i am facing a new problem with the imx package. I posted this thread: Streaming webcam video with the gstreamer-imx
If you could help me, i would be verry happy mate. I'm trying a lot of crazy stuff here but nothing works

0 Kudos

879 Views
jotes
Contributor IV

Hello!

Is it possible to make colorspace conversion for format 1920x1080 using imxipuvideotransform? I get the following error:

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

I guess it is because of the limitations in IPU (max 1024x1024), but why is it able to convert 1280x720 but not 1920x1080?

Regards!

879 Views
timharvey
Contributor IV

Jotes,

There are indeed some limitations when working with the IPU because of its need to have things byte aligned. The 1024x1024 resolution limit of the IPU is dealt with by the IPU driver by splitting the image conversion into stripes yet I've found that the Freescale IPU driver has an issue in scaling formulas that causes things that should be split evenly to not be so, resulting in alignment issues [1] - you may want to try that patch out.

If you can work with the fact that the GPU isn't as flexible in the formats it supports you can try imxg2dvideotransform as it does not have any byte alignment requirements. See Yocto/gstreamer/video – Gateworks  for more details.

Also, because gstreamer-imx is a community based project, you can discuss issues on its Github page: Issues · Freescale/gstreamer-imx · GitHub

Tim

[1] - ipu3: fix stripe calculation · Gateworks/linux-imx6@0d54f90 · GitHub

879 Views
jotes
Contributor IV

Tim,

Unfortunately, this patch did not help us. It seems that our problem is not related to any alignment issues. After displaying more details I get:

inw 2, onw 480, ilw 8, ilc 0, olw 1440, irw 8, irc 2, orw 1440, orc 480, difwr 18446744069533073408, lirr 34

mxc_ipu mxc_ipu: split mode output width overflow

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

If I understand it correctly, 1920 image is splitted into 480/1440 (why is so?). So it is still too large to be served by the IPU. Should I split it into more pieces?

I have also noticed, that the Y42B to I420 conversion works if the width is not bigger then 1360 pixels. Here is the pipeline that works:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1360, height=1080, framerate=30/1' ! imxipuvideosink sync=false

and the one that does not work:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw,format=(string)Y42B, width=1920, height=1080, framerate=30/1' ! imxipuvideotransform ! 'video/x-raw,format=(string)I420, width=1400, height=1080, framerate=30/1' ! imxipuvideosink sync=false

879 Views
timharvey
Contributor IV

Sorry, I don't have an answer for you. I'm not even sure what board, kernel, libs etc your using. I had to heavily instrument the Freescale IPUV3 driver in their downstream vendor kernel before I was able to find and fix the tiling issue that I ran into which I patched on the kernel we support for our boards. I would suggest posting a separate thread in IMX community starting off with all the details of what you are using (hardware/software) and see if you get any hits there.

Tim

0 Kudos

879 Views
angeloaresi
Contributor I

Hello,

Gstreamer-imx is a great set of plugins: it allows to create very complex pipelines (I could not create the same pipelines with gst-fsl-plugins
) and it is stable.

Now I have found an issue that I can't solve: there is a socket leakage using imxipuvideotransform and imxvpuenc at the same time.

I have posted this issue on github  imxvpuenc_h264 does't close all sockets on shut down operation · Issue #59 · Freescale/gstreamer-imx...

Has someone already solved this issue ? Can someone help to solve it ?

Thanks in advance,

Angelo

0 Kudos

879 Views
vikaspatil
Senior Contributor I

Thanks for the useful information and posting it.

I am interested in imxeglvivsink on wayland/weston. Could you provide the example pipeline?

Are gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 latest and stable available version?  Is there any known limitation and bugs documented somewhere with wayland/weston?

Thanks,

Vikash

0 Kudos

879 Views
timharvey
Contributor IV

Vikash,

We have not personally used the imxeglvivsink on wayland/weston yet.

We do feel that the gstreamer1.0-plugins-imx_0.11.1 and gstreamer1.0_1.4.5 are very stable and this is what drove our decision to support that combination instead of fsl-gst-plugins (which does not have a community developed codebase and is full of incompatibilities with other gstreamer elements such as v4l2src).

Regards,

Tim

0 Kudos

879 Views
vikaspatil
Senior Contributor I

Thanks for the information. So far I am able to run the following pipeline successfully till now. I am using

gstreamer1.0_1.4.5

gstreamer1.0-plugins-imx_0.11.1

Linux 3.14.28

Weston 1.8.0

#gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink

#gst-launch-1.0  imxv4l2videosrc ! imxg2dvideosink

#gst-launch-1.0 videotestsrc pattern=00 ! imxg2dvideosink framebuffer=/dev/fb0

However, it would be good if anyone can help me to run with "imxeglvivsink". I am getting following error

##gst-launch-1.0 -v imxv4l2videosrc device=/dev/video0 ! imxeglvivsink

Setting pipeline to PAUSED ...

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

Also not able to verify video encode and decode. Could you point me to some standard video stream for decode I can test with (may be h264)  and from where can I download?

##gst-launch-1.0 filesrc location=/home/root/song.mp4 ! h264parse ! imxvpudec ! imxipuvideotransform ! imxg2dvideosink

Setting pipeline to PAUSED ...

[INFO]  Product Info: i.MX6Q/D/S

Pipeline is PREROLLING ...

[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: GStreamer encountered a general stream error.

Additional debug info:

/data/work/vipatil/ELINA_081015/elina-distro/build-orinoco/tmp/work/cortexa9hf-vfp-neon-elina-linux-gnueabi/gstreamer1.0/1.4.5-r0/gstreamer-1.4.5/libs/gst/base/gstbaseparse.c(3264): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH264Parse:h264parse0:

streaming stopped, reason error

ERROR: pipeline doesn't want to preroll.

Setting pipeline to NULL ...

Freeing pipeline ...

Regards,

Vikash

0 Kudos

879 Views
vikaspatil
Senior Contributor I

Hi,

I got it working now. Need to export following before using imxeglvivsink.

export XDG_RUNTIME_DIR=/var/run/root/1000

Do you know what are the advantages and disadvantages using gstreamer1.0-plugins-imx_0.11.1 over gst1.0-fsl-plugin_4.0.3?

Regards,

Vikash

0 Kudos

879 Views
timharvey
Contributor IV

Vikash,

We discuss pros/cons of each in our page at http://trac.gateworks.com/wiki/Yocto/gstreamer

Basically the benefits of gstreamer-imx that we see are:

  • Open Source Development model: The project is on github and is very active (gst-fsl-plugins is open-source but not community developed, there is no public git tree, and no ability to get fixes upstream)
  • The main developer has been a GStreamer contributor for some time now and is very active in the GStreamer community
  • The source is very well documented and easy to follow
  • Things are done in a very standard GStreamer way (you can for example use the vpu encoder with a webcam via v4l2src, where-as this is not possible with gst-fsl-plugins)

Tim

0 Kudos