topic Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins in i.MX Processors

How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

ilkodossev — Mon, 24 Feb 2014 17:29:33 GMT

Intro

Most of the hardware nowadays is capable of doing DMA transfers on large amount of data. Noticeably, Camera Capture Drivers (or any driver which delivers data to from any outside source to memory) can request DMA-capable buffer and put data into it.

While V4L2 architecture allows any driver to configure its DMA memory usage, it does not facilitate export of such information to Clients. Speccifically, GStreamer plugin implementation suffers terribly from the need to do incredible amount of byte-pushing in User Space, in order to pass buffer data from plugin to plugin. The incurred performance penalty is enormous!

The implementation I worked on involved TW6869 Camera capture chip which is DMA-capable; the captured RGB (or YUV) buffer had to be converted and compressed before further processing. Naturally, IPU mirroring capabilities and VPU compression capabilities determined the pipeline configuration. Also as natural pipeline builder came GST, together with its performance penalty mentioned above.

Hence, the goal becomes to connect "v4l2src" plugin with FSL plugins in a manner which allows recognition and use of DMA buffers, thus avoiding memcpy of data altogether.

Freescale's plugins do employ internal architecture of passing DMABLE info to each other, but in addition to this internal architecture, these plugins do use a more universal method to detect a DMA address.

These methods employed by Freescale plugins require patches in both "Video4Linux2" subsystem and in GStreamer's "libgstvideo4linux2.so" library.

Credit is due to Peng Zhou from Freescale for providing me with to-the-point information regarding Freescale's DMA implementation.

V4L2 patch

V4L2 uses "v4l2_buffer" structure (videodev2.h) to communicate with its clients. This structure, however, does not natively indicate to the Client whether the buffer just captured has DMA address.

Two members in the structure need attention:

__u32 flags;

__u32 reserved;

For "flags" a new flag hasto be defined:

#define V4L2_BUF_FLAG_DMABLE 0x40000000 /* buffer is DMA-mapped, 'reserved' contains DMA phy address */

And, accordingly, the Driver has to set the flag and fill in the "reserved" member in both its "vidioc_querybuf" and "vidioc_dqbuf" implementations.

This done, all V4L2 clients will get the DMA information from the Driver, for each of its buffers.

GST plugin patch

This patch is more difficult to implement. Not just acquisition of a DMA address from V4L2 driver is necessary, but then passing along that information to the next plugin in the pipeline is a must.

Plugins communicate with each other using "GstBuffer" structure, which like its V4L2 cousin has no provisions for passing DMA information.

Naturally, in a pipeline where FSL plugins are involved, a FSL-compatible method should be implemented. These are the macros used in FSL plugins:

#define IS_DMABLE_BUFFER(buffer)

#define DMABLE_BUFFER_PHY_ADDR(buffer)

The most FSL-compatible way to implement DMABLE would be to use "_gst_reserved" extension in the structure. If maximum compatibility with FSL is the goal, then a closer look at "gstbufmeta.c" and "gstbufmeta.h" files in "gst-fsl-plugins-3.0.7/libs/gstbufmeta/" folder shall be taken. Alternatively, GST_BUFFER_FLAG_LAST and GST_BUFFER_OFFSET may be used.

The sources of GStreamer's "libgstvideo4linux2.so" need patches in several places. First of all, right after calls to VIDIOC_QUERYBUF and VIDIOC_DQBUF the information from "v4l2_buffer" must be put in the "GstBuffer" instance. Next, the "need_copy" and "always_copy" flags have to be overridden whenever GST_BUFFER_FLAG_LAST is set. For reasons I cannot explain, the macro PROP_DEF_ALWAYS_COPY in the GST sources is set to TRUE and it is the default for "always_copy"! The final -- and absolutely important patch, is to prevent overwriting of GST_BUFFER_OFFSET memeber with sequential buffer number. Very! Bad! Things! will happen to the system if improper value is passed along as physical address! This last fix can be made conditional, upon the presence of GST_BUFFER_FLAG_LAST.

The outcome

Once these patches are in place, the pipeline is ready to roll, ansd roll it will really fast! My measurements by injection timing traces in FSL's "vpuenc" plugin demonstrated that the time for acquisition of GST input buffer was reduced on the average from 3+ milliseconds to 3-5 microseconds; and this improvement does not include the savings from avoiding the copy inside "libgstvideo4linux2.so" caused by "always_copy" flag!

Universally DMABLE?

The most important conclusion/question from the implementation above ought to be:
"Isn't it time for both V4L2 and GST to enable DMA buffer recognition and physical pointer passing?"

It is my firm belief, that such feature will result in great performance improvements for all kinds of video/audio streaming.

Ilko Dossev

Qualnetics, Inc.

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

varsmolta — Tue, 04 Mar 2014 17:10:28 GMT

Nice work! is there a patch for this? Any plans or rolling this into the freescale gstreamer and kernel repositories?

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

ilkodossev — Thu, 06 Mar 2014 19:48:27 GMT

The only patch really necessary is for GST v4l2src plugin.

It will be posted shortly, right after I am done with my current & very urgent project. :smileyhappy:

Regarding "videodev2.h" -- it also needs the definition, but for now a workaround like this will be sufficient in any source file:

#ifndef V4L2_BUF_FLAG_DMABLE

#define V4L2_BUF_FLAG_DMABLE 0x40000000

#endif

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

fabiendelafonta — Mon, 16 Jun 2014 10:13:17 GMT

Hi,

I try also to capture and encode 8 D1 videos from tw6869 at 25ips.

What's the driver you are using for the tw6869? Have you modify the driver for he send dma address?

I tried to modify the mfw_v4L2src plugin for work with the tw6869, but gstreamer crash in "mfw_gst_ipu_core_start_convert" (in mfw_gst_ipu_csc.c) because the dma address is NULL.

Have you made a patch for v4l2src plugin and the driver ?

Thanks advance.

Fabien

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

timharvey — Thu, 29 Jan 2015 16:24:40 GMT

Ilko,

I have users asking about using TW6869 with IMX6 as well. Will you share your patches with the community?

Regards,

Tim

Re: Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Thu, 29 Jan 2015 20:00:59 GMT

Hi Tim,

we are currently working on a prototype boards based on tw6869, driver uses vb2 api, but is also compatible with freescale gst-fsl-plugins through a small transparent hack.

In the Linux kernel driver:

+static int buffer_init(struct vb2_buffer *vb)

+ struct tw6869_vbuffer *vbuf = to_tw6869_vbuffer(vb);

+ vbuf->dma = vb2_dma_contig_plane_dma_addr(vb, 0);

+ INIT_LIST_HEAD(&vbuf->list);

+ { /* this hack to make gst-fsl-plugins happy */

+ u32 *cpu = vb2_plane_vaddr(vb, 0);

+ *cpu = vbuf->dma;

+ }

+ return 0;

in tvsrc plugin:

@@ -220,22 +221,19 @@ mfw_gst_tvsrc_start_capturing (MFWGstTVS

GST_BUFFER_SIZE (v4l_src->buffers[i]),

PROT_READ | PROT_WRITE, MAP_SHARED,

v4l_src->fd_v4l, GST_BUFFER_OFFSET (v4l_src->buffers[i]));

- memset (GST_BUFFER_DATA (v4l_src->buffers[i]), 0xFF,

- GST_BUFFER_SIZE (v4l_src->buffers[i]));

{

gint index;

GstBufferMeta *meta;

index = G_N_ELEMENTS (v4l_src->buffers[i]->_gst_reserved) - 1;

meta = gst_buffer_meta_new ();

- meta->physical_data = (gpointer) (buf->m.offset);

+ meta->physical_data =

+ (gpointer)(*(unsigned int *)(GST_BUFFER_DATA (v4l_src->buffers[i])));

v4l_src->buffers[i]->_gst_reserved[index] = meta;

}

+ memset (GST_BUFFER_DATA (v4l_src->buffers[i]), 0xFF,

+ GST_BUFFER_SIZE (v4l_src->buffers[i]));

the current version of the patch and test scripts in an attachment,

known Issues:

1 synchronization of audio and video in the encoding pipeline (probably a bug in the alsasrc plugin in gst 0.10)

2 sometimes, encoding pipeline is stalled for a considerable time (0.5 - 1 second)

Alexander

Re: Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

timharvey — Fri, 06 Feb 2015 14:50:20 GMT

Alexander,

This is great - thank you very much for sharing your work.

Have you ever looked into using the IMX6 hardware (IPU/VPU/GPU) to composite multiple streams into one so as to efficiently take the 4 (or more) streams into a single stream prior to sending to vpuenc? I have a user asking specifically for this.

Regards,

Tim

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Thu, 12 Feb 2015 21:57:26 GMT

Hi Tim,

I think this is not necessary because VPU "can encode or decode multiple video clips with multiple standards simultaneously", but some frames are discarded even in one simple pipeline, so i think real problem - slow color space conversion (UYVY -> I420 or NV12 that required for VPU) and/or bugs in software. in any case, we have no plans to deep dive in these issues.

Alexander.

Re: Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

timharvey — Fri, 13 Feb 2015 00:36:10 GMT

Alexander,

Right - the VPU 'can' encode multiple streams simultaneously but what I'm interested in is combining multiple sources into a single stream (think videobox) before encoding it to a single stream. I haven't had time to integrate your patches yet to see if an IMX6DL or IMX6Q has enough CPU to combine 4x of these sources via a non VPU-enhanced videobox and keep up with 30fps. I also want to take a look at the gstreamer-imx plugins for gstreamer1.x. I don't see any documentation o them but it looks like there are a lot more options available.

Regards,

Tim

Re: Re: Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Fri, 13 Feb 2015 10:30:48 GMT

Hi Tim,

I think blit that uses CPU (videobox + videomix) is very inefficient on embedded processors. gstreamer-imx have very good plugins that use gal2d, for example, on i.MX 6Solo and 6 streams I did not see the frame dropping with imxg2dvideosink, unfortunately gal2d has important limitations "RGB and YUV formats can be set in source surface, but only RGB format can be set in destination surface"

Alexander

Re: Re: Re: Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Mon, 09 Mar 2015 20:22:15 GMT

Hello,

in an attachment - a stable version of the driver.

Alexander

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

psidhu — Mon, 23 Mar 2015 22:34:05 GMT

Hi Alexander,

Thanks for sharing your work. I'm capturing with the card, but I was curious what you had to patch in imxv4l2src to get it to work with gstreamer1.0 (your latest scripts show that you're using 1.0).

- Pushpal

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Fri, 27 Mar 2015 18:04:28 GMT

Hi Pushpal Sidhu,

https://github.com/Freescale/gstreamer-imx

"This is a set of GStreamer 1.0 plugins for Freescale's i.MX platform, which make use of the i.MX multimedia capabilities"

Alexander

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

josésanhueza — Fri, 10 Apr 2015 19:06:31 GMT

Hi:

I've followed all the tips i've found here, but gstreamer refuses to show the TW6869 on the screen. The patch to the 3.10.53 works wonderfully and gstreamer shows me no errors or warnings, yet doesn display anything. Currently i am using a VAR-SOM-MX6Q board with poky yocto daisy. I think it may have something to do with DMA but i am running out of ideas about what to try.

Thanks in advance, any help will be greatly appreciated.

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

Sasamy — Mon, 13 Apr 2015 22:43:08 GMT

Hi José Sanhueza,

all patches tested, but we use Buildroot, so I'm not sure that they will work unchanged in Yocto.

https://youtu.be/YwQH2tWzH3g

https://youtu.be/nT8mX-_Eu9s

There is an easier way - we can change the offset to physical addresses as in the freescale release 3.14.28

http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/tree/drivers/media/platform/mxc/subdev/mx6s_capture.c?h=imx_3.14.28_1.0.0_ga#n1213

I'll try it later, because now busy with other projects. I am sorry for the delay in replying.

Alexander

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

psidhu — Thu, 16 Apr 2015 21:13:11 GMT

Hi José,

Few things - do you see a blue screen on the display when you run the pipeline? If you don't, can you verify that a simple pipeline using videotestsrc works? If you do, then maybe you're capturing from the wrong device (i.e. /dev/video4 vs. /dev/video7).

Other than that, run your gstreamer pipeline with a --gst-debug=3 to absolutely verify that you're not getting any errors. Finally make sure to use Alexanders gst1-plugins-imx-tw6869-sk.patch.zip patch to gstreamer-imx. This way, you can use imxv4l2src (from gstreamer-imx) and pipe it into the ipu via imxipuvideosink.

- Pushpal

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

psidhu — Thu, 16 Apr 2015 22:27:31 GMT

all patches tested, but we use Buildroot, so I'm not sure that they will work unchanged in Yocto.

Works just fine in Yocto :smileyhappy:

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

josésanhueza — Thu, 16 Apr 2015 23:50:31 GMT

Thanks for answering. No blue screen, tried with gstreamer 0.1 and 1.0 . Driver loads OK patched directly into the kernel, shows DMA X ON when i activate gstreamer or try to access the video ports in any way. V4l2-ctl shows the driver info and activate buffers but aparently nothing is going in them.

Wich linux kernel are you using?, and gstreamer?. I have tried with 3.10.53. I don't know what else to try, i think there could be something wrong with the device tree or the kernel config.

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

josésanhueza — Thu, 16 Apr 2015 23:52:17 GMT

Wich yocto. I use a poky daisy variation for the VAR-SOM-MX6 from Variscite.

Re: How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

josésanhueza — Fri, 17 Apr 2015 00:00:46 GMT

I see your videos and that is exactly what i want. I was hoping to later pass it on to opencv for processing but i am stuck with an aparently faulty DMA. Is it posible for the device tree to disrupt DMA? The board came with a kernel with pre-configured drivers to use with an OV5640 camera and IPU modules for MXC_v4l2_capture, but i removed them for compatibility (errors at boot time because i had no camera installed) I am using a VAR-SOM-MX6 board from Variscite.

Best regards.