How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to pass DMA buffers from a Driver thru GStreamer 'v4l2src' plugin to Freescale IPU/VPU plugins

29,787 Views
ilkodossev
Contributor III

Intro

Most of the hardware nowadays is capable of doing DMA transfers on large amount of data. Noticeably, Camera Capture Drivers (or any driver which delivers data to from any outside source to memory) can request DMA-capable buffer and put data into it.

While V4L2 architecture allows any driver to configure its DMA memory usage, it does not facilitate export of such information to Clients. Speccifically, GStreamer plugin implementation suffers terribly from the need to do incredible amount of byte-pushing in User Space, in order to pass buffer data from plugin to plugin. The incurred performance penalty is enormous!

The implementation I worked on involved TW6869 Camera capture chip which is DMA-capable; the captured RGB (or YUV) buffer had to be converted and compressed before further processing. Naturally, IPU mirroring capabilities and VPU compression capabilities determined the pipeline configuration. Also as natural pipeline builder came GST, together with its performance penalty mentioned above.

Hence, the goal becomes to connect "v4l2src" plugin with FSL plugins in a manner which allows recognition and use of DMA buffers, thus avoiding memcpy of data altogether.

Freescale's plugins do employ internal architecture of passing DMABLE info to each other, but in addition to this internal architecture, these plugins do use a more universal method to detect a DMA address.

These methods employed by Freescale plugins require patches in both "Video4Linux2" subsystem and in GStreamer's "libgstvideo4linux2.so" library.

Credit is due to Peng Zhou from Freescale for providing me with to-the-point information regarding Freescale's DMA implementation.

V4L2 patch

V4L2 uses "v4l2_buffer" structure (videodev2.h) to communicate with its clients. This structure, however, does not natively indicate to the Client whether the buffer just captured has DMA address.

Two members in the structure need attention:

     __u32     flags;

     __u32     reserved;

For "flags" a new flag hasto be defined:

  #define V4L2_BUF_FLAG_DMABLE    0x40000000  /* buffer is DMA-mapped, 'reserved' contains DMA phy address */

And, accordingly, the Driver has to set the flag and fill in the "reserved" member in both its "vidioc_querybuf" and "vidioc_dqbuf" implementations.

This done, all V4L2 clients will get the DMA information from the Driver, for each of its buffers.

GST plugin patch

This patch is more difficult to implement. Not just acquisition of a DMA address from V4L2 driver is necessary, but then passing along that information to the next plugin in the pipeline is a must.

Plugins communicate with each other using "GstBuffer" structure, which like its V4L2 cousin has no provisions for passing DMA information.

Naturally, in a pipeline where FSL plugins are involved, a FSL-compatible method should be implemented. These are the macros used in FSL plugins:

  #define IS_DMABLE_BUFFER(buffer)

  #define DMABLE_BUFFER_PHY_ADDR(buffer)

The most FSL-compatible way to implement DMABLE would be to use "_gst_reserved" extension in the structure. If maximum compatibility with FSL is the goal, then a closer look at "gstbufmeta.c" and "gstbufmeta.h" files in "gst-fsl-plugins-3.0.7/libs/gstbufmeta/" folder shall be taken. Alternatively, GST_BUFFER_FLAG_LAST and GST_BUFFER_OFFSET may be used.

The sources of GStreamer's "libgstvideo4linux2.so" need patches in several places. First of all, right after calls to VIDIOC_QUERYBUF and VIDIOC_DQBUF the information from "v4l2_buffer" must be put in the "GstBuffer" instance. Next, the "need_copy" and "always_copy" flags have to be overridden whenever GST_BUFFER_FLAG_LAST is set. For reasons I cannot explain, the macro PROP_DEF_ALWAYS_COPY in the GST sources is set to TRUE and it is the default for "always_copy"! The final -- and absolutely important patch, is to prevent overwriting of GST_BUFFER_OFFSET memeber with sequential buffer number. Very! Bad! Things! will happen to the system if improper value is passed along as physical address! This last fix can be made conditional, upon the presence of GST_BUFFER_FLAG_LAST.

The outcome

Once these patches are in place, the pipeline is ready to roll, ansd roll it will really fast! My measurements by injection timing traces in FSL's "vpuenc" plugin demonstrated that the time for acquisition of GST input buffer was reduced on the average from 3+ milliseconds to 3-5 microseconds; and this improvement does not include the savings from avoiding the copy inside "libgstvideo4linux2.so" caused by "always_copy" flag!

Universally DMABLE?

The most important conclusion/question from the implementation above ought to be:
"Isn't it time for both V4L2 and GST to enable DMA buffer recognition and physical pointer passing?"

It is my firm belief, that such feature will result in great performance improvements for all kinds of video/audio streaming.

Ilko Dossev

Qualnetics, Inc.

Labels (2)
43 Replies

5,143 Views
timharvey
Contributor IV

Alexander,

I'm looking at your v0.5.1 driver on https://github.com/sasamy/tw6869​ and am trying to get it to behave as your prior version did with respect to signal loss. Your original driver would produce 'blue' frames when using a channel with no signal or when signal was lost yet I can't seem to get your current driver to do this.

I've tried setting R8_MISC_CONTROL1 to 0xE7 as your prior driver did as well as making the ISR ignore signal loss errors by masking out the low byte of FIFO_STATUS. The tw6869 documentation I have from Intersil's website states that its MISC_CONTROL2 that has this feature yet setting the bits there doesn't seem to do the trick either.

Any ideas?

How would you say your driver compares to that which was recently accepted upstream and will be present in Linux 4.8? It seems to me that the big difference is that you have created a userptr mechanism that can allow 0-copy buffers to the IMX IPU/VPU/GPU driver code but I'm not sure there is anything else you have that won't be upsteam soon, correct?

Regards,

Tim

0 Kudos
Reply

4,472 Views
timharvey
Contributor IV

Alexander,

oops - I borked up a kernel cmdline option that was causing my issues. Moving past that, with your current driver I can re-enable the 'blue' screen on no signal with the following patch:

@@ -152,7 +168,7 @@ static irqreturn_t tw6869_irq(int irq, void *dev_id)

                pars = tw_read(dev, R32_VIDEO_PARSER_STATUS);

                cmd = tw_read(dev, R32_DMA_CMD);

-              errs = (ints >> 24 | errs >> 24 | errs >> 16 | errs) & TW_VID;

+              errs = (ints >> 24 | errs >> 24 | errs >> 16) & TW_VID;

                ints = (ints | errs) & cmd;

                for_each_set_bit(id, &ints, TW_ID_MAX) {

@@ -167,6 +183,8 @@ static irqreturn_t tw6869_irq(int irq, void *dev_id)

                        } else {

                                spin_lock(&dev->rlock);

                                tw6869_dma_reset(dma);

+                              if (tw_dma_active(dma))

+                                      tw_dma_enable(dma);

                                spin_unlock(&dev->rlock);

                        }

                }

@@ -208,6 +226,10 @@ static void tw6869_reset(struct tw6869_dev *dev)

        tw_write(dev, R8_MISC_CONTROL1(0x0), 0x56);

        tw_write(dev, R8_MISC_CONTROL1(0x4), 0x56);

+      /* Show blue background if no signal */

+      tw_write(dev, R8_MISC_CONTROL2(0x0), 0xE7);

+      tw_write(dev, R8_MISC_CONTROL2(0x4), 0xE7);

+

        /* Audio DMA 4096 bytes, sampling frequency reference 48 kHz */

        tw_write(dev, R32_AUDIO_CONTROL1, 0x80000001 | (0x0A2C << 5));

        tw_write(dev, R32_AUDIO_CONTROL2, 0x0A2C2AAA);

It's not perfect however as occasionally on signal loss and re-detect (ie remove and re-attach your video source) the captured frames start showing what looks like interlacing effects (but on scenes with no motion) almost like internally its interlacing the fields backwards. If you stop and re-start the stream all is well.

What I'm after here is to have a constant capture source that doesn't care if it has a signal or if the signal is lost or re-acquired later. Your original driver had this same problem and I was hoping to not encounter it in the new one.

Any ideas?

Tim

0 Kudos
Reply

4,472 Views
Sasamy
Contributor IV

Hi Tim,

I added the blue screen in case of signal loss, reset of the channel after detection of a signal is made similar to the driver in upstream.

https://github.com/sasamy/tw6869/commit/f8f97e11e46932ed2d1adab5f7134633feee4c1e

0 Kudos
Reply

4,472 Views
timharvey
Contributor IV

Alexander,

Thanks - that's a great improvement. There is still an issue if you happen

to start streaming while a signal is not connected. I will see if I can

resolve that tomorrow.

Tim

0 Kudos
Reply

4,472 Views
Sasamy
Contributor IV

Hi Tim.

1. The main difference from the driver in upstream

https://git.linuxtv.org/media_tree.git/tree/drivers/media/pci/tw686x?id=1c9f47195ef83ffd59ff665f2006...

the possibility of transferring to the user space of physical addresses

tw6869/tw6869-video.c at master · sasamy/tw6869 · GitHub

the addresses used in the IPU, VPU, GPU for direct access to the data without CPU involvement

https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2src/v4l2_buffer_pool.c#L169

https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2src/v4l2_buffer_pool.c#L179

userptr DMA added mainly for testing

tw6869/tw6869-video.c at master · sasamy/tw6869 · GitHub

https://github.com/sasamy/mxc-v4l2-tvin-tw6869-vdi

2. tw6869 have a hardware bug - in some cases field order is wrong

http://www.mistralsolutions.com/newsletter/Jul13/Field_Dominance_Algorithm.pdf

Figure 4: Field-swap artifacts

field-swap.png

but that error can not be determined by the status registers. The upstream driver is trying to solve it

https://git.linuxtv.org/media_tree.git/tree/drivers/media/pci/tw686x/tw686x-video.c?id=1c9f47195ef83...

but this does not solve the problem - for a short time when you connect RCA Plugs, signal may disappear and reappear many times (contact bounce) and it is impossible to learn about it in such a way.

I tried to solve it in another way, but I had to turn off the blue screen. I agree that it is uncomfortable, so I try to find a better solution.

Alexander

0 Kudos
Reply

4,472 Views
athomas
Contributor III

I am having an issue with the driver and can not get a video out of the TW6869, I am also having the IMX6 lock up after trying to access a video device the 2-5+ time. I think it is related to the TW6869/DMAs and interrupts. I am seeing V4L2 select timeouts being logged in the output of attempting to play a video through Mplayer. I was wondering if anyone else had seen or resolved a similar issue.

Using 3.10.53 with a BD Trusty Rootfs and patched in the referenced TW6869 driver with MSI disabled on a custom IMX6Q board.

- Driver loads and video0-7 are created

- Using mplayer and checking the logs suggests it is able to properly read and communicate with the TW6869

- I can see some interrupts fire after trying to view a video, but only when I first try to access the video. I might say see only 6 interrupts after first trying to access a video (/dev/video0)

- Video is black/blank, and  Mplayer also reported 0 frames received/processed with 1 dropped when I kill mplayer or similar tool

When I try to use GStreamer or Mplayer to access the video, I get a messages that almost looks like the messages below with the core V4L2: select timeout being the same.

v4l2: select timeout ??% ??,?% 0 0

v4l2: select timeout ??% ??,?% 0 0

v4l2: select timeout ??% ??,?% 0 0

When the IMX6 locks up it is the normally the 2-3 attempt to access a video device, but it can much later if it does not happen the first couple attempts to access a video device. When the issue occurs, I can not change to another terminal (Ctrl-Alt-Fx) nor get any response from the console connected to another computer.

I have tried using a TW68 driver written by Intersil that works for the TW6869, but I get a similar response. It is not the same TW68 driver available in the 4.X+ Linux Kernel.

Could there be a missed V4L2 or MXC VPU/capture kernel configuration I am missing?

0 Kudos
Reply

4,472 Views
athomas
Contributor III

After turning up debug on the Intersil driver, I noticed a warning about my coherent DMA buffer pool being to small at 256 KiB. After setting it too be much larger via a kernel boot parameter, I can now DMA video streams. The change also stopped locking up my system. I have not switched back to the this driver, but will do so tomorrow to confirm it was the same root cause.

I did not have success with GStreamer 1.0 and the TW6869, but I was able to get it going with 0.10. When I switch back to this driver, I will test again.

0 Kudos
Reply

4,472 Views
tengri
Contributor IV

Dear A Thomas, frankba‌ and @josésanhueza 

Hi, I'm having the same issue on TW6865 chip. Were you able to solve it and can you kindly the share it with us. 

Thanks in Advance

0 Kudos
Reply

4,472 Views
kikimorisato
Contributor II

hi Alexander

1. i want change width 720 to 640

how can i change at "linux-imx_3.14.38_6qp_beta-tw6869"

(tw68 buffer setting... and etc...)

2. i want ntsc field drop

so width=720(or 640) / height=240

plz help me...

thk

0 Kudos
Reply

4,471 Views
Sasamy
Contributor IV

Hi ,

in an attachment the version of the driver in which is added support of several modes

mode 0 (default - interlaced frames) 720x480(576)

mode 1: 720x240(288)

mode 2: 640x240(288)

mode 3: 480x240(288)

mode 4: 360x240(288)

mode 5: 320x240(288)

mode 1-5 - even fields with horisontal scaling, vertical resolution depends on the current video standard.

Example of the pipeline (gstreamer-imx):

gst-launch-1.0 imxv4l2videosrc capture-mode=2 device=/dev/video0 ! imxg2dvideosink window-width=640 window-height=480 window-x-coord=0 window-y-coord=0


Alexander

0 Kudos
Reply

4,472 Views
kikimorisato
Contributor II

Thank you Alexander

Great!!!

0 Kudos
Reply

4,477 Views
frankba
Contributor III

Hi Alexander,

thanks for sharing the code. We don't see pictures yet (yocto 3.10.53 kernel) like José, but are quite optimistic with your driver. Do you have a public git repo for it? In my setup, loading (modprobe) and immediately unloading (modprobe -r) the driver gives a seg fault in kfree. IMHO the line

vdev->release = video_device_release;

is wrong, because vdev was never allocated directly but is part of a larger struct. Shouldn't it be

vdev->release = video_device_release_empty;

Frank

4,472 Views
Sasamy
Contributor IV

Hi Frank,

you're right, corrected. At the moment we have no git repo for the driver. I want to clarify, patches for plugins is required because of the difference in formats (I420, NV12 is not supported in the TW6869), therefore it isn't enough to add only kernel driver.

Alexander

0 Kudos
Reply

4,489 Views
josésanhueza
Contributor II

Hi:

   I've followed all the tips i've found here, but gstreamer refuses to show the TW6869 on the screen.  The patch to the 3.10.53 works wonderfully and gstreamer shows me no errors or warnings, yet doesn display anything.  Currently i am using a VAR-SOM-MX6Q board with poky yocto daisy.  I think it may have something to do with DMA but i am running out of ideas about what to try.

Thanks in advance, any help will be greatly appreciated.

0 Kudos
Reply

4,489 Views
psidhu
Contributor III

Hi José,

Few things - do you see a blue screen on the display when you run the pipeline? If you don't, can you verify that a simple pipeline using videotestsrc works? If you do, then maybe you're capturing from the wrong device (i.e. /dev/video4 vs. /dev/video7).

Other than that, run your gstreamer pipeline with a --gst-debug=3 to absolutely verify that you're not getting any errors. Finally make sure to use Alexanders gst1-plugins-imx-tw6869-sk.patch.zip patch to gstreamer-imx. This way, you can use imxv4l2src (from gstreamer-imx) and pipe it into the ipu via imxipuvideosink.

- Pushpal

4,489 Views
josésanhueza
Contributor II

I forgot, the videotestsrc pipeline works with all sinks i have tested.

0 Kudos
Reply

4,489 Views
josésanhueza
Contributor II

Thanks for answering. No blue screen, tried with gstreamer 0.1 and 1.0 . Driver loads OK patched directly into the kernel, shows DMA X ON when i activate gstreamer or try to access the video ports in any way. V4l2-ctl shows the driver info and activate buffers but aparently nothing is going in them.

Wich linux kernel are you using?, and gstreamer?. I have tried with 3.10.53. I don't know what else to try, i think there could be something wrong with the device tree or the kernel config.

0 Kudos
Reply

4,489 Views
Sasamy
Contributor IV

Hi José Sanhueza,

all patches tested, but we use Buildroot, so I'm not sure that they will work unchanged in Yocto.

https://youtu.be/YwQH2tWzH3g

https://youtu.be/nT8mX-_Eu9s

There is an easier way - we can change the offset to physical addresses as in the freescale release 3.14.28

http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/tree/drivers/media/platform/mxc/subdev/m...

I'll try it later, because now busy with other projects. I am sorry for the delay in replying.

Alexander

4,489 Views
josésanhueza
Contributor II

I see your videos and that is exactly what i want. I was hoping to later pass it on to opencv for processing but i am stuck with an aparently faulty DMA. Is it posible for the device tree to disrupt DMA? The board came with a kernel with pre-configured drivers to use with an OV5640 camera and IPU modules for MXC_v4l2_capture, but i removed them for compatibility (errors at boot time because i had no camera installed) I am using a VAR-SOM-MX6 board from Variscite.

Best regards.

0 Kudos
Reply

4,489 Views
psidhu
Contributor III
all patches tested, but we use Buildroot, so I'm not sure that they will work unchanged in Yocto.

Works just fine in Yocto :smileyhappy:

4,489 Views
josésanhueza
Contributor II

Wich yocto. I use a poky daisy variation for the VAR-SOM-MX6 from Variscite.

0 Kudos
Reply