How to reduce video latency caused by GStreamer on i.MX6?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to reduce video latency caused by GStreamer on i.MX6?

4,287 Views
egmedical
Contributor I

We are using i.MX6 to process video from HD camera to show live preview on the screen, record the short video clips, etc. We use GStreamer to achieve this, but the delay between camera and display is higher than what we expected. What could we do to lower the video latency?

HW used:

  • Variscite VAR-SOM-MX6 Solo (i.MX6 Solo based SoM) -- I know that this is not the standard reference board, but there are no HW issues, so I hope it does not matter.
    • i.MX6 Solo
    • 1 GB RAM
  • custom carrier board for VAR-SOM-MX6 providing following:
    • ADV7181C HDTV Video decoder -- Used to convert analog YPbPr signal to 16-bit YPbPr (connected to CSI interface)
    • data connections (HDMI, UARTs, Ethernet, ...)
  • HD video camera (Sony FCB-EH6300) -- We are currently using 1080p/25 (1920x1200, 25 FPS, progressive) mode.
  • custom LCD monitor (FullHD IPS panel and chip providing the HDMI connection)

SW used:

  • our Linux image has been bitbaked using Variscite's Yocto Dora release based on fsl-L3.10.17_1.0.0GA release.
  • kernel 3.0.35
  • GStreamer 0.10.36
  • gst-fsl-plugin 3.0.9

We use minimal GStreamer pipeline (only mfw_v4lsrc and mfw_isink, nothing more) to get the video signal from camera and show it on LCD.


We did some measurements on whole system with following results:

  • video latency of whole system (from camera to LCD display) is around 270 ms, which breaks down to:
    • camera latency of 120 ms (or 3 frames, because we use 25 fps, therefore each frame takes 40 ms) -- Since we cannot change this behavior, we do not care about it for now. Maybe we will switch mode to 50/60 fps or use different camera.
    • ADV7181C causes no latency (there are no buffers in the chip).
    • i.MX6 processing takes around 135 ms -- This is the part where we would like to save some time. We believe that the pipeline is so simple that there is no need to induce such delay.
    • LCD panel processing takes few milliseconds (we cannot measure it precisely, but we cannot reduce it anyway, so there is no need to know precise value).

Below you can see oscilloscope output from our measurement, where we flashed bright light (white LED) directly into the camera, which causes bright flash of white on the LCD display (before the camera auto-gain even reacts), so there is rapid measurable change.

The channels show following:

  • Channel 1 (yellow) shows voltage on the LED (therefore shows when we flashed the LED).
  • Channel 2 shows Y component of YPrPb signal from camera. You can clearly see when the flash appears in the video signal, so we can measure the latency caused by the camera itself.
  • Channel 3 shows one differential pair in HDMI signal to LCD display. We were quite surprised that you can observe the change in HDMI signal so easily (even when using only 70 MHz oscilloscope). We can use this information to calculate the latency caused by processing in i.MX6. The cursors clearly shows that the latency is around 135 ms.
  • Channel 4 shows voltage on photodiode that was sensing the intensity of light from LCD panel, so it shows when the user can observe the change on LCD.


po_celou_dobu_snimku.png

Summary:

  • We would like to minimize delay caused by GStreamer processing, which currently takes around 135 ms for us, while processing 1080p/25 video feed.
    • Do you believe that there is any way to eliminate part of the delay? We know that there will always be some latency caused by processing, but we would like to eliminate it as much as possible.
    • Is there any easy way to break-down the latency caused by GStreamer, so we would know what causes the delay (format conversion, buffer passing, whatever else happens in there, ...)?
  • The other thread dedicated to video latency (i.MX6 Video Latency) did not helped us much, because we are already using mfw_isink, but our video delay is quite longer than what is described there (135 ms vs 59 ms). I am not sure what framerate has been used in that experiment though.
Labels (4)
0 Kudos
8 Replies

2,381 Views
jamesbone
NXP TechSupport
NXP TechSupport

Hello Ivo,

Are you mounting the camera first? or you are just running the gstreamer pipeline,  I think the latency came from the ADV7181, but I do not have something to test it.

0 Kudos

2,381 Views
egmedical
Contributor I

Hi James,

I am sorry, but do not know what do you mean by "mounting the camera", could you please clarify that?

We believe that ADV7181C does not cause any significant delay, because the chip has no internal buffers that could cause delay of this order (it would be quite a lot of data, remember we are talking about 1080p25). I also thing that we in fact confirmed it when we did the measurements I have posted results from, but I have no data recorded to share with you.

_________________________________________________________

EDIT: I have measured delay caused by ADV7181C, results are attached.

Setup:

  • Channel 1: Voltage on LED, showing when the light is shining into the camera optics.
  • Channel 2: Y channel of YPbPr signal from camera, clearly showing that the signal is delayed by 3 frames.
  • Channel 3: CSI1-D9 input pin of i.MX6, which carries most significant bit of 16-bit parallel YPbPr signal from ADV7181C into the i.MX6.

Results:

Here you can clearly see when the signal from camera reacts to short pulse of light from LED. You can also see that the ADV7181C output is not significantly delayed.

adv7181c-delay-1.png

Zoomed in little bit to show blanking between two frames (the short "gap" in the Y component from camera.

adv7181c-delay-2.png

Zoomed in to show separate pulses in both signal and showing some change in signal so you can compare them.

adv7181c-delay-3.png

Conclusion:

As you can see, there is no significant delay caused by ADV7181C. There is definitely delay caused by AD conversion, but we are talking about 74.275 MHz. That is pixel clock for 1080p30 (which is used by ADV7181C, although we use only 25p).

0 Kudos

2,381 Views
jamesbone
NXP TechSupport
NXP TechSupport

We are reviewing your issue internally, as mention before it is difficult for us to reproduced, so this is taking some time to research. I apologize

0 Kudos

2,381 Views
egmedical
Contributor I

James, do you have any new information regarding the latency? Do you need my help? I can carry out some tests if you are able to instruct me.

0 Kudos

2,381 Views
jamesbone
NXP TechSupport
NXP TechSupport

There is a parameter in mfw_v4lsrc which allow you to preview the camera output directly to the display, it use a different IPU path, maybe it is faster.

another advice would be testing to get a better timing using unit-test.

0 Kudos

2,381 Views
egmedical
Contributor I

Dear James,

could you please clarify if there is any chance to use preview function in mfw_v4lsrc in recent version of gst-gsl-plugin? If not, is there any alternative providing the same function, or was this functionality simply ditched? Thank you for any answer.

Best regards

Ivo

0 Kudos

2,381 Views
egmedical
Contributor I

Unfortunately I cannot test mfw_v4lsrc with preview enabled, because the attribute is not writable. We are using gst-fsl-plugins version 3.0.9 right now.

It seems that the gst-fsl-plugin-2.0.3 introduced some changes (at least that is what I am able to tell from sources on http://repository.timesys.com/buildsources/g/gst-fsl-plugin/, as I cannot find official Freescale repository) that disabled possibility to use preview function along with the following comment:

  /*

   * FixME: The overlay channel will cause v4l error:

   * v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0 error

   * disable it

   */

I tried changing the code so the preview parameter is writable, but while I am able to set the parameter to true, the preview function seems to be broken.

Could you tell me more information about the unit test? I am not sure which of them is related to the latency.

0 Kudos

2,381 Views
egmedical
Contributor I

Thank you very much for the effort. If you need any more information, please let me know.

0 Kudos