We are using i.MX6 to process video from HD camera to show live preview on the screen, record the short video clips, etc. We use GStreamer to achieve this, but the delay between camera and display is higher than what we expected. What could we do to lower the video latency?
HW used:
- Variscite VAR-SOM-MX6 Solo (i.MX6 Solo based SoM) -- I know that this is not the standard reference board, but there are no HW issues, so I hope it does not matter.
- i.MX6 Solo
- 1 GB RAM
- custom carrier board for VAR-SOM-MX6 providing following:
- ADV7181C HDTV Video decoder -- Used to convert analog YPbPr signal to 16-bit YPbPr (connected to CSI interface)
- data connections (HDMI, UARTs, Ethernet, ...)
- HD video camera (Sony FCB-EH6300) -- We are currently using 1080p/25 (1920x1200, 25 FPS, progressive) mode.
- custom LCD monitor (FullHD IPS panel and chip providing the HDMI connection)
SW used:
- our Linux image has been bitbaked using Variscite's Yocto Dora release based on fsl-L3.10.17_1.0.0GA release.
- kernel 3.0.35
- GStreamer 0.10.36
- gst-fsl-plugin 3.0.9
We use minimal GStreamer pipeline (only mfw_v4lsrc and mfw_isink, nothing more) to get the video signal from camera and show it on LCD.
We did some measurements on whole system with following results:
- video latency of whole system (from camera to LCD display) is around 270 ms, which breaks down to:
- camera latency of 120 ms (or 3 frames, because we use 25 fps, therefore each frame takes 40 ms) -- Since we cannot change this behavior, we do not care about it for now. Maybe we will switch mode to 50/60 fps or use different camera.
- ADV7181C causes no latency (there are no buffers in the chip).
- i.MX6 processing takes around 135 ms -- This is the part where we would like to save some time. We believe that the pipeline is so simple that there is no need to induce such delay.
- LCD panel processing takes few milliseconds (we cannot measure it precisely, but we cannot reduce it anyway, so there is no need to know precise value).
Below you can see oscilloscope output from our measurement, where we flashed bright light (white LED) directly into the camera, which causes bright flash of white on the LCD display (before the camera auto-gain even reacts), so there is rapid measurable change.
The channels show following:
- Channel 1 (yellow) shows voltage on the LED (therefore shows when we flashed the LED).
- Channel 2 shows Y component of YPrPb signal from camera. You can clearly see when the flash appears in the video signal, so we can measure the latency caused by the camera itself.
- Channel 3 shows one differential pair in HDMI signal to LCD display. We were quite surprised that you can observe the change in HDMI signal so easily (even when using only 70 MHz oscilloscope). We can use this information to calculate the latency caused by processing in i.MX6. The cursors clearly shows that the latency is around 135 ms.
- Channel 4 shows voltage on photodiode that was sensing the intensity of light from LCD panel, so it shows when the user can observe the change on LCD.
Summary:
- We would like to minimize delay caused by GStreamer processing, which currently takes around 135 ms for us, while processing 1080p/25 video feed.
- Do you believe that there is any way to eliminate part of the delay? We know that there will always be some latency caused by processing, but we would like to eliminate it as much as possible.
- Is there any easy way to break-down the latency caused by GStreamer, so we would know what causes the delay (format conversion, buffer passing, whatever else happens in there, ...)?
- The other thread dedicated to video latency (i.MX6 Video Latency) did not helped us much, because we are already using mfw_isink, but our video delay is quite longer than what is described there (135 ms vs 59 ms). I am not sure what framerate has been used in that experiment though.
Hi James,
I am sorry, but do not know what do you mean by "mounting the camera", could you please clarify that?
We believe that ADV7181C does not cause any significant delay, because the chip has no internal buffers that could cause delay of this order (it would be quite a lot of data, remember we are talking about 1080p25). I also thing that we in fact confirmed it when we did the measurements I have posted results from, but I have no data recorded to share with you.
_________________________________________________________
EDIT: I have measured delay caused by ADV7181C, results are attached.
Setup:
- Channel 1: Voltage on LED, showing when the light is shining into the camera optics.
- Channel 2: Y channel of YPbPr signal from camera, clearly showing that the signal is delayed by 3 frames.
- Channel 3: CSI1-D9 input pin of i.MX6, which carries most significant bit of 16-bit parallel YPbPr signal from ADV7181C into the i.MX6.
Results:
Here you can clearly see when the signal from camera reacts to short pulse of light from LED. You can also see that the ADV7181C output is not significantly delayed.
Zoomed in little bit to show blanking between two frames (the short "gap" in the Y component from camera.
Zoomed in to show separate pulses in both signal and showing some change in signal so you can compare them.
Conclusion:
As you can see, there is no significant delay caused by ADV7181C. There is definitely delay caused by AD conversion, but we are talking about 74.275 MHz. That is pixel clock for 1080p30 (which is used by ADV7181C, although we use only 25p).
