Change in framebuffer contents causes distrotion in encoded video

[isaacnickaein]

We have a i.MX6 module which outputs a video the FPGA. At the same time, it receives an input video from the FPGA (on parallel CSI interace) and streams it on network after H.264 encoding. The problem we are facing is there are sudden distortions in the encoded video.

After debugging, it has been clear that the distortions occurs whenever the output image (framebuffer contents) of i.MX6 changes. For instance, when we write random data to framebuffer on periods of 10 sec, the distortions happens every 10 seconds.

To simplify the problem, we have setup a loopback architecture,

                         ==================   DISP0 (BT.1120)    |=================|

     LAN (H.264 stream)  |                |  =================>  |                 |

    <==================  |     i.MX6      |                      |      FPGA       |

                         |                |  <=================  |                 |

                         ==================   CSI0 (BT.1220)     |=================|

in which the FPGA acts as a buffer which receives and immediately sends the frames backs to i.MX6. It also outputs the same video on the SDI.

In i.MX6, the output video is GStreamer test pattern by the following command:

gst-launch videotestsrc ! mfw_v4lsink

To encode the feedback video from FPGA, we have tried both following pipelines:

# Encode and stream on network

gst-launch mfw_v4lsrc capture-mode=0 ! queue2 ! vpuenc codec=6 bitrate=5000000 seqheader-method=3 ! queue2 ! mpegtsmux ! udpsink host=10.1.1.169 port=6666

# Encode and store to a file

gst-launch mfw_v4lsrc ! queue ! vpuenc codec=0 ! matroskamux ! filesink location=output.mkv sync=false

The distortion happens in both cases (network stream and file storage). I have attached the output file of second pipeline.

It is strange that this distortion happens only when contents of framebuffer is changed. We haven't figured it out what can be possibly the cause of this issue.

Original Attachment has been moved to: output.mkv

[Yuri]

Hello,

  Looks like, we have a performance issue in the case. Perhaps the system is overloaded.

It make sense  to try the recent BSP L.3.14.28

http://www.freescale.com/products/arm-processors/i.mx-applications-processors-based-on-arm-cores/i.m...

Next, You may try the solution in the following thread

"Why encoded video is choppy when saved to file?"

https://community.freescale.com/thread/314230

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[isaacnickaein]

Hi Yuri,

I've performed another test:

Instead of my Qt application, I tried the example OpenGL Window application shipped with Qt.

The issue is still the same for this application: The video output from i.MX6 is fine, but the encoded video (whether it is streamed or recorded to a file) becomes distorted.

I played with some of the EGLFS parameters. The only thing that I find relevant was QT_QPA_EGLFS_SWAPINTERVAL parameter. Setting it to larger values reduced the distortion. However, it also slowed down the rotation of triangle. I think it reduces rendering rate ("swap" rate) which in turn reduces the rate of writes to the framebuffer.

I've attached two encoded videos, one with swapinterval=1 (default value) and one with swapinterval=10.

Bests,

Isaac

[isaacnickaein]

Hi Yuri,

We are currently stuck with Kernel 3.10 because this patch for BT1120 video output:Patch to Support BT656 and BT1120 Output For i.MX6 BSP and several other patches. We are planning to move to latest kernel but that is not possible in short time. I've also tried (echo 1 >/proc/sys/vm/lowmem_reserve_ratio) but that didn't help neither in file storage nor in streaming case.

One interesting observation is that I usually can get a jitter-free encoding by changing the device of mfw_v4lsink to /dev/video16:

gst-launch videotestsrc ! mfw_v4lsink device=/dev/video16

I've noticed after running this, the fbset reports tripled virtual y-resolution for framebuffer:

mode "1920x1080-25"

    # D: 74.250 MHz, H: 27.882 kHz, V: 25.278 Hz

    geometry 1920 1080 1920 3240 32

    timings 13468 20 3 20 2 720 1

    accel false

    rgba 8/0,8/8,8/16,8/24

endmode

And if the above pipeline has jitter, it can be fixed by first closing all pipelines (the pipeline itself and the encoding pipeline) and then restarting them.

However, I still cannot figure out a way to have similar result when my Qt EGLFS application is outputting the video (instead of videotestsrc).

I have tried

export FB_MULTI_BUFFER=3

and running the following commands:

./myapp -platform eglfs                                 ==> Distortion is still present

QT_QPA_EGLFS_FB=/dev/fb1     ./myapp -platform eglfs    ==> No video output

QT_QPA_EGLFS_FB=/dev/video16 ./myapp -platform eglfs    ==> Distortion is still present

This raises following questions:

1. Why using video16 in the videotestsrc pipeline fixes the distortion?
2. How a similar configuration (e.g. outputting to video16) for Qt EGLFS application can be achieved which hopefully doesn't have the distortion problem anymore.

Bests,

Isaac