Why encoded video is choppy when saved to file?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Why encoded video is choppy when saved to file?

7,379 Views
dilipkumar
Contributor III

I'm streaming video from a camera using the following pipeline.

     gst-launch-0.10 mfw_v4lsrc ! vpuenc codec=6 ! matroskamux ! filesink location=test4way.mkv sync=false async=false

My requirement is to save the encoded video in h.264 format. But when i try to view the saved file using gplay (or any player in my PC) the video is found to be choppy. It pauses every few seconds and continues with the loss of few frames while the video is paused. I dont think the problem is with the encoder. Because i've encoded and streamed the video over network and checked that the encoded video is not choppy. I think this problem occurs while saving to file locally. Is the imx6quad not capable of fast data transfer.? Do i need to change anything in the kernel like DMA for example.? I'm using a sabrelite development board (iMX6Q) from boundary devices running kernel version 3.0.35. Any suggestions?

42 Replies

932 Views
dilipkumar
Contributor III

HI Eric,

First of all thank you very much for trying out my pipeline before replying.:smileyhappy:

The reason I'm getting such low performance on my SD card could easily be because it is not a good quality one. So lets not worry too much about it.

The file you recorded contains video of VGA resoluion encoded in H.263 format at 30 fps for 20 seconds. I don't know why timestamps are missing in the file but the video is still playable though.

Was this video saved to your Lexar card or did you save it to a HDD?

I think you misunderstood what I meant when I said "choppy". If you notice very carefully at the video you recorded at exactly the 4th second and 19th second you will be able to notice that there is a small pause visible in it. Now there are 2 pauses in just a 20 second video. Just imagine how many such pauses will be there for say a 30minutes video which is unacceptable. Plus if I wanted to save full HD or 720p video, the video quality would definitely suffer.

These are the images I've tested this pipeline on :

Pre built :

Yocto Dora release (for i.MX6) http://boundarydevices.com/eula?file=fsl-image-gui-nitrogen6x-20130928181014.rootfs.sdcard.gz

i.MX6 kernel 4.1.0 http://boundarydevices.com/eula?file=ltib4.1-with-x-20130909.tar.gz

Custom built :

i.MX 6 Q/D/DL/S L3.0.35_4.1.0 GA release

The videos generated in all of them contain this pause.

P.S : My requirement is H.264 not H.263

0 Kudos

932 Views
EricNelson
Senior Contributor II

Oh. I was looking for something more obvious...

This brings up a handful of thoughts and suggestions of how to narrow things down.

Do you know whether this is related to the Matroska mux or the H.263 codec? (i.e. have you tried using other container formats or codecs?)

I'll also try to confirm your result that the problem doesn't occur if saving to a RAM disk. Have you tried using other filesystems (VFAT, NFS)?

It seems that since the issue is repeatable, it may be possible to use videotestsrc instead of the camera and generate two files (one in RAM disk and one in ext3) and compare the two in order to narrow things down. By using the 'num-buffers' property, we should be able to generate precisely the same data in multiple attempts.

This will likely require more knowledge, tools, and time than I have at my disposal, but there are some quick things that could be determined (file sizes), and using tools like libavformat to walk the segments of the good and bad files might show whether the issue is in the payload or data.

0 Kudos

930 Views
dilipkumar
Contributor III

Hi eric,

Ive tested encoding using mxc_vpu_test.out and i'm noting that the frame loss occurs because of long time for some write() calls in utils.c to complete. And i'm also suspecting that journaling in ext4 filesystem could be an issue here. My ext4 HDD partition is mounted in ordered mode.

     /dev/sda1 /media/sda1 ext4 rw,relatime,user_xattr,barrier=1,data=ordered 0 0

I unmounted the partition and remounted it in writeback mode and disabling write barrier.

mount -t ext4 /dev/sda1 /media/sda1 -o barrier=0,data=writeback

     /dev/sda1 /media/sda1 ext4 rw,relatime,user_xattr,barrier=0,data=writeback 0 0

Afterwards i encoded and saved 5000 frames to HDD using ./mxc_vpu_test.out The output is as follows.

[INFO]  Finished encoding: 5000 frames

[INFO]  enc fps = 258.28

[INFO]  total fps= 30.01

Previously the total fps is always lower. The video when played back also shows no missing or skipped frames. any ideas on this ? EricNelson LeonardoSandovalGonzalez

Also i tried adding the O_SYNC flag when opening the file to ensure constant flushing to HDD instead of caching each write and flushing when the number of dirty pages exceed dirty_background_ratio. This might help in the case of HDD rather than SD card. any suggestions would be helpful. Thank you.

0 Kudos

930 Views
LeonardoSandova
Specialist I

Dilip,

can you try others muxers (avimux or flvmux)?. We have found better performance on these two.

0 Kudos

930 Views
dilipkumar
Contributor III

Hi Leo,

I can assure you this issue is not related to any muxers as far as i have tested. The reason why you aren't able to see that pause effect when you use avimux or flvmux is because the skipped frames and their timestamps wont be there altogether. If you keep a moving object in front of the camera and record it you can observe this. There will be no pause but some frames will be missing. Only in matroskamux you will be able to see the pause whenever there is a frame loss.

If you think this is an issue related with muxers you can just try the sample code mxc_vpu_test.out to confirm. In that code check how much time it takes for each write operation. Sometimes a write operation takes more than 33ms ( around 180 ms in HDD and 2 full seconds in SD card ) . At that time no frame is dequeued from the camera. This is what is causing the frame loss.

0 Kudos

930 Views
LeonardoSandova
Specialist I

Thanks Dilip for your inputs.

Honestly, I did not check the timestamps so when I trie flv-avi, I thought the problem was solved and it was something related to muxer. I will check again these points. BTW, if you store the file in a non-SD device (eMMC for example), did you observe the problem? How are you measuring the system calls duration? strace?

0 Kudos

930 Views
dilipkumar
Contributor III

I couldn't test with an eMMC device because I don't have one. I tried with a sata HDD and different SD cards. I noted that this pause was shorter in HDD. I think that the type of filesystem also plays an important role here. As i mentioned in an earlier post, ext4 with barriers enabled and journaling mode set to ordered (default auto mount options for ext4) would block for 200ms at times. But if I opened the file with O_SYNC flag and mounted the same ext4 filesystem with barriers disabled and journaling mode set to writeback, all the write operations took less than 33ms and I observed no frame loss was present. In fact this was the only condition in which I have been able to save encoded video without frame loss.

If somebody with much knowledge about different filesystems could share their insight it would help debug the problem further. Thank you.

0 Kudos

930 Views
LeonardoSandova
Specialist I

Dilip,

I tried before your recommended ext4 parameter values (user_xattr,barrier=0,data=writeback) on SD and eMMC BUT I still see the hiccups (at least with matroskamx). As you point out, we can tune the file system parameters but at the end there may be some bottleneck someone else (perhaps on the SDIO module?). I will push this thread internally so an specialist can give us better insight.

0 Kudos

930 Views
dilipkumar
Contributor III

Hi Leo,

Thanks for your reply, please give this issue more priority as recording a video is a basic thing and if we are successful in doing that without any hiccups it opens a lot of possibilities in which we can use the iMX6 application processor.

0 Kudos

930 Views
LeonardoSandova
Specialist I

Dilip,

FYI http://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

These sysctrl parameters change the cache settings on the system, you may try to the values indicated there. As I said before, the issue is under investigation but it may take time before a finding the root cause and its solution.

0 Kudos

930 Views
dilipkumar
Contributor III

Hi Leo,

I tried increasing the caching size and increased the rate of flushing using the method specified in the link you posted.

vm.dirty_ratio = 80
vm.dirty_background_ratio = 50
vm.dirty_writeback_centisecs = 50

I couldn't notice any performance difference though. The write calls keep getting blocked after some time and blocks for around 900 ms now. I'm using an 8 GB class 10 (UHS 1) SanDisk Ultra microSD card with ext2 filesystem for testing. I'm using ext2 because i want to reduce overhead due to journaling in ext3/ext4.

ext2 is proven to be faster than ext3 and ext4 for write speeds. Check this link : A Comparison of Ext2, Ext3 and Ext4 Performance - High Performance Computing - Wiki - High Performan...

Could you try them out too and tell me if you can notice any difference? Or do I need to vary some other parameters to see the difference in performance?

0 Kudos

930 Views
LeonardoSandova
Specialist I

Hi Dilip.

On my side, files stored on ext2 are a bit better but still with problems (I am recording a timer, and around the second 30th, I see several frozen frames). I also checked the vm parameters and seems that do not play a strong role on this. Thanks for trying out. Issue is still investigated.

0 Kudos

930 Views
dbarton91
Contributor I

Was this issue ever resolved? I am running the 4.1.15 kernel and gstreamer 1.0, and I am still seeing frozen frames around the 35 second mark.

0 Kudos

930 Views
LeonardoSandova
Specialist I

Hi Dilip. Issue has been pushed internally.  You are right,  this is an important issue.

0 Kudos

930 Views
dilipkumar
Contributor III

No. I tried strace but it appears to slow down the overall process such that fps drops below 15.

So instead of strace , you can modify the source code of the sample application and find the time taken for each write operations by using gettimeofday before the write function is called and after it is completed. Then print the difference between the two values.

This is my modified fwriten function in utils.c

int    /* write n bytes to a file descriptor */

fwriten(int fd, void *vptr, size_t n)

{

    int nleft;

    int nwrite;

    char  *ptr;

struct timeval write_start,write_end;

int sec,usec, total_time = 0;

    ptr = vptr;

    nleft = n;

gettimeofday(&write_start,NULL);

    while (nleft > 0) {

        if ( (nwrite = write(fd, ptr, nleft)) <= 0) {

            perror("fwrite: ");

            return (-1);            /* error */

        }

        nleft -= nwrite;

        ptr   += nwrite;

    }

gettimeofday(&write_end,NULL);

    sec = write_end.tv_sec - write_start.tv_sec;

    usec = write_end.tv_usec - write_start.tv_usec;

    if (usec < 0) {

        sec--;

        usec = usec + 1000000;

    }

    total_time = (sec * 1000000) + usec;

    printf("wrote = %d bytes in %d micro sec\n",n, total_time);

    return (n);

} /* end fwriten */

0 Kudos

930 Views
LeonardoSandova
Specialist I

Dilip, I believe your findings deserve a new discussion thread based on your C code. Findings on the latter one will directly impact this one.

One more request: run your app with 'strace -T'. Do you observe the same delays?

Leo

0 Kudos

930 Views
dilipkumar
Contributor III

Hi Leo,

Previously I had tried this, ./mxc_vpu_test.out -C config_enc & strace -p $! -r

I thought strace was slowing down the process, but it was all the prints to stdout that was slowing it down. So I tried storing it to file.

I tried to avoide strace because it was not an efficient tool here because I do not know when this particular write call was gonna happen or how long it will take. Skimming through a long log file is tedious.

But as you suggested, I tried ./mxc_vpu_test.out -C config_enc & strace -T -p $! -o /media/sda1/strace.txt . Note that I saved the log file to a different drive so that it doesn't create additional write overhead.

The log file generated had more than 17000 lines just for encoding 1000 frames. So I'm sharing only a part of it. strace.txt - Pastebin.com . You can see that one particular write operation took 926 ms to complete. Which means about 30 frames where not dequeued from the camera in proper time which causes this skip in the encoded video when played back.

0 Kudos

930 Views
dilipkumar
Contributor III

I'm sure that this issue is not related to matroskamux or any codec in particular. I've tried saving without any mux and different muxers (avimux). I've also used all the different codecs available but still the frame loss is there. The videos do not contain the pause when using avimux or no muxer but the frame loss is there. The pause in matroska mux is because only the mkv files seem to contain proper timestamps. These files allow for seeking when played using VLC or other media player.

I've tried saving to an ext3, ext4, NTFS and FAT32 filesystem. The saved files look the same in all of them.

to check whether it was a limitation of gstreamer I also tried the vpu test application from Freescale in /unit_tests

/unit_tests/mxc_vpu_test.out -E "-o temp.h264 -f 2 -c 1000 -w 640 -h 480 -q 1" The output obtained contains frame loss which can be clearly seen if the video contains any moving object because there is no pause present.

If you wish to use videotestsrc instead of camera in gstreamer , I suggest using videotestsrc pattern=18 because it'll be easier to track the moving object.

0 Kudos

932 Views
LeonardoSandova
Specialist I

When I ran out of ideas, the last resource is EricNelson :smileyhappy:

Leo

0 Kudos

932 Views
EricNelson
Senior Contributor II

Hi guys,

My first suspicion is that there's something odd with the filesystem mount.

Since even with fast (SSD) I/O things are failing, but things work with a RAM disk, this must be a question of latency and not throughput, right?

I'm getting a write speed of 9MB/s using the command-line above on a plain-vanilla Lexar SD card:

~# dd if=/dev/zero of=/root/temp.img count=1 bs=500M conv=fsync 
1+0 records out
524288000 bytes (524 MB) copied, 56.1646 s, 9.3 MB/s
~# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,relatime,errors=continue,user_xattr,barrier=0,data=writeback 0 0
...
0 Kudos

932 Views
dilipkumar
Contributor III

I tried the above pipeline without the sync and async properties.. But the issue still exists... Also there is no such problem when rendering on the screen using mfw_v4lsink or mfw_isink.

I did notice two warnings though after setting GST_DEBUG=*:2 . But i believe they are immaterial to this issue...

0:00:01.159842898  33640xba718 WARN              vpuenc vpuenc.c:1390:gst_vpuenc_sink_event: Unsupport newsegment format 2

[INFO]  chromaInterleave 0, mapType 0, linear2TiledEnable 0

0:00:01.210577899  33640xb2a28 WARN                 bin gstbin.c:2380:gst_bin_do_latency_func:<pipeline0> did not really configure latency of 0:00:00.000000000

The caps from mfw_v4lsrc is

caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1

using gst-inspect vpuenc , i noticed that I420 is supported by vpuenc, but still I get the warning.

And I get the second warning when i'm rendering to display also. But rendering works without a problem.

0 Kudos