Video delay problem in Android

[sergeikotlyachk]

Hi,

My company is trying to make video playing in Android with i.MX6 board to work with live video sources. Unfortunately there is significant delay of 12-14 seconds between changes in the live video and showing these changes on the screen. I was trying various options and came to understanding that root cause of the problem is somewhere in Freescale code in OMX / VPU / GMPlayer, but can't figure out what should I change in firmware in order to reduce this delay. Please help, otherwise we will be unable to use Freescale for our commercial product.

Thanks.

[jack_mao]

Hi,

     I ran vlc 2.0.7 and OMXGM on JB4.2.2 to play rtsp stream simutaneously, there isn't such big difference between them. The delay seems less than 0.5 second. What's your test method to find the delay of 1.7-2.0 seconds?

     By the way, there is another patch suppose to improve the delay, you could have a try.

best regards

Jack

[sergeikotlyachk]

Hi,

- You attached exactly same patch as before. Must be some kind of a mistake.

- I measure delay manually, by starting stop watch when I put a hand over the camera and stopping stop watch when I see it on the screen. There could be small error around 0.2 s due to my reaction time, but subtracting such error from delay is still much higher than 0.5s.

[jack_mao]

Hi,

    Sorry I attached a old patch, please use this one, by the way , we test this issue with the stream you sent to us , setup the rtsp server, so there should be other system delay in your realtime camera test, so could you use rtsp server test this and compare the result ?

[jack_mao · Accepted solution]

Hi Sergi,

        Here is a patch from our R&D,   you could try on your side.

best regards

Jack

[sergeikotlyachk]

Hi Junping,

Thanks for the patch. It improved delay to be stable around 1.7 - 2.0 seconds. Since receiving the patch I have been working to try it out in JB4.2.2 / 1.0FSL. I thought it would be better, but today test showed that delay increased a little bit in JB (by half of a second). There are also other issues we are having with Jelly Bean thus for our first version of the product we will likely target ICS. I need this delay to be further reduced to be under 0.5 second matching delay in VLC running on desktop. Please let me know what can be done to further improve this. (It is crucial for Point-Tilt-Zoom cameras because 2 second delay feels like a very slow response when user controls it)

[jack_mao]

Hi Sergei,

      I understand your status and also reproduce this on our platform, I asked our R&D engineer to look this , we added trace and found some buffering data behavior during the streaming playback, seems code issue. I submit a ticket , our R&D engineer will look into this issue.

jack

[sergeikotlyachk]

Hi Junping,

Thank you for looking into this. It has been a week and I wonder if you have any updates?

[jack_mao]

Hi Sergei,

       The issue is handling  by the RTSP owner now, he is debugging this issue with my RTSP server, if there is any progress, will let you know.

Jack

[sergeikotlyachk]

Hi,

1. Can you estimate how long it will take to fix? We need to align our product plans with resolution of this problem. I hope it can be done by beginning of August.

2. We are planning to switch to Jelly Bean soon. If you are to make a patch for this fix, please do so for Jelly Bean.

Thanks.

[jack_mao]

Hi,

        The R&D engineer suppose to give a update next week.  If the change is in the openmax or RTSP code, I think we can send the patch to you to avoid the baseline alignment  issue.

best regards

Jack

[jack_mao]

Hi,

      I tried to reduce the size of the buffer data to 30*1000, the first frame come out more quickly, by the way I compare the mx player and native video player of android, the video player start very quickly, so it seems some delay is caused by the application, I read the log, due to no audio , the mx player will wait for timeout , so it  will take longer.

       what I did test is install the vlc player apk, and in the application select more-> manual rtsp -> select the player, different player has different result.

       So I think you need a realtime player.

best regards

Jack

[sergeikotlyachk]

Hi Junping,

Unfortunately, I will still need your help:

1. When you say 'native video player of android', what do you mean? We simply used MediaPlayer class and gave it rtsp url. Are you saying that MediaPlayer class is not supposed to play rtsp urls? Then why does it support it?

2. When you said ' the video player start very quickly', which player you were referring to?

3. What application you are referring to when you said 'delay is caused by the application'?

4. VLC Player can not be used for comparison because it uses software mode which pegs cpu ( check cpu usage when using MX player in HW accelerated mode and VLC player, you should see big difference). I have tried other software mode players and while they have smaller delay, they all peg cpu. We are not interested to use software mode because it will only play one stream and not more. We are interested in using Freescale VPU as explained in https://community.freescale.com/docs/DOC-93611

And since delay and de-syncronization is occurring in Freescale code/hardware, I am expecting your help on this matter. What is the reason of having VPU when you can not use it?

Thank you,

  Sergei.

[jack_mao]

Hi Sergei,

       1.The native video player means gallery which is pre-installed in android system. It will use the hardware VPU, all my test is based on the hardware VPU, I just want you to compare the different on your side, I need make sure that we meet the same result.

       2. Gallery start very quickly.

       3. mx player, I notice it spend some time on audio play by the log. please confirm this.

       4.  I use VLC Direct Pro apk  to use it RTSP streaming function, it will auto popup the window to let the user select the Gallery or mxplayer.

     All I did is base on the VPU,  VLC is using as the streaming protocol, mxplayer or gallery as the video player.

      Please try those to see if we meet the same result.

best regards

Jack

[sergeikotlyachk]

I tried what you asked for.

3. Mx player takes longer time to start, but video delay is the same as through Gallery.

4. VLC Direct Pro + Gallery: initial delay for 1080p resolution is 4.5 seconds. Initial delay for 800x600 resolution is ~2 seconds. After couple of minutes, delay for SD resolution becomes 5 seconds.

Problem is still there.

What did you do in firmware to ' reduce the size of the buffer data to 30*1000' ?

[jack_mao]

it's same as #define PACKET_CACHE_SIZE (4*OMX_TICKS_PER_SECOND), default is 4s buffer, you could reduce this time. How do you calculate the start time, when I change this define, I found gallery start immediately.

what I did is use the file you uploaded and streaming by VLC server. If you still get delay using gallery with my test way, maybe somgthing is different between us.

[sergeikotlyachk]

Junping, what other languages do you speak? We have Chinese developer who can help translate it back and forth, since I can see that we are spending a lot of time explaining things to each other and because of time difference it makes things worse. Using my colleague's help could speed things up.

[sergeikotlyachk]

I never calculated start time. It is not start time that we care about. Start time can be 10 seconds no problem, it is not such a big deal. We are measuring latency between the video transmitted and shown. If you start separate VLC on the desktop in playing mode that connects to the streaming VLC server, it shows less than second latency where as Freescale board shows the latency/video delay I talked about. When started, latency is at 2-4 seconds and then increasing with time of playing. The easiest is to see it side by side, explained, in this video. Just use headphones to listen to my explanation.

http://www.aimetis.com/public_downloads/FreescaleBoard/side-by-side-video-delay.mp4

[jack_mao]

Hi,

     Sorry for late, I have setup the environment and reproduce this issue , currently I 'm try do some modification and try on the board, if there is still a gap to the target  I will look for help internal.

[jack_mao]

It seems that the delay time is increasing during the play, we suspect the sync issue, so could you dump the rtsp stream to us for analysis. By the way, what's the encode format? how many reference frame will it support?

[sergeikotlyachk]

I have posted this file for you to try.

http://www.aimetis.com/public_downloads/FreescaleBoard/sample.mp4

You can see my hands in front of the camera at these times:

0:00 - 0:01

0:30 - 0:41

1:00 - 1:10

- When played through VLC 2.0.7 it works fine and shows my hands at proper times.

- I streamed it using VOD mode ( Multiple streaming and Video on demand ) to check if the problem can be reproduced through this method instead of using direct connection to camera. I also ran VLC client alongside connecting to the same instance of streaming VLC. The following happened for Freescale board:

- delay at 0:30 was 4 seconds

- delay at 0:40 was ~5.5 seconds

- at 55 seconds video on Freescale board stopped working while VLC client continued playing at proper times and till the end.

I hope this is good enough for you to fix it. If you don't have  Android app that can play url given to it, you can use MX player if you use Network Stream feature in it and HW acceleration enabled. I tried to use MX player and it seem to work the same way as our test app, except that MX player stopped playing video at 1:02 (Total length of the video is 1:31), may be this stopping of the video is random, I did not do too many tests.

Thanks for your help.