topic Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3 in i.MX Processors

How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yuuki — Wed, 06 Jan 2016 02:24:43 GMT

Dear all,

We run Android4.4.3 with MX6Q-SABRE-SDB board.

(This Android4.4.3 is IMX6_KK443_200_ANDROID_SOURCE_BSP of NXP.)

And, Extended Feature Packages(omx player package) are implemented.

On this system, We want to use Android MediaCodec API Demo.

https://github.com/vecio/MediaCodecDemo

This MediaCodec seems to use only one InputBuffer for the decode.

Can we increase the number of inputBuffer?

If it is possible, would you tell me the method (procedure)?

In the following cord.

==========================================================

ByteBuffer[] inputBuffers = decoder.getInputBuffers();

ByteBuffer[] outputBuffers = decoder.getOutputBuffers();

BufferInfo info = new BufferInfo();

boolean isEOS = false;

long startMs = System.currentTimeMillis();

while (!Thread.interrupted()) {

if (!isEOS) {

int inIndex = decoder.dequeueInputBuffer(100000);

==========================================================

When the dequeueInputBuffer function was called, value "0" is returned to Index, and this processing takes an about 20msec.

It seems to be caused by the fact that this inputBuffer is SingleBuffer.

We want to shorten the processing time by increasing InputBuffer.

May I have advice?

Best Regards,

Yuuki

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yuuki — Wed, 06 Jan 2016 09:55:16 GMT

Dear all,

We found the point that defined the number of InputBuffer.

external/fsl_imx_omx/OpenMAXIL/src/component/vpu_dec_v2/VpuDecComponent.cpp

===============================================================

#define DEFAULT_BUF_IN_CNT 0x1

#define DEFAULT_BUF_IN_SIZE (1024*1024)

#define DEFAULT_BUF_OUT_POST_ZEROCNT (0)

#define DEFAULT_BUF_OUT_POST_CNT (5) //5 will bring better performance than 4

#define DEFAULT_BUF_OUT_DEC_CNT (3)

===============================================================

We tried the change of the value of DEFAULT_BUF_IN_CNT.

However, it was not improved.

Our problem is that 1080p60fps video contents are played back at about 50fps if MediaCodec was used.

Which part of MediaCodec should we confirm?

May I have advice?

Best Regards,

Yuuki

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Mon, 18 Jan 2016 06:54:53 GMT

Hi,

Do you try to use "external\fsl_vpu_omx" codec package instead of "external/fsl_imx_omx" package ?

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Sat, 13 Feb 2016 11:34:38 GMT

HI Everyone,

We are facing the similar issue and see that the input buffers are allocates using the useBuffer

Cross Reference: /frameworks/av/media/libstagefright/ACodec.cpp which is not allocating on hardware side.

Is someone already have achieved the solution, Can someone help me on increasing the Input Buffers.

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

jamesbone — Mon, 15 Feb 2016 15:30:21 GMT

Hello Gurtaj,

Which package codec are you using? AS you can see from the thread we are recommending to use the "external\fsl_vpu_omx" which platform are you using and what version of BSP?

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Mon, 15 Feb 2016 17:40:00 GMT

Hi James,

We are using Android L5.1.1_2.1.0 GA. We have Android Multimedia Extended Codec as a part of BSP.

I was able to increase the buffer from DEFAULT_BUF_IN_CNT in VpuDecComponent.cpp.

Currently we are facing the issue with 60FPS video playback using Mediacodec API. The same used to work in Android JB4.2.2_1.0.0-ga.

We are trying to isolate the problem and in the process we see decoding or rendering could be slow.

Can you help me understanding the mechanism of how decoded data is rendered onto the surfaceflinger after MediaCodec.releaseOutputBuffer(true) is called.

Also, can you speculate on what exactly was changed in between JB4.2.2_1.0.0-ga. and next version with reference to rendering.

Thanks & Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Thu, 18 Feb 2016 03:54:00 GMT

Hi,

This is about OpenMax VPU decoder and surface render component performace , have created JIRA:

http://sw-jira.freescale.net/browse/MMFMWK-7080

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Thu, 18 Feb 2016 06:55:03 GMT

Hi,

I am not able to access the link being in company network. So can you shed some light on how the problem can be solved.

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Thu, 18 Feb 2016 07:38:47 GMT

Hi gurtajs4@gmail.com,

Do you mean you are using Mediacodec API which Android SDK expose too ?

Actually, Android use StageFright as default Multimedia framework. Fsl's fsl_imx_omx package use "OMXPlayer" as Multimedia framework.

So, do you test use "external\fsl_vpu_omx" package to test ?

That's mean don't apply Extended Feature Packages(omx player package) . That's "HAVE_FSL_IMX_CODEC" is "false".

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Thu, 18 Feb 2016 08:21:31 GMT

Hi,

I am using MediaCodec API with BSP Android L5.1.1_2.1.0 GA.

We know that the OMXPlayer uses the fsl_imx_omx. Again the striped version of "fsl_imx_omx" is "fsl_vpu_omx".

So in Android, Freescale uses fsl_vpu_omx to interface hardware codec with libstagefright.

Speaking of which, MediaCodec uses fsl_vpu_omx for decoding.

If that correct then what about the Rendering in MediaCodec ??

Does it uses fsl_vpu_omx as a renderer?? or Is it differently linked with SurefaceFlinger.?

See, the Green Path works with no issue, but the Mediacodec path are having issue with decoding + rendering the frame at 16.6 fps for 1080p60fps video.

I wanted to isolate the issue. Either its in decoder or renderer.

Can you suggest some way here?

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Thu, 18 Feb 2016 10:32:01 GMT

Hi Gurtaj Singh,

I agree with you, should distinguish the OpenMax VPU component issue or surface render display issue.

you said "the Green Path works with no issue" , is that mean if only use VPU component(no surface render handlle), the vpu decode speed can reached 60fps ? if yes, the issue happend on rendering mechanism.

In "fsl_imx_omx" package, OMX_player use OpenMAX component "OMX.Freescale.std.video_render.surface.sw-based" as render component.

In "fsl_vpu_omx" package, stagefright use Android libstagefright default render method "AwesomeRenderer".

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Thu, 18 Feb 2016 10:47:15 GMT

Hi Yifang Guo,yifangguo-b43107

Yes, if only VPU Component is used (no surface render handlle), the vpu decode speed can reached upto 65-75fps. This is with 1080p video 25MB bitrate and 60 FPS. But since we are using Android libstagefright default render, the overall output fps achieve ( both including decoding and rendering ) is less than 60. It is around 50 FPS

I am really doubting the Android libstagefright default render method. Can we verify this and isolate or narrow down the issue.

Thanks & Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Thu, 18 Feb 2016 10:51:06 GMT

Do you test to use "fsl_vpu_omx" package, delete "fsl_imx_omx" package folder ?

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Thu, 18 Feb 2016 10:56:52 GMT

Hi,

Ok I see the point.

I don't have fsl_imx_omx, we only have fsl_vpu_omx

The problem is I am using MediaCodec API and MediaCodec uses VPU for decoding and Android libstagefright default render method.

Decoding is fine, Rendering is slow and hence the overall video playback with MediaCodec is slow ( 1 Minute video takes 1.5 minutes to complete).

I still want to use MedaiCodec API's . There is no other option to decode h264 elementary stream in JAVA.

How can I speed up the rendering.

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Fri, 19 Feb 2016 02:18:07 GMT

Hi Gurtaj Singh,

"fsl_vpu_omx " package is only a OpenMax plugin worked for Android Multimedia stagefright.

Do you confirm that you are using fsl_vpu_omx to try , not fsl_imx_omx at present ?

if yes, I suggest to apply " Extended Feature Packages(omx player package)-fsl_imx_omx" to test MediaCodec API demo. Because fsl_imx_omx package implement another Multimedia framework "omx_player" instead of "stagefright" .

So, I mean you can test your MediaCodec API demo on two different Multimedia frameworks :

1) On Android default Multimedia framework---stagefright, it utilizes OpenMax VPU component plugin in "fsl_vpu_omx" package,

2) Or FSL Multimedia framework---omx_player, it is implemented by FSL instead of "stagefright" in "fsl_imx_omx" package. In "fsl_imx_omx" package, OMX_player use OpenMAX component "OMX.Freescale.std.video_render.surface.sw-based" as render component.

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Fri, 19 Feb 2016 02:35:45 GMT

Hi,

Sir, If I apply fsl_imx_omx patches, then the changes effect only mediaplayerservice and the Gallery will by default start using OMXPlayer instead of Awesome/NuPlayer.

But as you can see in the picture above, we wanted to use MediaCodec API to decode raw h264.

And at no point from the above diagram I see MediaCodec using Freescale renderer, it always uses Awesome renderer and which could be the issue.

Is my understanding correct???

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Fri, 19 Feb 2016 03:35:05 GMT

If apply fsl_imx_omx patches, then the changes effect only mediaplayerservice and the Gallery will by default start using OMXPlayer instead of Awesome/NuPlayer.

[A] Yes , it's right.

Can you confirm that your "MediaCodec API" can call in to "OMXPlayer" flow or not ?

As "i.MX_Android_Extended_Codec_Release_Notes.pdf" spec,

Android Codec spec of H.264 1080p decoder can only guarantee the MAX framerate 30fps . Don't guarantee more than 30fps in Android System expect an baremetal(no os) demo.

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Fri, 19 Feb 2016 05:50:09 GMT

Hi yifangguo-b43107

I am using the Freescale Android L5.1.1_2.1.0 GA, for our development.

""Can you confirm that your "MediaCodec API" can call in to "OMXPlayer" flow or not ?""

[Gurtaj] Can you verify as my knowledge is scare with respect to OpenMAX.

"Android Codec spec of H.264 1080p decoder can only guarantee the MAX framerate 30fps . Don't guarantee more than 30fps in Android System expect an baremetal(no os) demo."

[Gurtaj] I know that the spec says no guarantee for more than 30FPS, Can you answer the below questions

A. Have a look here Slow H264 1080P@60fps Decoding on Android Lollipop 5.0.2 SabreSD .

B. How can JB was able to decode and render 60FPS video and Lollipop or Kitkat don't.

C. Why there is no effect in the OMXPlayer or GMPlayer ( the green path in above sample) and they are able to decode the given video, But only the MediaCodec Path got effected?

Looking forward to a great discussion and logical answers to above question.

Regards,

Gurtaj

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

yifangguo-b4310 — Fri, 19 Feb 2016 06:43:04 GMT

Hi,

OMXPlayer (GMPlayer) is implemented by FSL as Media player framework. It use openMAX VPU decoder component, and re-construct render method in "OMX.Freescale.std.video_render.surface.sw-based" component although send to surfaceflinger too. So, OMXPlayer 's play frame rate maybe can reach to about 60fps under some low bitrate use case.

But Media Codec API use stagefright multimedia framework , can you mask the stagefright render code, don't display video frame, just only use VPU component decode( or save every decoded frame in FS by write an opened file), then test the frame rate. If it can reach to 60fsp, so the issue is in Android stagefright render part.

Re: How to increase the number of inputBuffer for MediaCodec on Android4.4.3

gurtajs4 — Fri, 19 Feb 2016 07:02:07 GMT

Hi yifangguo-b43107

Thanks for your explanation and keeping patience with me so far.

Yes, We wanted to use MediaCodec to only Decode (just only use VPU component decode and no rendering). Later save the decoded YUV frames to a file.

As you can see from the starting of the thread, I am asking for help to do this approach

Can you help me with best way and some test code to do

1. The YUV to RGB color conversion.

2. Render the converted image to framebuffer ( Zero Copy Approach ??).

If, i have a test code to perform the two methods in the minimum time, let say 2-3 millisecond. Then we will be able to narrow down and isolate the issue.

Regards,

Gurtaj