further information about stagefright

[javierrodriguez]

Hi,

I have been working with the stagefright interface for 4 months. After some hard work, I have finally adapted the ffmpeg's stagefright codec to works in android. It works very well It is able to decode 1080p video in h264 instantaneously ( < 1 ms per frame). As far as I know, the codec returns the decoded video in YUV420P but, in android, the surface works either RGB565 or RGB8888 (aka RGBA or RGB32). Theoretically, the colorspace conversion should be done in a really few miliseconds but the reality is that it expends around 20ms per frame so I get a delaying issues.

I am working with a modified version the linphone for android. The  app creates a SurfaceView that is rendered with OpenGL ES (I know that the app loads the library and the extensions) but it does not work properly.

I have been trying to pass the NativeWindow reference of the surfaceView to the decoder constructor as it indicates the powerpoint of this web but when I pass the reference, the program crashes throwing either SIGSEGV or SIGBUS signals.

please, could you give me further information of how to handle the data extracted from the VPU.

Thank you very much.

[andre_silva · Accepted solution]

Hi Javier,

There are some optimizations that we can make when using video as textures or when you need you texture to be applied very fast, in this case we can write data directly to the GPU buffers, avoiding any slow painting functions like glTexture2D.

As you are using android you will need deal with surfaceflinger in order do do this direct buffer access in your side.

regards,

Andre

[javierrodriguez]

Hi Andre,

First of all, thank you very much for your answer.

Currently, I'm working with an application in Android that "creates" a SurfaceView in Java however I am developing the native part of the application and, sincerelly, I don't know how to deal with the surfaceflinger or another GUI java element. I've achieved to open the framebuffer and the IPU at low-level model using the ioctl interface but It didn't work very well... As far as I know, there is no support to OpenGLES neither in linux nor android, is this correct?

However, I'm going crazy because the gallery3d application released by freescale is able to reproduce completely HD video smoothly and I can't... Currently, I've managed to decode H264 video through libstagefright via ffmpeg plugin but I can't do the colorspace conversion efficiently...

Please, help me...

Thanks!

[andre_silva]

Hi Javier,

we do support OpenGL ES 1.1, 2.0, OpenVG, OpenCL and in second half of this year we are going to Support WebGL as well.

Unfortunately in android, you will only reach the performance you want using the surfaceflinger, which provides you direct access to the GPU buffers. I´m going to look for some code and will share with you soon as possible, then you can use it as a guide to your development.

regards,

Andre

[javierrodriguez]

Hi Andre,

Thank you very much for your time!

[andre_silva]

Hi Javier,

please, find the code attached, this is a demo that it is supposed to show a texture in the display, but unfortunately you will not see it because I the display part is broken and as I used this code long time ago I didn´t have time to fix it, the customer I helped only needed the surfaceflinger part and it is working. So, you will need to do some minor fixes in order to make it works correctly (the display part only).

ps: it looks like java folder but the code is in C, find it in jini folder. Also, it is for i.MX53 board, that is all I have.

regards,

Andre

[javierrodriguez]

Hi Andre,

I've been studying the code you provided me and I guess that, finally, I'm seeing the day light. I have a question about that...

Firstly, you define the variable checkimage as follows:

static GLubyte checkImage[CHECK_IMAGE_HEIGHT][CHECK_IMAGE_WIDTH][4];

Why did you define the value 4? I have the image defined in 3 planes (YUV420), how can I do to handle each plane? Is it as easier as do a memcpy for each plane to this variable??? It's sounds like if the value 4 is defined por RGB and the gamma values... what should I do to adapt the code?

Secondly, is the instruction "memcpy(vaddr, checkImage, sizeof(checkImage));" performing the copy within the graphic buffer?

Sorry for this noob questions but I'm newer at Android graphic system.

Thank you very much!

[andre_silva]

Hi Javier,

the value 4 is defined for RGBA image, this format is defined in:

control = session->createSurface(getpid(), 0, dinfo.w, dinfo.h, android::PIXEL_FORMAT_RGBA_8888);

in this case, take a look and see if there is any YUV supported format.

the pointer ADDR contains the address of the native buffer, in this case, the gpu buffer, so, when you memcpy data to this address you are directly writing to the gpu buffers.

regards,

Andre

[javierrodriguez]

Hi Andre,

I'm working with the source code but it doesn't work. When I call to the engine_init_display( ); it fails and It doesn't display "initCheck = OK" so I can deduce that something goes wrong. Going deeper into the AOSP, I've found in the file SurfaceComposerClient that, by default, the client is no initialized when you instantiated this class...

• SurfaceComposerClient::SurfaceComposerClient()
•     : mStatus(NO_INIT), mComposer(Composer::getInstance())
• {
• }

It will self-initialize when "someone or something" call to the onFirstRef( ) callback...

• void SurfaceComposerClient::onFirstRef() {
•     sp<ISurfaceComposer> sm(getComposerService());
•     if (sm != 0) {
•         sp<ISurfaceComposerClient> conn = sm->createConnection();
•         if (conn != 0) {
•             mClient = conn;
•             mStatus = NO_ERROR;
•         }
•     }
• }

This callback is private and is called by the superclass RefBase and I guess It never happens... I need to solve this in order to continue... I'm a newbie working with the native code in android and I don't know what "sp< > class" means

Thank you very much!

Javier.

[andre_silva]

Hi Javier,

I only worked with android at the time I wrote this code, and it was long time ago. Unfortunately I didn`t touch android anymore since that. I believe you are close to the solution, keep the hard work and you will find it.

regards,

andre

[javierrodriguez]

Hi Andre,

I'm not able to understand your code because I don't have any idea about OpenGL but I'm going to try to use it. I wish there was a easier way to do what I need...

Any clue will be appreciated.

Thank you very much for your time.

Javier.

[andre_silva]

There was if you were using Linux, since you are using Android, this is the way to do that. =(

regards,

Andre

[javierrodriguez]

Hi Andre,

Yesterday I was a little ill and I guess I wasn't thinking clearly, sorry. So, I'm going to tell you what I have until now...

When the stagefright ends, it returns to me an array with the frame decoded in YUV420 format. I worked out the total size of a frame in YUV420@1080p and its size is 3110400 bytes is it correct?. Until now, I've managed to display the frames correctly but it takes so much time that it starts to add a display delay until a queue overflow, then the app crashes... hahaha.

So, what do I want to do? "Easy", I only need to do the colorspace conversion using the IPU and display it. I guess that the source code that you give me yesterday does what I need but I'm really noob in openGL framework... Could it be as easy as doing a memcpy from my array to a magic direction that It was able to do all that I need automatically?

I guess I haven't understand the SurfaceFlinger concept.

UPDATE: I've found the part of the application that I'm working with where it's defined the OpenGL operations... I've paid attention to this:

GL_OPERATION(glActiveTexture(GL_TEXTURE0))

    GL_OPERATION(glBindTexture(GL_TEXTURE_2D, gldisp->textures[gldisp->texture_index][type][Y]))

    GL_OPERATION(glUniform1i(gldisp->uniforms[UNIFORM_TEXTURE_Y], 0))

GL_OPERATION(glActiveTexture(GL_TEXTURE1))

    GL_OPERATION(glBindTexture(GL_TEXTURE_2D, gldisp->textures[gldisp->texture_index][type][U]))

    GL_OPERATION(glUniform1i(gldisp->uniforms[UNIFORM_TEXTURE_U], 1))

GL_OPERATION(glActiveTexture(GL_TEXTURE2))

    GL_OPERATION(glBindTexture(GL_TEXTURE_2D, gldisp->textures[gldisp->texture_index][type][V]))

    GL_OPERATION(glUniform1i(gldisp->uniforms[UNIFORM_TEXTURE_V], 2))

    GL_OPERATION(glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, squareVertices))

    GL_OPERATION(glEnableVertexAttribArray(ATTRIB_VERTEX))

    GL_OPERATION(glVertexAttribPointer(ATTRIB_UV, 2, GL_FLOAT, 1, 0, squareUvs))

    GL_OPERATION(glEnableVertexAttribArray(ATTRIB_UV))

    GL_OPERATION(glDrawArrays(GL_TRIANGLE_STRIP, 0, 4))

What do you think about this?

Thank you!

Regards,

Javi.

[andre_silva]

Hi Javier,

it could work, but the OpenGL functions are a bit slow, so if want to display 30fps video you will need to write directly to the GPU buffers, I touched this surfaceflinger code long time ago, but I´m pretty sure if you take a look in the code and google about it you will be able to put it to work.

the surfaceflinger is the way to have access to the gpu buffers in Android.

regards,

Andre

[javierrodriguez]

Hi Andre,

It works indeed, but too slow...

And, if I would manage to write into the gpu buffers, could I solve the delay problem???. should I need to interact with the IPU too? (through ioctls).

I'm going to try to understand the code you provide me... Could I contact you by email? I feel we could keep a more fluent conversation...

Thank you very much!

[andre_silva]

Hi Javier,

yes, using the surfaceflinger approach will solve your problem. The poor performance you are seeing is due the common OGLES drawing functions.

regards,

Andre