Android Graphic UI with GPU Hardware Acceleration

cancel
Showing results for 
Search instead for 
Did you mean: 

Android Graphic UI with GPU Hardware Acceleration

Android Graphic UI with GPU Hardware Acceleration

Graphics are a big topic in the Android platform, containing java/jni graphic framework and 2d/3d graphic engines (skia, OpenGL-ES, renderscript).

This document describes the general Android graphic stack and UI features on Freescale devices.

1. Android Graphic Stacks

  • All Android 3D apps and games have the following graphic stack:

Picture1.png

  • Android system UI and all Apps UI follow 2D graphic stack as below, the hardware render will accelerate Android 2D UI with GPU HW OpenGL-ES 2.0 to improve the whole UI performance.

New Picture.png

Hardware acceleration can be disabled on i.mx6 in device/fsl/imx6/soc/imx6dq.mk

USE_OPENGL_RENDERER := false

Then rebuild frameworks/base/core/jni, and replace libandroid_runtime.so

  • Surfaceflinger is responsible of all surface layers composition, and  then generate the framebuffer pixmap for display devices. these graphic surface layers are from 2D/3D apps.

New Picture (1).png

  • Hwcomposer is the alternative module of Surfaceflinger with OpenGL-ES. Hwcomposer is used to combine the specific surface layers supported by specific vendor devices. Freescale i.MX6 devices use GPU 2D to combine most surface layers, and the system power can be reduced with GPU 2D instead of GPU 3D. The typical power saving case is video playback.

Hwcomposer with GPU 2D can offload GPU 3D task when running game and benchmarks, it is proved to improve the overall system performance about 20%.

New Picture (2).png

2. Performance measurment

  • Show FPS for Android system performance

For NFS boot you can set “debug.sf.showfps” to 1 in init.freescale.rc (“setprop debug.sf.showfps 1”) and then reboot the system. For SD or EMMC boot, you can issue command “setprop debug.sf.showfps 1” in console, then find system_server thread by top and kill it to reset the system.

New Picture (3).png

  • Graphic benchmarks for 3D capability measurement

Quadrant

Full test benchmark cover CPU, Memory, IO, 2D and 3D

GLBenchmark

http://www.glbenchmark.com/

NenaMark2

https://market.android.com/details?id=se.nena.nenamark2

An3DBench

http://www.androidzoom.com/android_applications/tools/an3dbench_hnog.html

AnTutu

http://www.antutu.com/software.html

3DMark

http://www.futuremark.com/benchmarks/3dmark06/introduction/

  • Browser benchmarks

http://www.webkit.org/perf/sunspider/sunspider.html

http://v8.googlecode.com/svn/data/benchmarks/current/run.html

http://www.craftymind.com/guimark2/

http://www.craftymind.com/factory/guimark/GUIMark_HTML4.html

http://themaninblue.com/writing/perspective/2010/03/22/

3.  Android UI features

  • Dual display with same content

This feature is supported in the default image in Android i.MX 6 release package. In this feature, LVDS panel and HDMI output can be supported simultaneously. It is only enabled when the HDMI TV has been connected with the board.

  • Overscan for TV devices

Some TVs may miss display the contents in overscan area. To avoid the contents in overscan area being lost, the common implement is by underscanning with an adjustable black border and letitng the viewer adjust the width of the black border. The downscan operation is done by surfaceflinger when it does surface composition through HW OpenGL ES. There is no performance impact since all the work is done by GPU HW. Overscan can be configured in display setting in visual mode:

New Picture (4).png

  • 32 bits color depth

32bpp UI can be supported by adding “bpp=32” in uboot as below:

setenv bootargs ‘… video=mxcdi1fb:RGB666,XGA,bpp=32 …’, also can configure it in display setting.

Enable 32bpp frame buffer and application surface buffer will be allocate to RGBA8888 format instead of default RGB565 format, that means more system memory is allocated.

After enabling 32bpp, if some applications still don't have better UI quality, check to see if  there is hard code to request RGB565 format surface (should request RGBA8888 format to get better quality).

Sample code is attached to test for 32bpp (left is on 16bpp, right is on 32bpp)

New Picture (5).png

  • Display Visual Setting

The display setting is the add-on feature in FSL Android release, it is very convenient for end-users to change display property, mostly for the following features:

Dual display enablement

Display color depth setting(16bpp, 32bpp)

Overscan adjustment in horizontal and vertical orientation

New Picture (6).png

4. Issue Diagnosis

  • Application Compatibility

Some Android applications may not run correctly on some Android releases. It may cause application compatibility, so check the application in other platforms.

For example Neocore and Asphalt 5 can run on Eclair, Froyo, and Gingerbread, but will not correctly run on Honeycomb.

  • GPU Compatibility

Some game UIs may not correctly display on our Android release. When encountering this kind of issue, the customer can check whether it is caused by the game using an OpenGL extension which our GPU does not support. They can download another data package (for example not extension data package) to have a check.

  • Others

Enlarge GPU memory if you encounter UI abnormally displaying after running an application for a while. Some applications need Wifi connections, so monitor the console log to see whether there are any error reports.

Labels (3)
Attachments
Comments

very useful, thanks.

Hi, can you please help me with an example code that shows how is this processing done in the Android HAL layer. In particular, referring to hardware/imx/mx6/libcamera/CameraHAL.cpp, I want to take a buffer from the camera preview (in the preview thread) and apply gpu processing like in your example and submit it to display. I know how to take buffer from the capture preview queue and apply processing but how do you submit it to display? In CameraHAL.cpp I couldn't find references to the display buffers.

Hi Dragan,

In order to understand how the captured buffer is shown in preview window, I will give you some code flows as below:

1. How the preview window surface is registered to camera HAL

packages/apps/Camera//src/com/android/camera/Camera.java:

  startPreview --> setPreviewDisplay(mSurfaceHolder);

frameworks/base/services/camera/libcameraservice/CameraService.cpp:

  CameraService::Client::setPreviewDisplay --> CameraService::Client::setPreviewWindow -->CameraHardwareInterface(mHardware->setPreviewWindow)

frameworks/base/services/camera/libcameraservice/CameraHardwareInterface.h:

setPreviewWindow-->CameraModule(mDevice->ops->set_preview_window = camera_set_preview_window)

hardware/imx/mx6/libcamera/CameraModule.cpp:

camera_set_preview_window --> cameraHal(gCameraHals[fsl_dev->cameraid]->setPreviewWindow)

hardware/imx/mx6/libcamera/CameraHal.cpp:setPreviewWindow

mNativeWindow = window://the preview window is registered in CameraHal

2. How the captured buffer is submitted to preview window

hardware/imx/mx6/libcamera/CameraHal.cpp:

first of all, allocate native buffer form preview window:

mNativeWindow->dequeue_buffer in allocateBuffersFromNativeWindow

then prepare captured buffer to native buffer ( you know the processing, ignore the sample code)

finally submit the capture buffer to preview window:

mNativeWindow->enqueue_buffer in previewshowFrameThread

frameworks/base/services/camera/libcameraservice/CameraHardwareInterface.h:

__enqueue_buffer will call ANativeWindow::queueBuffer

then the subsequential flow:

  frameworks/base/libs/gui/SurfaceTextureClient.cpp

hook_queueBuffer(ANativeWindow::queueBuffer = hook_queueBuffer) -->SurfaceTextureClient::queueBuffer --> SurfaceTexture:;queueBuffer(mSurfaceTexture->queueBuffer)

frameworks/base/libs/gui/SurfaceTexture.cpp

SurfaceTexture::queueBuffer-->Layer::onFrameAvailable(listener->onFrameAvailable)

frameworks/base/services/surfaceflinger/layer.cpp

void Layer::onFirstRef()

{

    LayerBaseClient::onFirstRef();

    struct FrameQueuedListener : public SurfaceTexture::FrameAvailableListener {

        FrameQueuedListener(Layer* layer) : mLayer(layer) { }

    private:

        wp<Layer> mLayer;

        virtual void onFrameAvailable() {

            sp<Layer> that(mLayer.promote());

            if (that != 0) {

                that->onFrameQueued();

            }

        }

    };

so the camera preview window is a normal surface created from Surfaceflinger, you can refer to "3D stack with Surfaceflinger" and "Compostion with Hwcomposer" to know how it will be shown in display framebuffer.

Hope you can get whatever in submitting preview buffer in CameraHal, let me know if you have more questions.

Hi Xianzhong, your guidance was of great help. I now understand that capture display works by

ping-pong capture buffers between capture and display threads. I outlined bellow high-level

code for better understanding of the implementation details.

I plan to insert processing in the capture thread and before submitting to display thread.

I will now experiment with this setup to check how it works.

CameraHal::captureframeThread() {

  mCaptureThreadQueue.waitMessage(); // received signal that CameraHal::previewshowFrameThread() is ready

  mCaptureDevice->DevDequeue(&bufIndex); // get next display-ready buffer mCaptureBuffers[bufIndex]

  // ... do processing on mCaptureBuffers[bufIndex];

  mPreviewThreadQueue.postMessage(new CMessage(CMESSAGE_TYPE_NORMAL, bufIndex)); // signal CameraHal::previewshowFrameThread() that mCaptureBuffers[bufIndex] is ready

}

CameraHal::previewshowFrameThread() {

  msg = mPreviewThreadQueue.waitMessage(); // received signal that mCaptureBuffers[bufIndex] is ready (bufIndex is in msg->arg0)

  pInBuf = &mCaptureBuffers[msg->arg0]; // get hold of ready buffer

  mNativeWindow->enqueue_buffer(mNativeWindow, (buffer_handle_t * )pInBuf->native_buf); // submit ready buffer to display

  // remove already displayed buffer from display and find its buf_index within mCaptureBuffers[] array

  mNativeWindow->dequeue_buffer(mNativeWindow, &buf_h, &stride);

  SearchBuffer((void *)buf_h, &buf_index);

                  

  mCaptureDevice->DevQueue(buf_index); // Return buffer mCaptureBuffers[buf_index] to Capture Device

  // signal CameraHal::captureframeThread() that CameraHal::previewshowFrameThread() is ready

  mCaptureThreadQueue.postMessage(new CMessage(CMESSAGE_TYPE_NORMAL, buf_index));

}

This approach works. I was able to get to the buffer, do the processing in s/w and submit to display.

Hi Dragan, you implementation is ok.

Hi Xianzhong, before I start using GPU in capture buffer processing I'd like as an intermediate step to use IPU for simple copy operation. I was thinking to use 1:1 scale as a copy. Can you point me to an example IPU code for scaling or color space conversion from Android HAL, i.e. how to setup IPU task from Android HAL and get result. Thanks.

Hi Dragan, are you thinking to use IPU to copy a buffer? as I know, IPU cannot support data copy without scaling or conversion.

Hi Xianzhong, I was thinking that 1:1 scale will effectively work as data copy. If you have any IPU example that works from inside Android HAL I could give it a shot.

Hi Dragan, please refer to the IPU usage in resizeToSecFrameBuffer function in hardware/imx/mx5x/libgralloc/framebuffer.cpp

Hi Xianzhong, I got into some problems getting IPUv3 to work. If you could, can you please respond to this thread that I opened specifically on this topic:

https://community.freescale.com/thread/306462

Xianzhong Li, I want to try simple_draw in imx6 imx-android-13.4.1.04 build environment on sabresd_6dq board. I copied simple_draw to ~/myandroid and after starting build I get the following error:

target arm C++: simple_draw/draw <= frameworks/simple_draw/simple_draw.cpp

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp: In function 'void eglOpen()':

frameworks/simple_draw/simple_draw.cpp:123: error: 'class android::SurfaceComposerClient' has no member named 'openTransaction'

frameworks/simple_draw/simple_draw.cpp:125: error: 'class android::SurfaceComposerClient' has no member named 'closeTransaction'

frameworks/simple_draw/simple_draw.cpp:145: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:152: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp:160: warning: deprecated conversion from string constant to 'char*'

frameworks/simple_draw/simple_draw.cpp: In function 'int main(int, char**)':

frameworks/simple_draw/simple_draw.cpp:427: warning: deprecated conversion from string constant to 'char*'

make: *** [out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o] Error 1

I had to change the following line in simple_draw.cpp to get it past compilation stage:

    //session->openTransaction();

    session->openGlobalTransaction();

    control->setLayer(0x40000000);

    //session->closeTransaction();

    session->closeGlobalTransaction();

However, now I get the following link error. Can you please help:

target Executable: simple_draw/draw (out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/LINKED/simple_draw/draw)

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:114: error: undefined reference to 'android::SurfaceComposerClient::SurfaceComposerClient()'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:117: error: undefined reference to 'android::SurfaceComposerClient::getDisplayInfo(int, android::DisplayInfo*)'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:122: error: undefined reference to 'android::SurfaceComposerClient::createSurface(int, unsigned int, unsigned int, int, unsigned int)'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:124: error: undefined reference to 'android::SurfaceComposerClient::openGlobalTransaction()'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:125: error: undefined reference to 'android::SurfaceControl::setLayer(int)'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:127: error: undefined reference to 'android::SurfaceComposerClient::closeGlobalTransaction(bool)'

prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:129: error: undefined reference to 'android::SurfaceControl::getSurface() const'

collect2: ld returned 1 exit status

make: *** [out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/LINKED/simple_draw/draw] Error 1

Hi Dragan, simple_draw_ics.tgz is attached, please find it.

Xianzhong Li, it works now. Thanks a lot this is really a great help!!!


Will this version work on jelly bean too?

Dragan, it may not work on jelly bean, but you can modify the code to create Android surface referring to the example,

frameworks/native/services/surfaceflinger/tests/surface

Thank you, much appreciated. At this point I can start using GPU (in particular pixel shader) in further processing my captured camera video. I already have preview working from the gray scale camera. I capture each frame as 8-bit generic data and use IPU unit to convert 8-bit gray scale raw data into ARGB8888. To do that I use custom color space conversion matrix and YUV->RGB conversion IPU task. All capture buffers including one from the output of the IPU are allocated from the native window. At this point, I submit output from IPU to the preview window. I'd like to insert some GPU shading in between i.e. after IPU and before submitting to preview window. My system is imx6 android ics 13.4.1.04. Can you give me some ideas how this could be done?

Hi Dragan, I'm not a Android Camera guy, you can create a new question and send me the link, I can help you find the proper guy to answer your question.

Thanks Xianzhong. I created a new question at the following link:

https://community.freescale.com/thread/309817

Hi,

In the above Li say:

2. How the captured buffer is submitted to preview window

hardware/imx/mx6/libcamera/CameraHal.cpp:

first of all, allocate native buffer form preview window:

mNativeWindow->dequeue_buffer in allocateBuffersFromNativeWindow

On our platform, when we do the dequeue, the handle->phys, handle->base values make no sense (i.e. point to a strange memory area... Not the framebuffer or anything in RAM. I thought that phys was a physical pointer address. (and when I start the camera app, it will stall the device since the IPU will now access this wierd memory area).

Our platform is otherwise up and running.

Anyone have an idea of how to handle the dequeued data?

Hi Mads,  does video playback work on your platform? vpu also uses the "handle->phys", which is allocated from GPU driver.


That was a VERY good question :smileyhappy:  It was related to the allocation!

Btw: Do you have any idea why the preview window in the camera app is quite small. I seem to have a 202x269 image (V/camera  ( 3112): surfaceChanged. w=202. h=269). The preview size should be 640x480.

I suspect a scaling of the preview size displayed, but if I change the preview size it doesn't scale. Any ideas? (our screen is 800x480)


Android camera preview window is re-layout in Camera application, the scaling should be performed for the original camera image.

Hi,

Its really great article.

How can I enable software rendering in lollipop?

I dont want to use the hardware rendering.

Is it possible to use the software rendering in lollipop?

what are the changes i need to do to make it work?

LiuXuegang​, could you please help to enable software rendering for lollipop?

Hi Xuegang Liu,

I want to use software rendering with lollipop  android.

I am new to this and i want to know how to enable this and what are the things to take care of.

Can u please help me in this.

i am using L5.1.1_2.1.0-ga Android release. when i set USE_OPENGL_RENDERER := false in imx6.mk, i got compile error attached. please help.

frameworks/base/core/jni/android_view_Surface.cpp:383: error: undefined reference to 'android::uirenderer::AnimationContext::AnimationContext(android::uirenderer::renderthread::TimeLord&)'

frameworks/base/core/jni/android_view_Surface.cpp:410: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::syncAndDrawFrame(long long, long long, float)'

frameworks/base/core/jni/android_view_Surface.cpp:404: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::updateSurface(android::sp<ANativeWindow> const&)'

frameworks/base/core/jni/android_view_Surface.cpp:391: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::RenderProxy(bool, android::uirenderer::RenderNode*, android::uirenderer::IContextFactory*)'

frameworks/base/core/jni/android_view_Surface.cpp:392: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::loadSystemProperties()'

frameworks/base/core/jni/android_view_Surface.cpp:393: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::setSwapBehavior(android::uirenderer::renderthread::SwapBehavior)'

frameworks/base/core/jni/android_view_Surface.cpp:394: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::initialize(android::sp<ANativeWindow> const&)'

frameworks/base/core/jni/android_view_Surface.cpp:397: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::setup(int, int, android::uirenderer::Vector3 const&, float, unsigned char, unsigned char)'

frameworks/base/core/jni/android/graphics/Paint.cpp:809: error: undefined reference to 'android::uirenderer::Blur::convertRadiusToSigma(float)'

collect2: error: ld returned 1 exit status

make: *** [out/target/product/sabresd_7d/obj/SHARED_LIBRARIES/libandroid_runtime_intermediates/LINKED/libandroid_runtime.so] Error 1

Version history
Revision #:
1 of 1
Last update:
‎10-12-2012 02:33 AM
Updated by: