Android Graphic UI with GPU Hardware Acceleration Graphics are a big topic in the Android platform, containing java/jni graphic framework and 2d/3d graphic engines (skia, OpenGL-ES, renderscript). This document describes the general Android graphic stack and UI features on Freescale devices. 1. Android Graphic Stacks All Android 3D apps and games have the following graphic stack: Android system UI and all Apps UI follow 2D graphic stack as below, the hardware render will accelerate Android 2D UI with GPU HW OpenGL-ES 2.0 to improve the whole UI performance. Hardware acceleration can be disabled on i.mx6 in device/fsl/imx6/soc/imx6dq.mk USE_OPENGL_RENDERER := false Then rebuild frameworks/base/core/jni, and replace libandroid_runtime.so Surfaceflinger is responsible of all surface layers composition, and then generate the framebuffer pixmap for display devices. these graphic surface layers are from 2D/3D apps. Hwcomposer is the alternative module of Surfaceflinger with OpenGL-ES. Hwcomposer is used to combine the specific surface layers supported by specific vendor devices. Freescale i.MX6 devices use GPU 2D to combine most surface layers, and the system power can be reduced with GPU 2D instead of GPU 3D. The typical power saving case is video playback. Hwcomposer with GPU 2D can offload GPU 3D task when running game and benchmarks, it is proved to improve the overall system performance about 20%. 2. Performance measurment Show FPS for Android system performance For NFS boot you can set “debug.sf.showfps” to 1 in init.freescale.rc (“setprop debug.sf.showfps 1”) and then reboot the system. For SD or EMMC boot, you can issue command “setprop debug.sf.showfps 1” in console, then find system_server thread by top and kill it to reset the system. Graphic benchmarks for 3D capability measurement Quadrant Full test benchmark cover CPU, Memory, IO, 2D and 3D GLBenchmark http://www.glbenchmark.com/ NenaMark2 https://market.android.com/details?id=se.nena.nenamark2 An3DBench http://www.androidzoom.com/android_applications/tools/an3dbench_hnog.html AnTutu http://www.antutu.com/software.html 3DMark http://www.futuremark.com/benchmarks/3dmark06/introduction/ Browser benchmarks http://www.webkit.org/perf/sunspider/sunspider.html http://v8.googlecode.com/svn/data/benchmarks/current/run.html http://www.craftymind.com/guimark2/ http://www.craftymind.com/factory/guimark/GUIMark_HTML4.html http://themaninblue.com/writing/perspective/2010/03/22/ 3. Android UI features Dual display with same content This feature is supported in the default image in Android i.MX 6 release package. In this feature, LVDS panel and HDMI output can be supported simultaneously. It is only enabled when the HDMI TV has been connected with the board. Overscan for TV devices Some TVs may miss display the contents in overscan area. To avoid the contents in overscan area being lost, the common implement is by underscanning with an adjustable black border and letitng the viewer adjust the width of the black border. The downscan operation is done by surfaceflinger when it does surface composition through HW OpenGL ES. There is no performance impact since all the work is done by GPU HW. Overscan can be configured in display setting in visual mode: 32 bits color depth 32bpp UI can be supported by adding “bpp=32” in uboot as below: setenv bootargs ‘… video=mxcdi1fb:RGB666,XGA,bpp=32 …’, also can configure it in display setting. Enable 32bpp frame buffer and application surface buffer will be allocate to RGBA8888 format instead of default RGB565 format, that means more system memory is allocated. After enabling 32bpp, if some applications still don't have better UI quality, check to see if there is hard code to request RGB565 format surface (should request RGBA8888 format to get better quality). Sample code is attached to test for 32bpp (left is on 16bpp, right is on 32bpp) Display Visual Setting The display setting is the add-on feature in FSL Android release, it is very convenient for end-users to change display property, mostly for the following features: Dual display enablement Display color depth setting(16bpp, 32bpp) Overscan adjustment in horizontal and vertical orientation 4. Issue Diagnosis Application Compatibility Some Android applications may not run correctly on some Android releases. It may cause application compatibility, so check the application in other platforms. For example Neocore and Asphalt 5 can run on Eclair, Froyo, and Gingerbread, but will not correctly run on Honeycomb. GPU Compatibility Some game UIs may not correctly display on our Android release. When encountering this kind of issue, the customer can check whether it is caused by the game using an OpenGL extension which our GPU does not support. They can download another data package (for example not extension data package) to have a check. Others Enlarge GPU memory if you encounter UI abnormally displaying after running an application for a while. Some applications need Wifi connections, so monitor the console log to see whether there are any error reports. Android Graphics & Display i.MX6_All Re: Android Graphic UI with GPU Hardware Acceleration i am using L5.1.1_2.1.0-ga Android release. when i set USE_OPENGL_RENDERER := false in imx6.mk, i got compile error attached. please help. frameworks/base/core/jni/android_view_Surface.cpp:383: error: undefined reference to 'android::uirenderer::AnimationContext::AnimationContext(android::uirenderer::renderthread::TimeLord&)' frameworks/base/core/jni/android_view_Surface.cpp:410: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::syncAndDrawFrame(long long, long long, float)' frameworks/base/core/jni/android_view_Surface.cpp:404: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::updateSurface(android::sp const&)' frameworks/base/core/jni/android_view_Surface.cpp:391: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::RenderProxy(bool, android::uirenderer::RenderNode*, android::uirenderer::IContextFactory*)' frameworks/base/core/jni/android_view_Surface.cpp:392: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::loadSystemProperties()' frameworks/base/core/jni/android_view_Surface.cpp:393: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::setSwapBehavior(android::uirenderer::renderthread::SwapBehavior)' frameworks/base/core/jni/android_view_Surface.cpp:394: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::initialize(android::sp const&)' frameworks/base/core/jni/android_view_Surface.cpp:397: error: undefined reference to 'android::uirenderer::renderthread::RenderProxy::setup(int, int, android::uirenderer::Vector3 const&, float, unsigned char, unsigned char)' frameworks/base/core/jni/android/graphics/Paint.cpp:809: error: undefined reference to 'android::uirenderer::Blur::convertRadiusToSigma(float)' collect2: error: ld returned 1 exit status make: *** [out/target/product/sabresd_7d/obj/SHARED_LIBRARIES/libandroid_runtime_intermediates/LINKED/libandroid_runtime.so] Error 1 Re: Android Graphic UI with GPU Hardware Acceleration Android Graphic UI with GPU Hardware Acceleration Re: Android Graphic UI with GPU Hardware Acceleration Hi Xuegang Liu, I want to use software rendering with lollipop android. I am new to this and i want to know how to enable this and what are the things to take care of. Can u please help me in this. Re: Android Graphic UI with GPU Hardware Acceleration LiuXuegang, could you please help to enable software rendering for lollipop? Re: Android Graphic UI with GPU Hardware Acceleration Hi, Its really great article. How can I enable software rendering in lollipop? I dont want to use the hardware rendering. Is it possible to use the software rendering in lollipop? what are the changes i need to do to make it work? Re: Android Graphic UI with GPU Hardware Acceleration Android camera preview window is re-layout in Camera application, the scaling should be performed for the original camera image. Re: Android Graphic UI with GPU Hardware Acceleration That was a VERY good question It was related to the allocation! Btw: Do you have any idea why the preview window in the camera app is quite small. I seem to have a 202x269 image (V/camera ( 3112): surfaceChanged. w=202. h=269). The preview size should be 640x480. I suspect a scaling of the preview size displayed, but if I change the preview size it doesn't scale. Any ideas? (our screen is 800x480) Re: Android Graphic UI with GPU Hardware Acceleration Hi Mads, does video playback work on your platform? vpu also uses the "handle->phys", which is allocated from GPU driver. Re: Android Graphic UI with GPU Hardware Acceleration Hi, In the above Li say: 2. How the captured buffer is submitted to preview window hardware/imx/mx6/libcamera/CameraHal.cpp: first of all, allocate native buffer form preview window: mNativeWindow->dequeue_buffer in allocateBuffersFromNativeWindow On our platform, when we do the dequeue, the handle->phys, handle->base values make no sense (i.e. point to a strange memory area... Not the framebuffer or anything in RAM. I thought that phys was a physical pointer address. (and when I start the camera app, it will stall the device since the IPU will now access this wierd memory area). Our platform is otherwise up and running. Anyone have an idea of how to handle the dequeued data? Re: Android Graphic UI with GPU Hardware Acceleration Thanks Xianzhong. I created a new question at the following link: https://community.freescale.com/thread/309817 Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, I'm not a Android Camera guy, you can create a new question and send me the link, I can help you find the proper guy to answer your question. Re: Android Graphic UI with GPU Hardware Acceleration Thank you, much appreciated. At this point I can start using GPU (in particular pixel shader) in further processing my captured camera video. I already have preview working from the gray scale camera. I capture each frame as 8-bit generic data and use IPU unit to convert 8-bit gray scale raw data into ARGB8888. To do that I use custom color space conversion matrix and YUV->RGB conversion IPU task. All capture buffers including one from the output of the IPU are allocated from the native window. At this point, I submit output from IPU to the preview window. I'd like to insert some GPU shading in between i.e. after IPU and before submitting to preview window. My system is imx6 android ics 13.4.1.04. Can you give me some ideas how this could be done? Re: Android Graphic UI with GPU Hardware Acceleration Dragan, it may not work on jelly bean, but you can modify the code to create Android surface referring to the example, frameworks/native/services/surfaceflinger/tests/surface Re: Android Graphic UI with GPU Hardware Acceleration Xianzhong Li, it works now. Thanks a lot this is really a great help!!! Will this version work on jelly bean too? Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, simple_draw_ics.tgz is attached, please find it. Re: Android Graphic UI with GPU Hardware Acceleration Xianzhong Li, I want to try simple_draw in imx6 imx-android-13.4.1.04 build environment on sabresd_6dq board. I copied simple_draw to ~/myandroid and after starting build I get the following error: target arm C++: simple_draw/draw <= frameworks/simple_draw/simple_draw.cpp frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:46: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp: In function 'void eglOpen()': frameworks/simple_draw/simple_draw.cpp:123: error: 'class android::SurfaceComposerClient' has no member named 'openTransaction' frameworks/simple_draw/simple_draw.cpp:125: error: 'class android::SurfaceComposerClient' has no member named 'closeTransaction' frameworks/simple_draw/simple_draw.cpp:145: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:152: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp:160: warning: deprecated conversion from string constant to 'char*' frameworks/simple_draw/simple_draw.cpp: In function 'int main(int, char**)': frameworks/simple_draw/simple_draw.cpp:427: warning: deprecated conversion from string constant to 'char*' make: *** [out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o] Error 1 I had to change the following line in simple_draw.cpp to get it past compilation stage: //session->openTransaction(); session->openGlobalTransaction(); control->setLayer(0x40000000); //session->closeTransaction(); session->closeGlobalTransaction(); However, now I get the following link error. Can you please help: target Executable: simple_draw/draw (out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/LINKED/simple_draw/draw) prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:114: error: undefined reference to 'android::SurfaceComposerClient::SurfaceComposerClient()' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:117: error: undefined reference to 'android::SurfaceComposerClient::getDisplayInfo(int, android::DisplayInfo*)' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:122: error: undefined reference to 'android::SurfaceComposerClient::createSurface(int, unsigned int, unsigned int, int, unsigned int)' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:124: error: undefined reference to 'android::SurfaceComposerClient::openGlobalTransaction()' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:125: error: undefined reference to 'android::SurfaceControl::setLayer(int)' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:127: error: undefined reference to 'android::SurfaceComposerClient::closeGlobalTransaction(bool)' prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/simple_draw.o: in function eglOpen():simple_draw/simple_draw.cpp:129: error: undefined reference to 'android::SurfaceControl::getSurface() const' collect2: ld returned 1 exit status make: *** [out/target/product/sabresd_6dq/obj/EXECUTABLES/simple_draw/draw_intermediates/LINKED/simple_draw/draw] Error 1 Re: Android Graphic UI with GPU Hardware Acceleration Hi Xianzhong, I got into some problems getting IPUv3 to work. If you could, can you please respond to this thread that I opened specifically on this topic: https://community.freescale.com/thread/306462 Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, please refer to the IPU usage in resizeToSecFrameBuffer function in hardware/imx/mx5x/libgralloc/framebuffer.cpp Re: Android Graphic UI with GPU Hardware Acceleration Hi Xianzhong, I was thinking that 1:1 scale will effectively work as data copy. If you have any IPU example that works from inside Android HAL I could give it a shot. Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, are you thinking to use IPU to copy a buffer? as I know, IPU cannot support data copy without scaling or conversion. Re: Android Graphic UI with GPU Hardware Acceleration Hi Xianzhong, before I start using GPU in capture buffer processing I'd like as an intermediate step to use IPU for simple copy operation. I was thinking to use 1:1 scale as a copy. Can you point me to an example IPU code for scaling or color space conversion from Android HAL, i.e. how to setup IPU task from Android HAL and get result. Thanks. Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, you implementation is ok. Re: Android Graphic UI with GPU Hardware Acceleration This approach works. I was able to get to the buffer, do the processing in s/w and submit to display. Re: Android Graphic UI with GPU Hardware Acceleration Hi Xianzhong, your guidance was of great help. I now understand that capture display works by ping-pong capture buffers between capture and display threads. I outlined bellow high-level code for better understanding of the implementation details. I plan to insert processing in the capture thread and before submitting to display thread. I will now experiment with this setup to check how it works. CameraHal::captureframeThread() { mCaptureThreadQueue.waitMessage(); // received signal that CameraHal::previewshowFrameThread() is ready mCaptureDevice->DevDequeue(&bufIndex); // get next display-ready buffer mCaptureBuffers[bufIndex] // ... do processing on mCaptureBuffers[bufIndex]; mPreviewThreadQueue.postMessage(new CMessage(CMESSAGE_TYPE_NORMAL, bufIndex)); // signal CameraHal::previewshowFrameThread() that mCaptureBuffers[bufIndex] is ready } CameraHal::previewshowFrameThread() { msg = mPreviewThreadQueue.waitMessage(); // received signal that mCaptureBuffers[bufIndex] is ready (bufIndex is in msg->arg0) pInBuf = &mCaptureBuffers[msg->arg0]; // get hold of ready buffer mNativeWindow->enqueue_buffer(mNativeWindow, (buffer_handle_t * )pInBuf->native_buf); // submit ready buffer to display // remove already displayed buffer from display and find its buf_index within mCaptureBuffers[] array mNativeWindow->dequeue_buffer(mNativeWindow, &buf_h, &stride); SearchBuffer((void *)buf_h, &buf_index); mCaptureDevice->DevQueue(buf_index); // Return buffer mCaptureBuffers[buf_index] to Capture Device // signal CameraHal::captureframeThread() that CameraHal::previewshowFrameThread() is ready mCaptureThreadQueue.postMessage(new CMessage(CMESSAGE_TYPE_NORMAL, buf_index)); } Re: Android Graphic UI with GPU Hardware Acceleration Hi Dragan, In order to understand how the captured buffer is shown in preview window, I will give you some code flows as below: 1. How the preview window surface is registered to camera HAL packages/apps/Camera//src/com/android/camera/Camera.java: startPreview --> setPreviewDisplay(mSurfaceHolder); frameworks/base/services/camera/libcameraservice/CameraService.cpp: CameraService::Client::setPreviewDisplay --> CameraService::Client::setPreviewWindow -->CameraHardwareInterface(mHardware->setPreviewWindow) frameworks/base/services/camera/libcameraservice/CameraHardwareInterface.h: setPreviewWindow-->CameraModule(mDevice->ops->set_preview_window = camera_set_preview_window) hardware/imx/mx6/libcamera/CameraModule.cpp: camera_set_preview_window --> cameraHal(gCameraHals[fsl_dev->cameraid]->setPreviewWindow) hardware/imx/mx6/libcamera/CameraHal.cpp:setPreviewWindow mNativeWindow = window://the preview window is registered in CameraHal 2. How the captured buffer is submitted to preview window hardware/imx/mx6/libcamera/CameraHal.cpp: first of all, allocate native buffer form preview window: mNativeWindow->dequeue_buffer in allocateBuffersFromNativeWindow then prepare captured buffer to native buffer ( you know the processing, ignore the sample code) finally submit the capture buffer to preview window: mNativeWindow->enqueue_buffer in previewshowFrameThread frameworks/base/services/camera/libcameraservice/CameraHardwareInterface.h: __enqueue_buffer will call ANativeWindow::queueBuffer then the subsequential flow: frameworks/base/libs/gui/SurfaceTextureClient.cpp hook_queueBuffer(ANativeWindow::queueBuffer = hook_queueBuffer) -->SurfaceTextureClient::queueBuffer --> SurfaceTexture:;queueBuffer(mSurfaceTexture->queueBuffer) frameworks/base/libs/gui/SurfaceTexture.cpp SurfaceTexture::queueBuffer-->Layer::onFrameAvailable(listener->onFrameAvailable) frameworks/base/services/surfaceflinger/layer.cpp void Layer::onFirstRef() { LayerBaseClient::onFirstRef(); struct FrameQueuedListener : public SurfaceTexture::FrameAvailableListener { FrameQueuedListener(Layer* layer) : mLayer(layer) { } private: wp mLayer; virtual void onFrameAvailable() { sp that(mLayer.promote()); if (that != 0) { that->onFrameQueued(); } } }; so the camera preview window is a normal surface created from Surfaceflinger, you can refer to "3D stack with Surfaceflinger" and "Compostion with Hwcomposer" to know how it will be shown in display framebuffer. Hope you can get whatever in submitting preview buffer in CameraHal, let me know if you have more questions. Re: Android Graphic UI with GPU Hardware Acceleration Hi, can you please help me with an example code that shows how is this processing done in the Android HAL layer. In particular, referring to hardware/imx/mx6/libcamera/CameraHAL.cpp, I want to take a buffer from the camera preview (in the preview thread) and apply gpu processing like in your example and submit it to display. I know how to take buffer from the capture preview queue and apply processing but how do you submit it to display? In CameraHAL.cpp I couldn't find references to the display buffers. Re: Android Graphic UI with GPU Hardware Acceleration very useful, thanks.
查看全文
