i.MX6 demosaic using GPU in Android framework code

[matthewdavis]

Hello all,I'm working on some GPU-based demosaic code for a RAW-only image sensor and I'm running into some trouble getting Android to allow me to access the GPU hardware using OpenGL ES.  Here's what I've done:

0)  Wrote a kernel driver for my sensor and modifed the kernel driver framework to pass through grayscale images.

1)  Created a custom DeviceAdapter by inheriting from OvCamera in libcamera2.  This class treats the camera sensor as a grayscale camera but tells the Android layer that the images are RGB565.  Verified that this puts images up on the screen at a smooth frame rate.  So far so good. 

2)  Wrote some cod to handle the demosiac logic using thee CPU to get the pixel format to match what Android expects.  I've inserted this in the camera thread, so images come off of the camera in raw bayer format and are immediately converted to RGB565.  This works well, but obviously performance is not adequate.

3)  Wrote a replacement demosaic module using OpenGL ES and attempted to swap it in.  This fails at runtime (no pixels are read back out after processing).  It looks like the problem starts immediately at eglGetDisplay(), because I see this in the logs:E/libEGL  (  154): call to OpenGL ES API with no current context (logged once per thread)

I interpret the error to mean that something else has already opened up an OpenGL context in another thread and I need to make my calls from that context.  I've seen others mention in the forum that they've done everything on the camera capture thread (making sure to make all OpenGL calls on the capture thread, which I have done), but this error happens immediately as I try to initialize the library.  Since this this is Android framework code, I guess I shouldn't be surprised that I'm not the only one in my process space hoping to use the GPU, but I don't know how others have gotten around this problem.  The rules for writing stand-alone native apps seem like they don't apply to this situation.

Does anybody have any suggestions?

Thanks,

-Matt

[Bio_TICFSL]

Hi Matthew,

IPU can't support converting bayer format to RGB/YUV on the fly, it can only use generic mode to receive the CSI input bayer data to memory. The CPD can only be used to convert 10 bits color data to 8 bits. please check  https://community.freescale.com/thread/302769

I suggest for demosaicing do it with OpenCL and conversion format to RGB888, such as mention on: Efficient, High-Quality Bayer Demosaic Filtering on GPUs

other communities link that can be helpful:

https://community.freescale.com/message/343717#comment-343717

Hope this helps

[matthewdavis]

I've written and tested an OpenCL implementation of the logic and it works on my test workstation, but as with the OpenGL version, I'm having trouble with Android.

It looks like the Android BSP comes with OpenCL headers but no libraries.  I've seen the .so libraries posted in other forums and I've downloaded them, but I can't get the Android OS build to link against them when building the framework.  Most of the documentation for using prebuilt libraries is for apps, not core framework code.  Do you know how to get Android to link against those prebuilt libraries when building the framework? It looks like I have two implementations that should work if I can just get them integrated with our modified imx_KK4.4.3_2.0.0-ga Android BSP.  I'll use whichever one I can get working.

[matthewdavis]

I'll post here what I've been able to figure out in case somebody else wanders in here in vain hope of clarification.  Facts I've figured out:

1)  OpenCL doesn't appear to be supported in Android.  Don't even bother.

2)  OpenGL processing performance for demosaic is good, but Vivante's driver/OpenGL implementation has a *staggeringly* slow glReadPixels() implementation that makes it completely worthless for getting data out of the GPU's memory space.

3)  There apparently used to be support for a framebuffer backend to allow you to mmap() , but it doesn't appear to be supported in Android anymore (or at all?).  I've never seen a version of the Android libraries that contains the -fb version, but it looks like the Linux BSP does.

4)  It looks like *maybe* the solution is to render to texture and use Vivante's glTexDirectVIV family of function to get direct pointers to the texture memory.  There's documentation for this readelf shows that functions by that name, but since I've never seen any header files that declare it (only online docs and forum posts), I'm having trouble getting it to link, so I can't tell you whether it actually works.  It also seems to have some pretty severe format limitations.

So those are the wrong things you should avoid.  Not sure what the right is yet, but a few more months of just trying every .so file I can find from every source except NXP may yield results.  If it does, I'll surely post the right answer.

[matthewdavis]

Thanks for the OpenCL idea.  I'm using the logic in the document you referenced, but OpenGL is clearly getting in the way.  Hopefully OpenCL won't worry about who has a display handle or which thread is the display thread.

I'll experiment with this and post my results.