Error using IPUV3 Library from Android

[DraganOstojic]

Hi, I'm trying to use IPUv3 block to do some scaling/cropping of the camera capture buffer. Platform is Android r13.4, board is MCIMX6Q-SDP board.

First I tried to use sp<PostProcessDeviceInterface> class (interface is in hardware/imx/mx6/libcamera/PostProcessDeviceInterface.cpp) to access IPUV3 Library but I get failure. I also tried to access imx_ipu device directly through ioctl(..., IPU_QUEUE_TASK, ...) interface but again it fails. I do see /dev/mxc_ipu device when I do ls -la /dev/mxc_* from the serial console. From within Android LogCat I get the following errors:

01-02 01:06:41.559: E/FslHwcomposer(2276): init:63,open ipu dev failed

...

01-02 01:09:35.799: E/FslCameraHAL(2279): open ipu dev fail

The first error is from hardware/imx/mx6/hwcomposer/blit_ipu.cpp:

int blit_ipu::init()//, hwc_layer_t *layer, struct output_device *output

{

  int status = -EINVAL;

    mIpuFd = open("/dev/mxc_ipu", O_RDWR, 0);

    if(mIpuFd < 0) {

       HWCOMPOSER_LOG_ERR("%s:%d,open ipu dev failed", __FUNCTION__, __LINE__);

        return status;

    }

    return 0;

}

The second error is from me when I tried to do open("/dev/mxc_ipu", ...) from camera HAL.

I assumed that the open call ends up in mxc_ipu_open() in kernel_imx/drivers/mxc/ipu3/ipu_device.c so I instrumented this function but I don't see this log show up when open("/dev/mxc_ipu", ...) is made. I do see that ipu_device.c is built when the kernel is built.

At this point I'd like some guidance on how to get IPUv3 driver working from within Android so that I can try to use it.

Any help will be greatly appreciated.

[DraganOstojic]

Hi Xiaowen, I found in iMX6 SDK firmware guide that IPU has limitation of width=1024 output pixels because of FIFO line limitation. Can you confirm that? If this is indeed the case I'll not use IPU because my min output  width is 1296.

Is it possible to use DMA from the user mode to accelerate s/w memcpy() in a similar way that IPU is programmed through imx_ipu device? Can you point me to some examples?

Thanks.

[Ivan_liu]

hi, Ostojic，

The limitation means that it will split the width larger than 1024 into several parts.

So, it can handle the width larger than 1024 but with low performance.

You may try it.

BRs,

Ivan

[DraganOstojic]

Hi Xiaowen, I got 3x scaling from 640x480 to (640*3)x480 working. I have one more memcpy into native surface for display which I tried to implement as 1x scale but it fails. It appears that IPU requires at  least something different between source and destinqtion. So instead I scale 2x horizontally.  After these changes it worked. The speed improvement over s/w scaling is unbeliveible (5 times).

The same 3x scale from 2592x1944 failed. Do you think it's alignment issue?

[Ivan_liu]

Hi Ostojic,

It does not support 1x scale in IPU driver.

3x scale from 2592x1944 should be ok.

BRs,

Ivan

[DraganOstojic]

Hi Xiaowen, I made a typo. I'm scaling from 864x1944 3x to 2592x1944. I expect that it should work using split scaling. However, it fails with bellow errors:

(my log) t->set.sp_setting.iw=0xd0e22600 + t->set.sp_setting.i_right_pos=0x6 > iw=0x360

...

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: ERR: no-0x0,ipu_queue_task err:-22

in drivers/mxc/ipu3/ipu_device.c:

static int update_split_setting(struct ipu_task_entry *t, bool vdi_split)

{

...

  if ((t->set.sp_setting.iw + t->set.sp_setting.i_right_pos) > iw)

      return IPU_CHECK_ERR_SPLIT_INPUTW_OVER;

}

Any ideas what I'm doing wrong?

[DraganOstojic]

I couldn't make built in driver banding feature work but I found a workaround which looks pretty good to me (i.e. no visible artifacts). Idea is that work is divided into chunks that are no larger than 1024x1024 but instead of bands, buffer is divided into blocks. So for example, to scale rgb8888 432x968 to rgb8888 1296x968 (this is 3x scale horizontally), 1296 exceeds 1024 output limitation and 968 doesn't. To get correct number of passes, start from the output. 1296/2=648 which is within output limit and 968 can be left unchanged because it's within limit. Work can be done in 2 passes with the following code (Buf_input and Buf_temp are Android DMA_BUFFER pointers but they could be allocated differently, (please refer to rogeriorps/ipu-examples · GitHub for examples how to setup IPU, allocate and map memory in a generic Linux fashion):

             struct ipu_task task;

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.input.paddr = Buf_input->phy_offset;

             task.output.paddr = Buf_temp->phy_offset;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

                 CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.input.paddr = Buf_input->phy_offset+216*968*4;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

In case when both horizontal and vertical size exceed 1024x1024 limit work can be split similarly but may need more than 2 passes. For example to scale 864x1944 to 2592x1944 the following code will work (requires 8 equal size passes and 9th pass for the leftover):

          memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset;

             task.output.paddr = Buf_temp->phy_offset;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

                 CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*2;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*2;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*3;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*3;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*4;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*4;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*5;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*5;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*6;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*6;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 968;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 968;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*7;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*7;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

             memset(&task, 0, sizeof(task));

             task.input.width    = 216;

             task.input.height   = 32;

             task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.width   = 216*3;

             task.output.height  = 32;

             task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

             task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

             task.input.paddr = Buf_input->phy_offset+216*968*4*8;

             task.output.paddr = Buf_temp->phy_offset+216*3*968*4*8;

             if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

               CAMERA_LOG_ERR("scale x3: ioct(IPU_QUEUE_TASK) fail");

             }

As a side note, I also found a way to accelerate memcpy() with IPU block. 1:1 scale won't work, but if some change is specified (for example horizontal flip) it will work:

            memset(&task, 0, sizeof(task));

            task.input.width    = 216*3;

            task.input.height   = 968;

            task.input.format   = v4l2_fourcc('R', 'G', 'B', '4');

            task.output.width   = 216*3;

            task.output.height  = 968;

            task.output.format  = v4l2_fourcc('R', 'G', 'B', '4');

            task.output.rotate  = IPU_ROTATE_HORIZ_FLIP;

            task.input.paddr = Buf_temp->phy_offset;

            task.output.paddr = Buf_input->phy_offset;

            if (ioctl(mIpuFd, IPU_QUEUE_TASK, &task) < 0) {

              CAMERA_LOG_ERR("ioct(IPU_QUEUE_TASK) fail");

            }

To undo horizontal flip another such operation will be required, so with a temp buffer, fast memcpy can be achieved.

[ShawnBai]

Hi Dragan Ostojic,

    Your posts are very helpful.

    And I have a question here, do you mind giving me some advices, thank you in advance.

    Now we're working on Android 4.3 on imx6solo.

    For camera, there is no zoom support in the HAL layer.

    some changes have been made in

int StreamAdapter::renderBuffer(StreamBuffer *buffer)

{

    status_t ret = NO_ERROR;

    GraphicBufferMapper& mapper = GraphicBufferMapper::get();

  // <<==================  ew added to test if set_crop works or not

    mNativeWindow->set_crop(mNativeWindow, 100/16 /*xOff/bytesPerPixel*/, 100/16 /*yOff*/,

                                     1982/16, 1080/16);

    // unlock buffer before sending to stream

    mapper.unlock(buffer->mBufHandle);

    ret = mNativeWindow->enqueue_buffer(mNativeWindow, buffer->mTimeStamp,

                                        &buffer->mBufHandle);

    if (ret != 0) {

        FLOGE("Surface::queueBuffer returned error %d", ret);

    }

    return ret;

}

     But it seems set_crop() doesnot work well.

    From your posts, it seems the camera capture buffer from the kernel driver with

./DeviceAdapter.cpp:608:    ret = ioctl(mCameraHandle, VIDIOC_DQBUF, &cfilledbuffer);

could be used as the input arguments to IPU, and in android HAL a temp buffer could be used as the output result from IPU, and after that, the temp buffer is gived to android uppler layer for their further operation, am I right?

    Could you share your code for me to learn further, anyway, thank you for your posts, really helpful.

[Ivan_liu · Accepted solution]

please check the authority of /dev/mxc_ipu file node.

and add the camera access authority for /dev/mxc_ipu.

you may change the authority like this:

modify

/dev/mxc_ipu    0660   graphics   system

to

/dev/mxc_ipu    0666   graphics   system

in ueventd.freescale.rc.

or you may have a test on command line before camera started.

chmod 0666 /dev/mxc_ipu

to check if it can works.

[DraganOstojic]

Hi Xiaowen, this made it work. Thanks a lot for you your help. Much appreciated.

[paul_lee]

Hi,

I have ported android on my custom hardware.

My hardware supports the IPU.

I am getting the same error while opening IPU device.

if( open("/dev/mxc_ipu", O_RDWR, 0)< 0)

{

     pr_err("open ipu dev failed \n);

}

Paul.

[Ivan_liu]

Hi, Paul

It is the same problem met by DraganOstojic.

Please check it with the same method mentioned above.

BRs,

Ivan

[paul_lee]

Hi,

I have made change " ueventd.freescale.rc".

But when i boot the hardware there is not /dev/mxc_ipu.

There may be issue in boardconfig file to enable the IPU.

What changes required to enable IPU?

Thanks,

Paul

[Ivan_liu]

Hi Paul

What change do you make?

Can you try it by chmod 0666 /dev/mxc_ipu in command line.

If it works, then you change the ueventd.freescale.rc.

BRs,

Ivan