IPU support for Custom Image format for MIPI camera

[titusstalin]

Dear All,

I've have a custom camera which will give the customized format data, say TYPE31 format.

I've tried to enable the our customized format data in imx drivers.

mxc_v4l2_capture.c

For IPU, ipu_csi_enc.c, ipu_prp_enc.c and ipu_still.c files.

I have used "IPU_PIX_FMT_GENERIC_32" configuration for my camera.

I'm getting interrupts and able to get the frame too, but the data was corrupted.

I don't want to process anything in IPU but just give what my MIPI camera gives.

What IPC configration I should use ?

Any idea on this, how can I fix the problem ?

Can you please any one help me on this ?

Regards,

Titus S.

[igorpadykov]

Hi Titus

seems such format can be handled as bayer, examples can be found on

https://community.freescale.com/message/331888

i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0) 

IPU - bayer pattern conversion 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[titusstalin]

Thanks for your reply.

Let me look into it and get back to you.

Thanks.

[titusstalin]

Hi igor,

Thanks for your reply.

1) I'm sending the frames 800x600x3 (Custom image) from MIPI excersior and set IPU as IPU_PIX_FMT_GENERIC_32 and got the interrupt & received frame.

But its totally not matched for IPU_PIX_FMT_GENERIC_32  IPU format setting.

2) If I set IPU as IPU_PIX_FMT_GENERIC_16 then I'm receiving exact data but upto 960000 bytes i.e 800x600x2 and rest of the datas are filled with zeros.

3) If I set IPU as IPU_PIX_FMT_GENERIC then I'm receiving exact data but upto 480000 bytes i.e 800x600 and rest of the datas are filled with zeros.

Do we have any IPU GENERIC for 24bit ?

IPU_PIX_FMT_GENERIC_24

And also, why there is no "ipu_ch_param_set_field" function is not used for "IPU_PIX_FMT_GENERIC_32" ?

drivers/mxc/ipu3/ipu_param_mem.h

case IPU_PIX_FMT_GENERIC:
/*Represents 8-bit Generic data */
ipu_ch_param_set_field(&params, 0, 107, 3, 5); /* bits/pixel */
ipu_ch_param_set_field(&params, 1, 85, 4, 6); /* pix format */
ipu_ch_param_set_field(&params, 1, 78, 7, 63); /* burst size */

break;
case IPU_PIX_FMT_GENERIC_16:
/* Represents 16-bit generic data */
ipu_ch_param_set_field(&params, 0, 107, 3, 3); /* bits/pixel */
ipu_ch_param_set_field(&params, 1, 85, 4, 6); /* pix format */
ipu_ch_param_set_field(&params, 1, 78, 7, 31); /* burst size */

break;
case IPU_PIX_FMT_GENERIC_32:
/*Represents 32-bit Generic data */
break;

Then I used RGB24 & BGR24, it received 14400000 bytes (800x600x3) of data with no zeros but datas are missing in between values.

Received data for RGB24 & BGR24:

0000000 0301 0500 0007 0b09 0d00 000f 1311 1500
0000010 0017 1b19 1d00 001f 2321 2500 0027 2b29
0000020 2d00 002f 3331 3500 0037 3b39 3d00 003f
0000030 4341 4500 0047 4b49 4d00 004f 5351 5500
0000040 0057 5b59 5d00 005f 6361 6500 0067 6b69
0000050 6d00 006f 7371 7500 0077 7b79 7d00 007f

It seems, the data are got swaped and some data's are missed.

Actual data:

0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000010 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
0000020 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
0000030 3130 3332 3534 3736 3938 3b3a 3d3c 3f3e
0000040 4140 4342 4544 4746 4948 4b4a 4d4c 4f4e
0000050 5150 5352 5554 5756 5958 5b5a 5d5c 5f5e

I've also tried to disable the "_ipu_ch_params_set_packing(&params, 5, 1, 5, 6, 5, 11, 1, 0);" function for BGR24 and RGB24, the behavior is same.

These are the files I have modified for "IPU_PIX_FMT_GENERIC_32"

ipu_prp_enc.c

ipu_csi_enc.c

ipu_still.c

Do I need to change any other files ?

How can I disable the IPU stuff (should not process anything) and it should pass the the data comes to memory ?

Which IPU format I have to select to get the data in 800x600x3 ?

Can you please help me on this issue ?

Regards,

Titus S.

[igorpadykov]

Hi Titus

do you know exact data structure sent by camera, according to

mipi specs TYPE31 format  is User Defined Byte-based Data.

If it is not known, seems it is not possible to use this camera. If known

you can convert it to format which could be used by i.MX mipi and

use provided examples.

Best regards
igor

[titusstalin]

Thanks igor.

Yes, I knew how the TYPE31 format will be.

How can I convert to format which could be used by imx MIPI ?

What files I need to change and how we can change ?

Can you please help me on this ?

Really appreciate your help on this.

Regards,

Titus S.

[igorpadykov]

Hi Titus

in general you can receive camera data as MIPI RAW format,

then process data in memory by software and convert it to formats usable by IPU.

NXP has special service for helping customers with porting software: Professional Services:

http://www.nxp.com/support/nxp-professional-services:PROFESSIONAL-SERVICE

TYPE31 format means that vendor of that camera "closed" data format using some non-standard

format, probably to prevent non-authorizied usage.

Best regards
igor

[titusstalin]

Hi igor,

As per your advise, I've created the post in profession services too.

Understood.

If I want to get RAW format output then what shall I change ?

Actually I'm getting the exact data (correct payload information) if I set IPU_PIX_FMT_16, but not with IPU_PIX_FMT_32.

How its working ? :smileysad:

For this IPU is able to pass the values to memory and memory to userspace.

Do we have any document or AN to use IPU_PIX_FMT_GENERIC ??

[titusstalin]

I've also added one more entry to process GENERIC_24 and got the below output.

drivers/mxc/ipu3/

Changed the file "ipu_idmac.c" "ipu_disp.c" "ipu_capture.c"  "ipu_common.c" "ipu_dma.h" "ipu.h"  and "ipu_param_mem.h" files

IP_PIX_FMT_GENERIC_24

0000000 0000 0080 8400 0000 0088 8c00 0000 0090
0000010 9400 0000 0098 9c00 0000 00a0 a400 0000
0000020 00a8 ac00 0000 00b0 b400 0000 00b8 bc00
0000030 0000 00c0 c400 0000 00c8 cc00 0000 00d0
0000040 d400 0000 00d8 dc00 0000 00e0 e400 0000
0000050 00e8 ec00 0000 00f0 f400 0000 00f8 fc00

Actual data:

0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000010 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
0000020 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
0000030 3130 3332 3534 3736 3938 3b3a 3d3c 3f3e
0000040 4140 4342 4544 4746 4948 4b4a 4d4c 4f4e
0000050 5150 5352 5554 5756 5958 5b5a 5d5c 5f5e

[igorpadykov]

so you sure that "Actual data" are GENERIC_24 format,

is this confirmed by this sensor specs ?

[titusstalin]

Yes, right now I've configured the camera like that.

800*600*3, fps is 30, type31 format.

BSP : GitHub - boundarydevices/linux-imx6 at boundary-imx_3.14.52_1.1.0_ga 

Board : Sabrelite and Nitrogen6_MAX

Here is some more doubts for me when I modify the IPU.

1) What to change if IPU_PIX_FMT_GENERIC_24 ?
drivers/mxc/ipu3/ipu_common.c

        if (pixel_fmt == IPU_PIX_FMT_GENERIC)
            /* 8 bits per pixel */
            burst_size = burst_size >> 4;
        else if (pixel_fmt == IPU_PIX_FMT_GENERIC_16)
            /* 16 bits per pixel */
            burst_size = burst_size >> 3;
        else
            burst_size = burst_size >> 2;
        _ipu_smfc_set_burst_size(ipu, channel, burst_size-1);

2) Here, what I have to change it for IPU_PIX_FMT_GENERIC_24 ?
Really nothing is required for IPU_PIX_FMT_GENERIC_32 ?

drivers/mxc/ipu3/ipu_param_mem.h
    case IPU_PIX_FMT_GENERIC:
        /*Represents 8-bit Generic data */
        ipu_ch_param_set_field(&params, 0, 107, 3, 5);    /* bits/pixel */
        ipu_ch_param_set_field(&params, 1, 85, 4, 6);    /* pix format */
        ipu_ch_param_set_field(&params, 1, 78, 7, 63);    /* burst size */

        break;
    case IPU_PIX_FMT_GENERIC_16:
        /* Represents 16-bit generic data */
        ipu_ch_param_set_field(&params, 0, 107, 3, 3);    /* bits/pixel */
        ipu_ch_param_set_field(&params, 1, 85, 4, 6);    /* pix format */
        ipu_ch_param_set_field(&params, 1, 78, 7, 31);    /* burst size */

        break;
    case IPU_PIX_FMT_GENERIC_24:
        /* Titus Modified */
        /*Represents 24-bit Generic data */
        ipu_ch_param_set_field(&params, 0, 107, 3, 1);    /* bits/pixel */
        ipu_ch_param_set_field(&params, 1, 85, 4, 7);    /* pix format */
        ipu_ch_param_set_field(&params, 1, 78, 7, 19);    /* burst size */
        
        break;
    case IPU_PIX_FMT_GENERIC_32:
        /*Represents 32-bit Generic data */
        /* Titus : Really nothing is required ? */
        break;

3) Why we didn't put IPU_PIX_FMT_GENERIC_32 code ?
drivers/mxc/ipu3/ipu_capture.c

    case IPU_PIX_FMT_GENERIC:
    case IPU_PIX_FMT_GENERIC_16:
//    case IPU_PIX_FMT_GENERIC_32: /* Titus : Added, why we didn't add this line */
    case IPU_PIX_FMT_GENERIC_24:
        cfg_param.data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
        break;

4) Is it correct ?
drivers/dma/ipu/ipu_idmac.c
static uint32_t bytes_per_pixel(enum pixel_fmt fmt)
{
    switch (fmt) {
    case IPU_PIX_FMT_GENERIC:    /* generic data */
    case IPU_PIX_FMT_RGB332:
    case IPU_PIX_FMT_YUV420P:
    case IPU_PIX_FMT_YUV422P:
    default:
        return 1;
    case IPU_PIX_FMT_RGB565:
    case IPU_PIX_FMT_YUYV:
    case IPU_PIX_FMT_UYVY:
        return 2;
    case IPU_PIX_FMT_GENERIC_24:    /* Titus : generic data 24bit */
    case IPU_PIX_FMT_BGR24:
    case IPU_PIX_FMT_RGB24:
        return 3;
    case IPU_PIX_FMT_GENERIC_32:    /* generic data */
    case IPU_PIX_FMT_BGR32:
    case IPU_PIX_FMT_RGB32:
    case IPU_PIX_FMT_ABGR32:
        return 4;
    }
}

Thanks for your support.

[igorpadykov]

what is the reason to modify the IPU drivers, if you know

how this TYPE31 format data is arranged, then you can convert

it to any formats which is already supported in BSP, for example ov5640,42.

If you do not know how this TYPE31 format data is arranged, then there

is no way to use this camera.

[titusstalin]

Thanks for your support.

Understood.

I don't want to convert it to any format but send the data to userspace what received.

Don't  want IPU to process anything just pass the data.

Then what IPU format I can use now?