i.mx6 Aptina AR0330 MIPI driver

[ivankozic]

Hi all,

I am having major difficulties in making AR0330 run on an i.mx6 system. I am fairly new to i.mx6 / Linux, but I'm getting into it quite fast - Nevertheless, I was wondering if someone could give me some pointers on how to proceed, as currently I am stuck.

I have written a driver for Aptina AR0330 based on OV5640 MIPI driver. The concept is the same, although Omnivision has some different commands and settings. Virtual channel should be fixed to 0 by Aptina, whereas with Omnivision it is selectable through a register. MIPI is functional, but after that something goes wrong.

Currently, I believe that the major issue is the format - OV has YUV, while Aptina has Bayer pattern. The RM for i.mx6Q specifies that on-the-fly processing on the IPU for Bayer pattern is not supported (I don't really know what this means - I guess it has to buffer a frame to the memory first).

I have set up 2 lanes and MIPI is working I suppose (no ERR regs). Mipi format is set to RAW12 (as sensor outputs 12 bits). V4L puzzles me - there is a ioctl in the driver which is used to set the V4L format (ioctl_g_ifparm) with the following code:

     p->u.bt656.clock_curr = ar0330_data.mclk;

     p->u.bt656.mode = V4L2_IF_TYPE_BT656_MODE_NOBT_8BIT;

     p->u.bt656.clock_min = AR0330_XCLK_MIN;

     p->u.bt656.clock_max = AR0330_XCLK_MAX;

     p->u.bt656.bt_sync_correct = 1;  /* Indicate external vsync */

I am not really sure why the clock is used here (I have just copied OV settings) - AR0330 uses fixed clock that comes from crystal (24MHz)?

Also not sure why BT656 is used in the whole story?

Also, OV5640 is connected to CSI1 of IPU0, using the Virtual channel 0. In RM, in the CSI2IPU gasket it is specified that VC0 is connected to CSI0 of IPU0, so I'm not really sure what is going on there as well, as people seem to be using this driver "as-is" and it works?

So in general, I have an issue understanding V4L2 - as far as I could see from DS and RM the pipeline is something like:

sensor->mipi_phy->mipi_csi2->csi2ipu_gasket->IPU->memory?

I am guessing V4L is used to init this whole pipeline (IPU mainly, as MIPI is initialized by the driver) and fetch data from this memory. Please correct me if I'm wrong.

So, I would be very grateful if someone could answer my questions and point me in the right direction if I'm wrong somewhere...

Thanks in advance!

[max_tsai]

hi

For "bt656", the sensor supports bt656 output, so kernel add BT656 interface for get/set parameter from sensor driver, as my understand. The commit is the below.

---

commit 08256ea0da18db20f2edc2e8c935cf74c33ad564

Author: Sakari Ailus <sakari.ailus@nokia.com>

Date:   Thu Aug 30 09:20:39 2007 -0300

    V4L/DVB (6217): V4L: Int if: Add BT.656 interface support

    This patch adds BT.656 interface settings for [gs]_ifparm.

--

IPU needs the "clock_curr" of bt565 to decide CSI_SENS_PRTCL.

For virtual channel, the "virtual channel" of CSI2IPU gasket is different from the defition of "v_channel" of
mipi_csi2_platform_data of kernel driver, as my understand. The virtual channel of CSI2IPU decides how mipi/csi2 maps to IPU/CSI. The virtual channel of mipi_csi2_platform_data is part of MIPI DATA IDENTITY.

For "sensor->mipi_phy->mipi_csi2->csi2ipu_gasket->IPU->memory?", it should be right.

[stonealexander]

Hello ,I'm working on an I.MX6Q board with AR0330.

I tried many ways, but till now I just cannot receive any data from the MIPI-CSI2.

I use IPU1 CSI1 with virtual channel 3.

Both IPU_INT_STAT_1/2 status registers are 0.

The MIPI_CSI_PHY_STATE is 0x300.

I set the IOMUXC_GPR1 bit 20 to 0.
The problem confused me for days, I hope someone can help me.

[meflo]

So basically, if the sensor supports bt656 output, one can work with bt656 timing/data mode protocol with MIPI even though the reference manual states the nongated mode should be used with mipi?

[ivankozic]

Hi Max,

Well, I have to say that Freescale documentation is rather hectic then - virtual channel is never mentioned in two senses in RM. Anyway, it seems that my MIPI is working, but I get garbled data from the sensor. It might very well be a clock issue.

I'm interested in this "clock_curr" - is this a physical clock in the tree (if so, please tell me which one)?

My sensor is supplied with CLK2 (LVDS clock) at 24 MHz (from crystal), so this is rather fixed. There is no way to change it. However in Kernel source, several clocks are mentioned (driver mclk, bt656 clock_curr) and this is not documented at all in any of Freescale's documents. DO you know how these are mapped to the actual HW?

[max_tsai]

FSL BSP provides an interface, named "set_mclk_rate", to change the mclk for FSL reference boards. "mclk" of "sensor_data" structure provides a convinent space to reserve the expect main clock whatever from board definition (for ex, board_mx6q_sabresd.c) or hard-code within sensor driver. The design is for different mclk with different FSL reference board.

For "clock_curr", it just an interface which kernel provides to sensor driver for set/get parm. FSL BSP use the field to decide Sensor Protocol of IPUx_CSIn_SENS_CONF register. It's not a physical clock.

//"drivers/media/video/mxc/capture/mxc_v4l2_capture.c"

        if (ifparm.u.bt656.clock_curr == 0)

                csi_param.clk_mode = IPU_CSI_CLK_MODE_CCIR656_INTERLACED;

        else

                csi_param.clk_mode = IPU_CSI_CLK_MODE_GATED_CLK;

//"drivers/mxc/ipu3/ipu_capture.c"

                cfg_param.ext_vsync << CSI_SENS_CONF_EXT_VSYNC_SHIFT |

                cfg_param.clk_mode << CSI_SENS_CONF_SENS_PRTCL_SHIFT |

                cfg_param.pack_tight << CSI_SENS_CONF_PACK_TIGHT_SHIFT |

[ivankozic]

I am in the process of checking clocks right now - I will post the results when I'm finished.

However, I have found one more quite puzzling thing (it could also be a root of the garbled data problem). In ipu_still.c, there are these lines (default by Freescale):

memset(&params, 0, sizeof(params));

err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

However, params structure is init to all zeros and ipu_init_channel uses some data which actually needs to be initialized before the function is called. I mean specifically these fields which I added before I call ipu_channel_init:

params.csi_mem.mipi_en = 1;
params.csi_mem.mipi_id = 0x2C;

params.csi_mem.mipi_vc = 0;

It seems that IPU would be completely useless before with default Freescale configuration  because CSI0_DATA_SOURCE (bit 28 of IPU_CONF) is set to '0' by default, which effectively means that only parallel input is possible. BUT! When I set this correctly, I receive zero data from sensor (all zeros) and it takes quite a while to get it, while with wrong config (Freescale's) I get the garbled data.

This situation is quite puzzling. Do you know anything about this?

[max_tsai]

The default BSP doesn't fetch the data from kernel to user space via system call, "read". "prp_still_start" is called by "mxc_v4l_read", and "mxc_v4l_read" is registered as a system call.

The value of IPU CSI register is assigned by "mxc_v4l_open", which value comes from sensor driver via "vidioc_int_g_ifparm".

[ivankozic]

Hi Max,

It is true that IPU_SENS_CONF is updated via mxc_v4l_open, and this is done more-less correctly by default Freescale BSP (there are some bugs - data width is not done properly according to the RM, but I wrote a few lines which fix it - there are also patches available on the web for this issue).

However, when using still image capture, IPU_CONF register (not IPU_SENS_CONF) is initialized by using ipu_init_channel function, found in ipu_common.c. The function which calls ipu_init_channel is prp_still_start from ipu_still.c. Before calling it, prp_still_start initializes params structure to all '0's, including the MIPI stuff. In turn, one of the first things which are checked in ipu_init_channel is whether it's MIPI or parallel port by using mipi_en field of params structure which is all zeros:

/*SMFC setting*/

        if (params->csi_mem.mipi_en) {

            printk("R I G H T !\n");

            ipu_conf |= (1 << (IPU_CONF_CSI0_DATA_SOURCE_OFFSET +

                params->csi_mem.csi));

            _ipu_smfc_init(ipu, channel, params->csi_mem.mipi_vc,

                params->csi_mem.csi);

            _ipu_csi_set_mipi_di(ipu, params->csi_mem.mipi_vc,

                params->csi_mem.mipi_id, params->csi_mem.csi);

        } else {

            printk("W R O N G !\n");

            ipu_conf &= ~(1 << (IPU_CONF_CSI0_DATA_SOURCE_OFFSET +

                params->csi_mem.csi));

            _ipu_smfc_init(ipu, channel, 0, params->csi_mem.csi);

        }

This is all default Freescale BSP and there's clearly a bug there, as mipi_en will never be 1.

I only added printk's.

My question is:

When I fix this, so that mipi_en is 1, and set the correct datatype (RAW12 is 0x2c according to mipi_csi2.h), I get data from the sensor, but it's all zeros, while with wrong config (where CSI port is set to parallel data), I get garbled data which is somewhat correct (it's completely unusable but I see the test pattern from the sensor). I'm just wondering why do I see anything when CSI port is configured incorrectly and nothing when I correctly configure it?

[max_tsai]

hi,

Regarding "ipu_init_channel" called by multiple logic path, usually it won't called by prp_still_start. Unless you want to read data vis "read" system call. You may add "dump_stack()" into "ipu_init_channel" to check the logic flow in your environment.

---

   2    234  drivers/media/video/mxc/capture/ipu_bg_overlay_sdc.c <<csi_enc_setup>>

             err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

   3    165  drivers/media/video/mxc/capture/ipu_csi_enc.c <<csi_enc_setup>>

             err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

   4    243  drivers/media/video/mxc/capture/ipu_fg_overlay_sdc.c <<csi_enc_setup>>

             err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

  11    124  drivers/media/video/mxc/capture/ipu_still.c <<prp_still_start>>

             err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

---

For mipi_en, I trace ov5640_mipi driver, it's enabled via "ov5640_init_mode"->"mipi_csi2_enable". And "csi_enc_setup" check IPU&CSI id to determine if mipi_en is true.

For bayer format, there are some discussion here.

--

https://community.freescale.com/message/327253#327253

https://community.freescale.com/message/309833#309833

--

IPU doesn't handle bayer data, and IPU takes bayer data as generic format. But the generic format is not supported by default BSP.

-Max

[ivankozic]

Hi Max,

I'll bold out important questions.

Actually, I am using a (heavily) modified mxc_v4l2_still.out (part of imx-test package), so I am in fact using mxc_v4l_read to get a single still frame. And prp_still_start is called in this case. I am not sure if there is a better way to capture Bayer data?

Regarding mipi_en, yes the MIPI subsystem is enabled by ar0330 driver (I do it the same as for ov5640_mipi), but this is enabling MIPI receiver and I have verified this to work already. The issue here is in IPU, since MIPI receiver is connected to IPU via CSI2IPU gasket. I am not really sure how this works internally, as of course, Freescale will not let us know that :smileyhappy: But what I gather is that IPU_CONF bit 28 is muxing between data incoming from CSI parallel port or CSI2IPU gasket. Maybe I'm mistaken how it's routed, but this bit selects either incoming parallel or incoming serial comm to IPU.

The mipi_en that I'm talking about is a field in params structure which is not set in prp_still_start (and therefore, Bit 28 is not set, so IPU should be seeing only the CSI parallel port - as far as I understand it, MIPI should be completely gated here). So mipi_en is not to enable MIPI receiver, but to configure IPU to receive data from MIPI CSI instead from parallel CSI. The problem is, that I get better results with MIPI not being connected to IPU than when it is (with B28 = 0, I get better results than with B28 = 1) - this is quite weird, because, as I said - if B28 is not set, IPU should not see any MIPI traffic (and it does), while with B28 reset, IPU should see at least some MIPI traffic (and I get all '0'). I don't really understand it? It would be great if you can see how Bit28 of IPU_CONF gates the data on the hardware level - this would explain a lot.

For Bayer stuff - I have done my homework and I am aware of IPU not being able to process anything that is Bayer or generic - this is why I am using it only to capture data (we are going to see later if SW demosaic would be sufficient or an FPGA needs to do it before even sending it to i.mx). But I always look for more info, as you see :smileyhappy: and the first link that you've posted does not work for me:

Unauthorized

Access to this place or content is restricted. If you think this is a mistake, please contact your administrator or the person who directed you here. 

Can you make it accessible?

[max_tsai]

hi I get your point but have no idea about IPU_CONF:B28 question. Suggest you to fetch data by mxc_v4l2_capture with CSI_MEM channel to avoid ipu_still issue. Besides, the following is a workable patch for sensor output JPEG data, and IPU taskles as generic format. That includes a little bugs (not handle JPEG varying length) but work. For your reference.

Subject: [PATCH] Enable JPEG camera.

---

drivers/media/video/mxc/capture/SENSOR_mipi.c     |   12 +++++++++---

drivers/media/video/mxc/capture/ipu_csi_enc.c      |    2 ++

drivers/media/video/mxc/capture/mxc_v4l2_capture.c |    5 +++++

3 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/media/video/mxc/capture/SENSOR_mipi.c b/drivers/media/video/mxc/capture/SENSOR_mipi.c

index e94fe3c..4612269 100644

--- a/drivers/media/video/mxc/capture/SENSOR_mipi.c

+++ b/drivers/media/video/mxc/capture/SENSOR_mipi.c

@@ -4486,8 +4486,10 @@ static int SENSOR_init_mode(enum SENSOR_frame_rate frame_rate,

                 mipi_csi2_reset(mipi_csi2_info);

             if (SENSOR_data.pix.pixelformat == V4L2_PIX_FMT_UYVY)//V4L2_PIX_FMT_YUYV)

-                mipi_csi2_set_datatype(mipi_csi2_info, 0x1E); //0x1e 8bit

-            else if (SENSOR_data.pix.pixelformat == V4L2_PIX_FMT_RGB565)

+                mipi_csi2_set_datatype(mipi_csi2_info, 0x1e); //0x1e 8bit

+            else if (SENSOR_data.pix.pixelformat == V4L2_PIX_FMT_JPEG) {

+                mipi_csi2_set_datatype(mipi_csi2_info, 0x30);

+            } else if (SENSOR_data.pix.pixelformat == V4L2_PIX_FMT_RGB565)

                 mipi_csi2_set_datatype(mipi_csi2_info, MIPI_DT_RGB565);

             else

                 pr_err("currently this sensor format can not be supported!\n");

@@ -4502,6 +4504,8 @@ static int SENSOR_init_mode(enum SENSOR_frame_rate frame_rate,

     if ((mode == SENSOR_mode_INIT) || (mode ==0)) {

         pr_err("SENSOR_init_mode init mode_0329\n");

+    SENSOR_data.pix.width = 640;

+    SENSOR_data.pix.height = 480;

         if(mode == 0)

         {

             icount = 0;

@@ -4570,6 +4574,8 @@ static int SENSOR_init_mode(enum SENSOR_frame_rate frame_rate,

          // set SENSOR to capture mode //

         pr_err("SENSOR_init_mode capture mode\n");

+    SENSOR_data.pix.width = 2560;

+    SENSOR_data.pix.height = 1920;

         //capture

             //        pr_err("987 start to capture\n");

              /*

@@ -5202,7 +5208,7 @@ static int SENSOR_probe(struct i2c_client *client,

     SENSOR_data.io_init = plat_data->io_init;

     SENSOR_data.i2c_client = client;

-    SENSOR_data.pix.pixelformat = V4L2_PIX_FMT_UYVY;//V4L2_PIX_FMT_YUYV;

+    SENSOR_data.pix.pixelformat = V4L2_PIX_FMT_JPEG;//V4L2_PIX_FMT_YUYV;

     SENSOR_data.pix.width = 640;

     SENSOR_data.pix.height = 480;

     SENSOR_data.streamcap.capability = V4L2_MODE_HIGHQUALITY |

diff --git a/drivers/media/video/mxc/capture/ipu_csi_enc.c b/drivers/media/video/mxc/capture/ipu_csi_enc.c

index 0261a7e..70a8df2 100644

--- a/drivers/media/video/mxc/capture/ipu_csi_enc.c

+++ b/drivers/media/video/mxc/capture/ipu_csi_enc.c

@@ -122,6 +122,8 @@ static int csi_enc_setup(cam_data *cam)

         pixel_fmt = IPU_PIX_FMT_BGR32;

     else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_RGB32)

         pixel_fmt = IPU_PIX_FMT_RGB32;

+    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_JPEG)

+        pixel_fmt = IPU_PIX_FMT_GENERIC;

     else {

         printk(KERN_ERR "format not supported\n");

         return -EINVAL;

diff --git a/drivers/media/video/mxc/capture/mxc_v4l2_capture.c b/drivers/media/video/mxc/capture/mxc_v4l2_capture.c

index 68afd70..73e3049 100644

--- a/drivers/media/video/mxc/capture/mxc_v4l2_capture.c

+++ b/drivers/media/video/mxc/capture/mxc_v4l2_capture.c

@@ -378,6 +378,7 @@ static inline int valid_mode(u32 palette)

         (palette == V4L2_PIX_FMT_UYVY) ||

         (palette == V4L2_PIX_FMT_YUYV) ||

         (palette == V4L2_PIX_FMT_YUV420) ||

+        (palette == V4L2_PIX_FMT_JPEG) ||

         (palette == V4L2_PIX_FMT_NV12));

}

@@ -903,6 +904,10 @@ static int mxc_v4l2_s_fmt(cam_data *cam, struct v4l2_format *f)

             size = f->fmt.pix.width * f->fmt.pix.height * 3 / 2;

             bytesperline = f->fmt.pix.width;

             break;

+        case V4L2_PIX_FMT_JPEG: /* JPEG8, 2bits per pixel */

+            size = f->fmt.pix.width * f->fmt.pix.height / 4;

+            bytesperline = f->fmt.pix.width;

+            break;

         default:

             break;

         }

--

1.7.4.1

[ivankozic]

Hi Max,

Quick update - it behaves the same way like before. So when correct config is applied (with mipi_en = 1), I get a lot of weird inconsistent data (basically '0' and 'FF' bytes that don't seem to be too much related - the output file has patches of areas where all bits are set or reset). When wrong config is applied (mipi_en = 0 - I just comment out the line params.csi_mem.mipi_en = true), I get garbled data (bits are slipping like there's not enough bandwidth but it is the correct pattern). Slipping is as follows:

Correct RG pattern: 0F 00 00 00 0F 00 00 00 ... (R = 0x0F00, G = 0x0000)

My RG pattern: 0E E8 00 00 E0 00 0E E8 00 00 E0 00 ... (?).

So instead of getting 2 bytes per pixel, I get 3. And this is with the wrong IPU config case. As I said, for correct config I get blocks of either 0x00 or 0xFF, and the size of the blocks is not really consistent.

So, the main question stays - how does Bit 28 of IPU_CONF gate the data to CSI0 of IPU1? In order to continue, I think I really need this piece of information. Can you maybe direct me to someone who knows this?

Thanks in advance!

[max_tsai · Accepted solution]

Regarding IPU_CONF:B28, which decides sensor data comes from Parallel interface or MIPI/CSI2(via CSI2IPU gasket) according to RM, and it's a MUST to configure GPR1(MIPI_IPU1_MUX) of IOMUX for IPU0. Board configuration source (for ex, board_mx6q_sabresd.c) includes that already. Suppose you modified already based on your environment, but you may check again.

For MX6QD, IPUx/CSI0 can be connect to parallel interface or connect to mipi (via CSI2IPU gasket). Virtual channel of CSI2IPU is fixed as "IPU0/CSI0 VC0", "IPU0/CSI1 VC1", "IPU1/CSI0 VC2", and "IPU1/CSI1 VC3". Kernel board configuration source mentions virtual channel mapping. The mapping is fixed.

So CSI2IPU is no need to configure, there is only a concern about IPU_CONF:B28 and GPR1.

[ivankozic]

Hi Max,

Ok, I've abandoned the Still image capture concept as you've suggested - it seems that ipu_csi_enc.c is much more complete and better written than ipu_still.c (mipi_en is correctly handled here and IPU seems to be configured in the right way). I have added references to IPU_FMT_GENERIC earlier and I'm using the CSI->MEM path instead of default CSI->PRP_ENC->MEM path as I have no use of IC, since it cannot process Bayer.

However, I'm still facing the issue of mipi_en. I still didn't test it enough (hopefully today I'll have time for it), but it seems that it behaves the same way (when using the mipi config I get all zeros and somewhere 0xff). I'll verify and post back.