imx6 support for 8-bit RAW camera

[ambiguity]

Hi,

I am having trouble integrating our camera module & driver with the IPU.  I have the majority of the driver complete (probe succeeds, i2c comm is good, I see dma debug (errors though)), however, I believe I am running into a pixel formatting / dma setup issue with the IPU driver.  We expect (our User Layer app) to use V4L2_PIX_FMT_GREY, but this is "not supported".  While investigating this issue, I continued to add cases to the mxc_v4l2_capture layer to accept V4L2_PIX_FMT_GREY, but the problems get deeper into the IPU layer.

Does imx6 support 8-bit RAW camera data?  If so, what do I need to do?

Futher investigation into the IPU layer I discovered there is a "generic" format (or BAYER).  I investigated a little... I believe a certain format is expected (header info maybe?) - but our CMOS sensor is outputting raw pixel data so I am not sure how I could use this format.

Here is a snippet of my debug:

In MVC:mxc_poll

In MVC:mxc_v4l_ioctl

In MVC: mxc_v4l_do_ioctl c0445611

   case VIDIOC_DQBUF

In MVC:mxc_v4l_dqueue

ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

In MVC:mxc_v4l_ioctl

In MVC: mxc_v4l_do_ioctl 40045613

   case VIDIOC_STREAMOFF

In MVC:mxc_streamoff

_ipu_csi_wait4eof: channel = 335544276

_ipu_csi_wait4eof: IPU_IRQ_PRP_ENC_OUT_EOF

_ipu_csi_wait4eof: CSI stop timeout - 5 * 10ms

imx-ipuv3 imx-ipuv3.0: warning: disable ipu dma channel 20 during its busy state

Just before the ERROR the imager is capturing, but it just sits there an times out.  I believe the camera/csi is setup correctly (vsync_pol, hsync_pol, etc.).  I also see that the buffer(s) is (are) "queued".

Thanks!

[davidsoto]

Hi,

  I am interested on capturing 960x540 on gray scale but I would need to generate the pattern from an FPGA, this implies that I would need to know the timing specs. Could you let me know which camera you were using so I could use its datasheet as reference for the timing and to add this support to my system?

Regards,

-David

[Selea]

Important FIX:

in file ipu_still.c

static int prp_still_start(void *private)

{

    cam_data *cam = (cam_data *) private;

    u32 pixel_fmt;

    int err;

    ipu_channel_params_t params;

    pr_err("%s:ipu_channel_request still %d %d \n", __func__,cam->ipu_id,cam->csi);   

    if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_YUV420)

        pixel_fmt = IPU_PIX_FMT_YUV420P;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_NV12)

        pixel_fmt = IPU_PIX_FMT_NV12;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_YUV422P)

        pixel_fmt = IPU_PIX_FMT_YUV422P;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_UYVY)

        pixel_fmt = IPU_PIX_FMT_UYVY;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_YUYV)

        pixel_fmt = IPU_PIX_FMT_YUYV;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_BGR24)

        pixel_fmt = IPU_PIX_FMT_BGR24;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_RGB24)

        pixel_fmt = IPU_PIX_FMT_RGB24;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_RGB565)

        pixel_fmt = IPU_PIX_FMT_RGB565;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_BGR32)

        pixel_fmt = IPU_PIX_FMT_BGR32;

    else if (cam->v2f.fmt.pix.pixelformat == V4L2_PIX_FMT_RGB32)

        pixel_fmt = IPU_PIX_FMT_RGB32;

    else {

        printk(KERN_ERR "format not supported\n");

        return -EINVAL;

    }

    memset(&params, 0, sizeof(params));

    params.csi_mem.csi = cam->csi;  // OMAR : otherwise the CSI number is missing from here and forever

    err = ipu_channel_request(cam->ipu, CSI_MEM, &params, &cam->ipu_chan);

    if (err) {

        pr_err("%s:ipu_channel_request %d\n", __func__, err);

        return err;

    }

    err = ipu_init_channel_buffer(cam->ipu, CSI_MEM, IPU_OUTPUT_BUFFER,

                      pixel_fmt, cam->v2f.fmt.pix.width,

                      cam->v2f.fmt.pix.height,

                      cam->v2f.fmt.pix.width, IPU_ROTATE_NONE,

                      cam->still_buf[0], cam->still_buf[1], 0,

                      0, 0);

    if (err != 0)

        return err;

#ifdef CONFIG_MXC_IPU_V1

    ipu_clear_irq(IPU_IRQ_SENSOR_OUT_EOF);

    err = ipu_request_irq(IPU_IRQ_SENSOR_OUT_EOF, prp_still_callback,

                  0, "Mxc Camera", cam);

    if (err != 0) {

        printk(KERN_ERR "Error registering irq.\n");

        return err;

    }

    callback_flag = 0;

    callback_eof_flag = 0;

    ipu_clear_irq(IPU_IRQ_SENSOR_EOF);

    err = ipu_request_irq(IPU_IRQ_SENSOR_EOF, prp_csi_eof_callback,

                  0, "Mxc Camera", cam);

    if (err != 0) {

        printk(KERN_ERR "Error IPU_IRQ_SENSOR_EOF\n");

        return err;

    }

#else

    callback_eof_flag = 0;

    buffer_num = 0;

    ipu_clear_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF);

    err = ipu_request_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF,

                  prp_still_callback,

                  0, "Mxc Camera", cam);

    if (err != 0) {

        printk(KERN_ERR "Error registering irq.\n");

        return err;

    }

    ipu_select_buffer(cam->ipu, CSI_MEM, IPU_OUTPUT_BUFFER, 0);

    ipu_enable_channel(cam->ipu, CSI_MEM);

    cam_ipu_enable_csi(cam);

#endif

    return err;

}

[vinodmaverickr0]

Hi,

I have remove the camera from the IPU1. And also remove the entry for same(camera) from .dtsi file. Now I have only one node entry (/dev/video0). So it should be node for my sensor module.

Now when I run the gstreamer from command line I am getting the below errors:

root@imx6qsabresd:~# gst-launch mfw_v4lsrc device=/dev/video0 ! autovideosink

MAX resolution 1024x768

MFW_GST_V4LSRC_PLUGIN 3.0.11 build on Sep 27 2014 22:08:22.

Setting pipeline to PAUSED ...

MFW_GST_V4LSINK_PLUGIN 3.0

mxc_v4l2_s_param: vidioc_int_s_parm returned an error -22

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Total rendered:0

[--->FINALIZE v4l_sink

Freeing pipeline ...

[vinodmaverickr0]

Hi Omar,

I am able to capture the video on /dev/video0. My sensor worked on the NON-GATED clock mode. So I changed the CLK mode.

- csi_param.clk_mode = IPU_CSI_CLK_MODE_GATED_CLK;

+ csi_param.clk_mode = IPU_CSI_CLK_MODE_NONGATED_CLK;

Thanks for your suggestion and immense support.

[richc128]

I'm running into the same problem as I want a pixel format of V4L2_PIX_FMT_GREY for my custom camera.

Can you tell me what files besides the camera driver you touched? I see a pattern in mxc_v4l2_capture.c where I could add a new GREY format, I see IPU_PIX_FMT_GREY in a lot of files...do I need to edit all of them?

[ambiguity]

Hi Richard, the files that I modified are listed below:

..\drivers\media\video\mxc\capture\ipu_csi_enc.c

..\drivers\media\video\mxc\capture\mxc_v4l2_capture.c

..\drivers\mxc\ipu3\ipu_capture.c

..\drivers\mxc\ipu3\ipu_common.c

..\drivers\mxc\ipu3\ipu_param_mem.h

Basically, where there is IPU_PIX_FMT_GENERIC, add another case for IPU_PIX_FMT_GREY.  Also, in mxc_v4l2_capture.c:

-- You need to add support for V4L2_PIX_FMT_GREY if your camera driver is using/setting to that fmt. --

@@ -890,6 +995,10 @@ static int mxc_v4l2_s_fmt(cam_data *cam,

  size = f->fmt.pix.width * f->fmt.pix.height * 3 / 2;

  bytesperline = f->fmt.pix.width;

  break;

+ case V4L2_PIX_FMT_GREY:

+ size = f->fmt.pix.width * f->fmt.pix.height;

+ bytesperline = f->fmt.pix.width;

+ break;

  default:

  break;

  }

-- AND (per my previous post) -- :

Change mxc_capture_inputs to use "CSI MEM" depending on what input you are using.  I #ifdef'd CONFIG_xxx it for now.  If there is an easier way to do it... let me know.  I just haven't had time to search where this is actually set.

[tommyhu]

I want to add V4L2_PIX_FMT_GREY support in IMX6 3.0.35 kernel. Following ambiguity's guide, I modify below files:

..\drivers\media\video\mxc\capture\ipu_csi_enc.c

..\drivers\media\video\mxc\capture\mxc_v4l2_capture.c

..\drivers\mxc\ipu3\ipu_capture.c

..\drivers\mxc\ipu3\ipu_common.c

..\drivers\mxc\ipu3\ipu_param_mem

Further more, I modify the "CSI MEM" in mxc_capture_inputs. But when I capture the frame, the data size is correct (frame_width * frame_height), the image messed up (I have attached it). Basically I follow the way that where there is IPU_PIX_FMT_GENERIC, I add an IPU_PIX_FMT_GREY case, and I modify the mxc_v4l2_s_fmt in mxc_v4l2_capture.c too. Did I miss anything? thanks for help.

[aravinthkumarja]

Hi tommy hu,

Can you verify the camera initialization code.

Which camera driver you are using?

Regards,

Aravinth

[tommyhu]

Hi aravinthkumar,

     I am using OV3640 camera&driver. The default output format of OV3640 is UYVY, it works well if I capture with V4L2_PIX_FMT_UYVY. From the OV3640 datasheet, it also supports Y8 format, so I change the registry setting, modify the linux 3.0.35 kernel as I mention above, and capture with V4L2_PIX_FMT_GREY, the image messed up.

    If I only add V4L2_PIX_FMT_GREY support in ..\drivers\media\video\mxc\capture\, and remove the IPU changes in ..\drivers\mxc\ipu3\, the image looks much better, except that half the image is totally black. I am wondering if it's related to the changes in ..\drivers\mxc\ipu3\ipu_param_mem.h, especially _ipu_ch_param_init(), here is the code:

	switch (pixel_fmt) {
	case IPU_PIX_FMT_GENERIC:
	case IPU_PIX_FMT_GREY:
		/*Represents 8-bit Generic data */
		ipu_ch_param_set_field(&params, 0, 107, 3, 5);	/* bits/pixel */
		ipu_ch_param_set_field(&params, 1, 85, 4, 6);	/* pix format */
		ipu_ch_param_set_field(&params, 1, 78, 7, 63);	/* burst size */

break;

[vinodmaverickr0]

Hi tommy,

iMX6 support the grey scale code. Apart form selecting the pixel format to  IPU_PIX_FMT_GREY; I think you need to set other parameter like scaling and input parameters information.

[tommyhu]

Since no update yet, close this thread (disappointed). Finally I have to give up the GREY support, and working with UYVY with some tricky. It's still amazing that imx6q doesn't support  V4L2_PIX_FMT_GREY by default, which is so widely used.

[Selea]

I doing the same thing and your answer hope we will solve it for mu board. Can you please clarify what you mean with

"

-- AND (per my previous post) -- :

Change mxc_capture_inputs to use "CSI MEM" depending on what input you are using.  I #ifdef'd CONFIG_xxx it for now.  If there is an easier way to do it... let me know.  I just haven't had time to search where this is actually set.

"

thanks

Omar

[ambiguity]

Hi,

I believe I was referring to this portion of code in mxc_v4l2_capture.c:

static struct v4l2_input mxc_capture_inputs[MXC_V4L2_CAPTURE_NUM_INPUTS] = {

  {

  .index = 0,

#ifdef CONFIG_MXC_MYCAMERA

  .name = "CSI MEM",

#else

  .name = "CSI IC MEM",

#endif /*CONFIG_MXC_MYCAMERA*/

  .type = V4L2_INPUT_TYPE_CAMERA,

  .audioset = 0,

  .tuner = 0,

  .std = V4L2_STD_UNKNOWN,

  .status = 0,

  },

  {

  .index = 1,

  .name = "CSI MEM",

  .type = V4L2_INPUT_TYPE_CAMERA,

  .audioset = 0,

  .tuner = 0,

  .std = V4L2_STD_UNKNOWN,

  .status = V4L2_IN_ST_NO_POWER,

  },

};

I believe it has to do with the .index value... I just rolled with the above change for my own testing purposes.

[Selea]

Hi,

i followed you patch.

but still I have a strange image captured.

this is the pipe i'm using:

gst-launch mfw_v4lsrc num-buffers=1 device=/dev/video3 ! filesink location=/mnt/nfs/home/sample.raw

the sensor is a 1280x960 greyscale so the capture should be 1228800 bytes.

the raw i have is 1800000 bytes it seems to be as a 4:2:0....

take a look at it, using gimp opening it as raw 1280x960.

[Selea]

I'm really diggin iside CSI-IPU- chain.

I can say that there are some missing control and "case"... Mainly it works if the sensor come in as YUV422 and the output from the ipu is YUV420....

any other case (Bayer ...  grey scale... )  are not considered so they are not working proprely.

So i think the CSI ---> IPU ---> chain enned a "serious" patch to include the missing cases.... I think the most common cases could be sensor with Bayer output and greyscale output.

Omar

[Selea]

now i understood what you mean on

static struct v4l2_input mxc_capture_inputs[MXC_V4L2_CAPTURE_NUM_INPUTS] = {

i'm working on your patch, so maybe if i can do it i will send you my idea so you can check. it

[vinodmaverickr0]

Hi Omar,

Thanks for your reply.

In my DTSI file I have add:

v4l2_cap_2 {  // To make the video node for the device which is connecte to ipu2 with CSI1

          compatible = "fsl,imx6q-v4l2-capture";

          ipu_id = <1>;

          csi_id = <1>;

          mclk_source = <0>;

          status = "okay";

}

So it should create the node (/dev/video2) for IPU2 with CSI1. And my sensor device is attached on CSI1 of IPU2. So It should capture the data from video2 node and then send it to either display or memory.

As my sensor is continuously sending the data (which I have verified by probe the data line on oscilloscope). But I am not able to even capture the raw data, which is sending by the sensor.

I have not connected the data enable pin because there is no data enable pin in sensor side. So you are asking to enable it from the processor side. And if we will add it in the pin multiplexing. then how it will effect the working.

One more thing there are only 7 data lines, a HSYNC, VSYNC, and one pixel clock.

P.S. My sensor only support the Bayer RGB data format.

[Selea]

plase take a look at the files included here: (the dts and dtsi)

IPU2 parallel port on IMX6Q

this is my setup, is close to work... bu at least i see the dev/video3  (in my case) and i can acquire (forget the AR0134 driver... that is an old version with errors, i will upload the new one soon...)

[vinodmaverickr0]

Hi,

I have also implemented the things in same way (which I have posted in earlier post).

But I have some doubts:

1) Is DATA_ENABLE pin must or not (because there is no data enable pin in my sensor). Right now data enable pin is low.

2) What should be the path for Bayer data (from CSI->IC->MEM or CSI->MEM).

Please upload the patch, when it will work.

[Selea]

the pin should be high otherwise tohave tochange it' spolarity in the IPU_IC_CONF

but this is not the reason why you do not see the device. ...

The strange is the ERROR "slave not found"

even if the data_en is wrong you should have read timeout error (the one I had) but the dev should be present...

even the path shoul not care.. but for the bayer the right one depends on what you want to do...

you can use CSI-> MEM so you will have the raw in memory, then maybe you have to do a color conversion...

but the first thing you have to fix is to have the V4L2 dev working and present... even if it will not be really acquiring.

Omar