OV5640 and OV5642 cameras on iMX6

[vsv]

Hi all,

We are trying to use OV5640 (MIPI interface) and OV5642 in our custom iMX6 based board as rear camera and as a front camera respectively. Are these two cameras capable of coexisting on a single board based on iMX6 processor?

Thanks for your inputs,

Vasan

[konstantynproko]

Here is the patch which enables two sensors: MIPI and parallel working at the same time.

The kernel is used: Boundary Devices Linux kernel with their patch ver. 3.0.35

Git:https://github.com/boundarydevices/linux-imx6.git

I was using BoundaryDevices development board based on SabreLite design with IMX6 processor quad-core and dual-lite-core. For our purposes quad-core was different in adding second IPU and enabling virtually 4 sensors to be attached through: IPU0-CSI0/CSI1 and IPU1-CSI0/CSI1

In my configuration both sensors are connected to IPU0 only. OV5642 (parallel data interface) to CSI0 and OV5640 (MIPI) to CSI1.

Board support module: arch/arm/mach-mx6/board-mx6_nitrogen6x.c

In .config, enable MIPI support and OV5640_mipi and OV5642

The patch can be used as reference to extract some portions for other boards and configurations.

Also you may notice, I added another memory type for MXC V4L layer: DMAMAP. This option remaps physical address to be used by DMA for frame buffers. I'm reserving some memory while kernel booting and using it to map my own frame ring-buffers. This way it is faster and I can be sure that contiguous space is always available without dynamically searching for it.

Good luck.

Regards,

Konstantyn

[makislivadas]

Hi All,

I had a go at modifying as per Konstantyn suggestion the the icu_csi_enc.c file from my LTIB BSP and managed to get a version that can now run both cameras concurrently. Did not attempt to midify any other files.

I have attached the patch that shows the mods.

I am using gstreamer to capture the data.

There seems to be a slight issue though. I am using two LVDS screens with resolution 768x1024 and when I capture with both camera I am getting some sort of flicker. I wonder if this is a performance related issue. I am using SabreSD board with imx6Q processor, two on board camera (MIPI and CSI)

Makis

[konstantynproko]

Hello Vasan,

I've had experience of configuring and running several cameras on MX6 using SabreLite clone boards from BoundaryDevices.

I was able to configure and program the following sensor configurations:

1. Two Parallel OV5642 sensors on different I2C bus.

2. One Parallel OV5642 and one MIPI OV5640 sensors

Sensors are configured for MX6 dual core lite and MX6 quad core

I've just finished working on integrating our own application and image conversion with MX6 hardware CSI/IPU/VPU

If you have more questions about image conversion, compression, I'll be glad to help also.

Tomorrow I'm going to prepare patch for two parallel OV5642 sensors for boundary devices kernel and send it to you. You can download BoundaryDevices kernel from their website.

Also, just to look in a meantime:

1. Check board support package arch/arm/mach-mx6/board-mx6_[your board]

   - Check for ov5642 initialization init/power functions.

   - Check I2C bus initialization for ov5642. I'm using separate I2C busses (2 and 3) to avoid ID collision.

   - Check "capture_data" structure for initialization of sensor capture info such as IPU/CSI ids, clock.

  - Check pads-mx6_...... pads confguration for your board to make sure all sensor pads configured correctly.

2. Check IPU configuration in drivers/mxc/ipu3

   - Make sure you enable proper channels for CSI0 and CSI1 either parallel or MIPI. (ipu_common.c)

3. MXC V4L and CSI Encoder drivers (mxc_v4l2_capture, ipu_csi_enc) in drivers/media/video/mxc/capture

   - I've duplicated ipu_csi_enc encoder driver to work with different memory channels for several sensors.

   - In a nutshell, for several sensors to work in parallel I've separated memory channels and IRQs by duplicating encoder driver. There is probably other way to do it but that was easiest.

   - I would suggest to use IPU streaming all the time switching modes in run-time, I couldn't get IPU_still working for both sensors. Also I added more working resolutions to OV5642 initialization tables. Now I've got 2592x1944, 1600x1200, 1024x768, 800x600, 1080p, 720p, 640x480 and 320x240 working and I can dynamically switch between them.

   - Also for MXC V4L layer I've implemented another memory allocation model called DMAMAP which is direct physical memory map for DMA operations. The physical memory area is reserved during kernel boot and used by kernel drivers to map DMA buffers without dynamic allocations. Note that raw YUYV 2592x1944 x 16bits is close to 10MBytes and it is painful to allocate contiguous space for DMA in run-time.

Also I'm doing color conversion in IPU hardware and VPU encoding of still JPEGs and motion JPEG. Now I'm working on H.264 video stream from both sensors.

Regards,

Konstantyn

[Raana]

Hi Konstantyn,

Thanks for your detailed explanation.

I was able to get the OV5640 MIPI and OV7725 working individually on our custom i.Mx6Q board. But the problem is, both a camera not working at the same time on different display (LCD and HDMI).

If I tried to run overlay application for both a cameras, I also faced the same issue as mentioned as LeonardoSandovalGonzalez's post (imx-ipuv3 imx-ipuv3.0: handler already installed on irq 0)

I got patch file in the i.mx6：two camera work at the same time link. But that also not helpful for me.

Please help me to do further.

Thanks in Advance.

Best Regards,

J.P.Raja

[konstantynproko]

Hello J.P. Raja,

I think the problem with your IRQ handler is that you are trying to reuse the same end-of-frame IRQ for both sensor channels. Request IRQ for IPU3 is called from capture encoding module: drivers/media/video/mxc/capture/ipu_csi_enc.c in csi_enc_enabling_tasks() function

I've resolved it by duplicating ipu_csi_enc.c into ipu_csi0_enc and ipu_csi1_enc modules. In each module I've changed:

1. CSI_MEM to CSI_MEM0 and CSI_MEM1 respectively

2. IPU_IRQ_CSI0_OUT_EOF to IPU_IRQ_CSI0_OUT_EOF and IPU_IRQ_CSI1_OUT_EOF

3. Exported function to initialize encoders: csi_enc_select to csi0_enc_select and csi1_enc_select

OK, now in MXCV4L  mxc_v4l2_capture.c module I've added to open() call:

==========================================================================

                if (strcmp(mxc_capture_inputs[cam->current_input].name,

                           "CSI MEM 0") == 0) {

#if defined(CONFIG_MXC_IPU_CSI_ENC) || defined(CONFIG_MXC_IPU_CSI_ENC_MODULE)

                        err = csi0_enc_select(cam);

#endif

                }

                else if (strcmp(mxc_capture_inputs[cam->current_input].name,

                           "CSI MEM 1") == 0) {

#if defined(CONFIG_MXC_IPU_CSI_ENC) || defined(CONFIG_MXC_IPU_CSI_ENC_MODULE)

                        err = csi1_enc_select(cam);

#endif

                } else if (strcmp(mxc_capture_inputs[cam->current_input].name,

                                  "CSI IC MEM") == 0) {

#if defined(CONFIG_MXC_IPU_PRP_ENC) || defined(CONFIG_MXC_IPU_PRP_ENC_MODULE)

                        err = prp_enc_select(cam);

#endif

                }

=============================================================================

In the same module I've modified capture input structure to identify proper name for the input ID (0 or 1)

static struct v4l2_input mxc_capture_inputs[MXC_V4L2_CAPTURE_NUM_INPUTS] = {

        {

         .index = 0,

         .name = "CSI MEM 0",

         .type = V4L2_INPUT_TYPE_CAMERA,

         .audioset = 0,

         .tuner = 0,

         .std = V4L2_STD_UNKNOWN,

         .status = 0,

         },

        {

         .index = 1,

         .name = "CSI MEM 1",

         .type = V4L2_INPUT_TYPE_CAMERA,

         .audioset = 0,

         .tuner = 0,

         .std = V4L2_STD_UNKNOWN,

         .status = V4L2_IN_ST_NO_POWER,

         },

};

There are probably better ways to implement this by providing channel ID to the encoder module, etc... but this was faster for me.

Hope this helps.

I'm preparing patch soon and will post it here. It has all the modifications and new modules

Regards,

Konstntyn

[ct3paul]

Hi Konstntyn

My two csi camera also can't work when I do like you said.

My camera chip is ADV7180.It can work well when I loaded one chip driver.

But when I add two devices to i2c struct like this:

static struct i2c_board_info mxc_i2c0_board_info[] __initdata = {

  #if 1

  {

  I2C_BOARD_INFO("adv7180", 0x20),

  .platform_data = (void *)&adv7180_data,

  },

  #endif

#if 1

  {

  I2C_BOARD_INFO("adv7180", 0x21),

  .platform_data = (void *)&camera2_data,

  },

  #endif

};

I can't load my driver.I enter a loop and I cat't break it when I loadind drivers if I add two devices to i2c struct.

My drivers:

insmod ipu_bg_overlay_sdc.ko

insmod ipu_csi_enc.ko

insmod ipu_fg_overlay_sdc.ko

insmod ipu_prp_enc.ko

insmod ipu_still.ko

insmod adv7180_tvin.ko

insmod mxc_v4l2_capture.ko

Thanks

Chen

[makislivadas]

Hi Konstanyn,

The proposed changes to the mxc_v4l2_capture.c seem to diable the driver function prp_enc_select(cam). This function seems to be similar to csi_enc_select(cam), and can be used to capture camera input. Not sure what the differences are exactly, but it looks that the function is more difficult to modify to support two camera's concurrently.

One artefact I get from these changes moving support from prp_enc_select() to csi_enc_select() is that the color of the captured image has a distinctive red  tone which is not correct. This happens with camera capture software that tries to use prp_enc_select() but now uses csi_enc_select() instead. Camera capture software that uses csi_enc_select() does work fine even after the modes.

Makis

[vsv]

Hi Konstantyn,

Thanks for your detailed response.

I was able to get the OV5640 and OV5642 working on our custom iMX6 board. The major problem was that the CSI ids for each of these cameras was referred incorrectly in the code because of which it refused to get recognised. There were some other minor issues that also had to be taken care of.

Regards

Vasan

[LeonardoSandova]

Yes. Both cameras can work concurrently. You can easily do the test on a sabre sd board just by loading both cameras modules and running two gstreamer pipelines with following elements each: mfw_v4lsrc device=/dev/videoX ! vpuenc ! fakesink. More info on the MM User Guide.

[1990rahulreddy]

i know "i.MX_Linux_User's_Guide", please post MM User Guide@

[makislivadas]

I tried using two cameras on the sabreSD board concurently and is not working for me.

Below is what I tried from the command line. Running first camera is OK, no error reported. When I then tried to run the second camera while the first camera is still working I get the error see below.

------------------------------------------------------------------------------------------------------

root@freescale /home/user$ gst-launch mfw_v4lsrc device=/dev/video0 ! vpuenc ! f

akesink &

root@freescale /home/user$ MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Sep 17 2013 17:27:44.

Setting pipeline to PAUSED ...

[INFO]  Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

        plugin: 3.0.7

        wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Sep 17 2013 17:26:13)

        vpulib: 5.4.12

        firmware: 2.1.9.36350

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

[INFO]  chromaInterleave 0, mapType 0, linear2TiledEnable 0

root@freescale /home/user$ gst-launch mfw_v4lsrc device=/dev/video1 ! vpuenc ! f

akesink &

root@freescale /home/user$ MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Sep 17 2013 17:27:44.

Setting pipeline to PAUSED ...

[INFO]  Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

        plugin: 3.0.7

        wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Sep 17 2013 17:26:13)

        vpulib: 5.4.12

        firmware: 2.1.9.36350

imx-ipuv3 imx-ipuv3.0: handler already installed on irq 0             <-- This is error indicating someting is not working.

Error registering rot irq

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

[LeonardoSandova]

I have reproduced the issue on latest BSP, there is indeed a IRQ issue. The problem is that at GST level, running both on-board cameras simultaneously is not possible due to an IRQ conflict. The issue has been escalated internally, so someone @ FSL should update this thread soon.

root@freescale ~$ gst-launch mfw_v4lsrc ! fakesink &

root@freescale ~$ MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Aug 16 2013 20:15:40.

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

root@freescale ~$ gst-launch mfw_v4lsrc device=/dev/video1 ! fakesink

MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Aug 16 2013 20:15:40.

Setting pipeline to PAUSED ...

imx-ipuv3 imx-ipuv3.0: handler already installed on irq 0

Error registering rot irq

ERROR: Pipeline doesn't want to pause.

Setting pipeline to NULL ...

Freeing pipeline ...

root@freescale ~$

[vsv]

Did you check up the CSI id that was used for OV5640 and OV5642 definitions inside the board specific file?

As per the BSP document from Freescale, these have to be two different CSI ids and has to be consistently used across the entire code base. Tomorrow morning, I will add up a note here on what all changes that I had to do on our custom Sabrelite board based on iMX6 to get OV5640 and OV5642 working together.

[LeonardoSandova]

First pipeline should have been run with /dev/video1 as device, right?

Leo