i.mx6：two camera work at the same time

[GaoJianzhong]

Hi,

I am using i.mx6q SD board to get two camera(ov5640 and ov3640) work at the same time.

It is no need to display the image on display, i just want to get the camera datas in memory and

save them to local files.

To confirm is it support two camera at the same time, I do the follow things:

1. Put the OV5640 camera image on display with mxc_v4l2_overlay.

2. then capture the OV3640 data to local file with use mxc_vpu_test

But i got following error when i do second step:

imx-ipuv3.0: handler already installed on irq 0

So i checked the drivers,and found that there is only CSI_MEM(which is CSI_MEM0),

I think that if want use two cameras at the same time,

It is need to add a CSI_MEM1  to let second camera data use CSI_MEM1 channel to memory.

Am i right? and is there any ideas to save two camera data to memory at same time?

Thank you.

[gaozhihe]

附件为3.14.28版本内核解决此问题补丁。

[LeonardoSandova]

Hi Gao,

Unfortunately, the system can not handle two captures and two video sinks simultaneously. But in your case is different, you want to just capture + [encode] + save, right? before digging into the code, try to use GStreamer, which can be seen as a wrapper to codecs. Try these pipelines :

gst-launch mfw_v4lsrc device="/dev/video0" ! filesink location=camera0.yuv &

gst-launch mfw_v4lsrc device="/dev/video1" ! filesink location=camera1.yuv

In case you need to stored encoded data, just add vpuenc between the src and the sink. BTW, before running these, make sure you have the devices nodes (/dev/video[0|1]) and the modules loaded (modprobe ...).

Leo

[GaoJianzhong]

Hi Leo,

Thanks for your help.

I tried the way you said. But the error is the same:

imx-ipuv3 imx-ipuv3.0: handler already installed on irq 0

Error registering rot irq

It seems that BSP only support the CSI_MEM0,

Now I added a driver use CSI_MEM1, and it is worked now.

And another question,

I want to get following worked:

Can gst-launch do this?

Thank you.

[edison_fernande]

Hi Gao,

I'm using a Sabre SDP board and i want to use both cameras at the same time and i realized that you were able to make them work and i wonder if you could guide me a little bit through the steps you followed to have the 2 cameras working at the same time. 

Regards,

Edison

[GaoJianzhong]

How can i attach the source code to you ?

[edison_fernande]

Hi Gao,

I'm now able to use both cameras (i modified the ipu_csi_enc.c so the driver uses a different DMA channel for each camera) but when i'm displaying both channels one of them has some interference. I would like to compare what i'm doing with what you did. I don't know if you can attach source code here but if not my email is edison.fernandez@ridgerun.com.

Thanks,

Edison

[edison_fernande]

Hi,

I'm able to enable both cameras at the same time by using the following patch:

Index: kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/ipu_csi_enc.c

===================================================================

--- kernel.orig/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/ipu_csi_enc.c 2013-10-07 14:42:58.927083104 -0600

+++ kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/ipu_csi_enc.c 2013-10-07 14:44:19.031082649 -0600

@@ -56,6 +56,24 @@

}

/*!

+ * Get CSI dma channel

+ * @param csi       csi number

+ *

+ * @return  DMA channel

+ */

+static ipu_channel_t csi_get_dma_channel(int csi)

+{

+ switch(csi){

+ case 0:

+ return CSI_MEM0;

+ case 1:

+ return CSI_MEM1;

+ default:

+ return CSI_MEM0;

+ }

+}

+

+/*!

  * CSI ENC enable channel setup function

  *

  * @param cam       struct cam_data * mxc capture instance

@@ -74,6 +92,8 @@

  int csi_id;

#endif

+ uint32_t chan = csi_get_dma_channel(cam->csi);

+

  CAMERA_TRACE("In csi_enc_setup\n");

  if (!cam) {

  printk(KERN_ERR "cam private is NULL\n");

@@ -162,13 +182,13 @@

  }

#endif

- err = ipu_init_channel(cam->ipu, CSI_MEM, &params);

+ err = ipu_init_channel(cam->ipu, chan, &params);

  if (err != 0) {

  printk(KERN_ERR "ipu_init_channel %d\n", err);

  return err;

  }

- err = ipu_init_channel_buffer(cam->ipu, CSI_MEM, IPU_OUTPUT_BUFFER,

+ err = ipu_init_channel_buffer(cam->ipu, chan, IPU_OUTPUT_BUFFER,

       pixel_fmt, cam->v2f.fmt.pix.width,

       cam->v2f.fmt.pix.height,

       cam->v2f.fmt.pix.bytesperline,

@@ -180,7 +200,7 @@

  printk(KERN_ERR "CSI_MEM output buffer\n");

  return err;

  }

- err = ipu_enable_channel(cam->ipu, CSI_MEM);

+ err = ipu_enable_channel(cam->ipu, chan);

  if (err < 0) {

  printk(KERN_ERR "ipu_enable_channel CSI_MEM\n");

  return err;

@@ -197,18 +217,20 @@

  *

  * @return  status

  */

-static int csi_enc_eba_update(struct ipu_soc *ipu, dma_addr_t eba, int *buffer_num)

+static int csi_enc_eba_update(unsigned int csi, struct ipu_soc *ipu, dma_addr_t eba, int *buffer_num)

{

  int err = 0;

+ uint32_t chan = csi_get_dma_channel(csi);

+

  pr_debug("eba %x\n", eba);

- err = ipu_update_channel_buffer(ipu, CSI_MEM, IPU_OUTPUT_BUFFER,

+ err = ipu_update_channel_buffer(ipu, chan, IPU_OUTPUT_BUFFER,

  *buffer_num, eba);

  if (err != 0) {

- ipu_clear_buffer_ready(ipu, CSI_MEM, IPU_OUTPUT_BUFFER,

+ ipu_clear_buffer_ready(ipu, chan, IPU_OUTPUT_BUFFER,

        *buffer_num);

- err = ipu_update_channel_buffer(ipu, CSI_MEM, IPU_OUTPUT_BUFFER,

+ err = ipu_update_channel_buffer(ipu, chan, IPU_OUTPUT_BUFFER,

  *buffer_num, eba);

  if (err != 0) {

  pr_err("ERROR: v4l2 capture: fail to update "

@@ -217,7 +239,7 @@

  }

  }

- ipu_select_buffer(ipu, CSI_MEM, IPU_OUTPUT_BUFFER, *buffer_num);

+ ipu_select_buffer(ipu, chan, IPU_OUTPUT_BUFFER, *buffer_num);

  *buffer_num = (*buffer_num == 0) ? 1 : 0;

@@ -234,6 +256,7 @@

{

  cam_data *cam = (cam_data *) private;

  int err = 0;

+ uint32_t irq;

  CAMERA_TRACE("IPU:In csi_enc_enabling_tasks\n");

  cam->dummy_frame.vaddress = dma_alloc_coherent(0,

@@ -250,8 +273,10 @@

     PAGE_ALIGN(cam->v2f.fmt.pix.sizeimage);

  cam->dummy_frame.buffer.m.offset = cam->dummy_frame.paddress;

- ipu_clear_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF);

- err = ipu_request_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF,

+ irq = IPU_IRQ_CSI0_OUT_EOF + cam->csi;

+

+ ipu_clear_irq(cam->ipu, irq);

+ err = ipu_request_irq(cam->ipu, irq,

       csi_enc_callback, 0, "Mxc Camera", cam);

  if (err != 0) {

  printk(KERN_ERR "Error registering rot irq\n");

@@ -283,9 +308,11 @@

  int csi_id;

#endif

- err = ipu_disable_channel(cam->ipu, CSI_MEM, true);

+ uint32_t chan = csi_get_dma_channel(cam->csi);

+

+ err = ipu_disable_channel(cam->ipu, chan, true);

- ipu_uninit_channel(cam->ipu, CSI_MEM);

+ ipu_uninit_channel(cam->ipu, chan);

  if (cam->dummy_frame.vaddress != 0) {

  dma_free_coherent(0, cam->dummy_frame.buffer.length,

@@ -341,7 +368,8 @@

  /* free csi eof irq firstly.

  * when disable csi, wait for idmac eof.

  * it requests eof irq again */

- ipu_free_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF, cam);

+

+ ipu_free_irq(cam->ipu, IPU_IRQ_CSI0_OUT_EOF + cam->csi, cam);

  return ipu_disable_csi(cam->ipu, cam->csi);

}

Index: kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.c

===================================================================

--- kernel.orig/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.c 2013-10-07 14:42:58.939083104 -0600

+++ kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.c 2013-10-07 14:43:15.471083010 -0600

@@ -429,7 +429,7 @@

  list_del(cam->ready_q.next);

  list_add_tail(&frame->queue, &cam->working_q);

  frame->ipu_buf_num = cam->ping_pong_csi;

- err = cam->enc_update_eba(cam->ipu, frame->buffer.m.offset,

+ err = cam->enc_update_eba(cam-> csi, cam->ipu, frame->buffer.m.offset,

   &cam->ping_pong_csi);

  frame =

@@ -437,7 +437,7 @@

  list_del(cam->ready_q.next);

  list_add_tail(&frame->queue, &cam->working_q);

  frame->ipu_buf_num = cam->ping_pong_csi;

- err |= cam->enc_update_eba(cam->ipu, frame->buffer.m.offset,

+ err |= cam->enc_update_eba(cam-> csi, cam->ipu, frame->buffer.m.offset,

    &cam->ping_pong_csi);

  spin_unlock_irqrestore(&cam->queue_int_lock, lock_flags);

  } else {

@@ -2615,7 +2615,7 @@

  struct mxc_v4l_frame,

  queue);

  if (cam->enc_update_eba)

- if (cam->enc_update_eba(cam->ipu, ready_frame->buffer.m.offset,

+ if (cam->enc_update_eba(cam->csi, cam->ipu, ready_frame->buffer.m.offset,

  &cam->ping_pong_csi) == 0) {

  list_del(cam->ready_q.next);

  list_add_tail(&ready_frame->queue,

@@ -2625,7 +2625,7 @@

  } else {

  if (cam->enc_update_eba)

  cam->enc_update_eba(

- cam->ipu, cam->dummy_frame.buffer.m.offset,

+ cam->csi, cam->ipu, cam->dummy_frame.buffer.m.offset,

  &cam->ping_pong_csi);

  }

Index: kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.h

===================================================================

--- kernel.orig/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.h 2013-10-07 14:42:58.963083104 -0600

+++ kernel/linux-3.0.35-4.0.0/drivers/media/video/mxc/capture/mxc_v4l2_capture.h 2013-10-07 14:43:15.471083010 -0600

@@ -178,7 +178,7 @@

  struct v4l2_rect crop_defrect;

  struct v4l2_rect crop_current;

- int (*enc_update_eba) (struct ipu_soc *ipu, dma_addr_t eba, int *bufferNum);

+ int (*enc_update_eba) (unsigned int csi, struct ipu_soc *ipu, dma_addr_t eba, int *bufferNum);

  int (*enc_enable) (void *private);

  int (*enc_disable) (void *private);

  int (*enc_enable_csi) (void *private);

but when i'm displaying at 480p i get some interference in both channels:

The output looks better when using a 720p resolution:

I'm using kernel 3.0.35-4.0.0.

Any help will be appreciated!!

Regards,

Edison

[makislivadas]

I experienced same issue with my board after modifying the driver as per similar thread below:

OV5640 and OV5642 cameras on iMX6

I have now upgraded my sources to L3.0.35_4.1.0, which is the latest kernel and drivers.

Clearly Freescale has added more changes to the drivers. Unfortunately ipu_csi_enc.c (driver) is now worse off.

gestreamer seems to use this driver (the way I use it)  to capture video from the cameras and before modifying it I could capture with the previous release L3.0.35_4.0.0 with one camera at the time. In current release once one capture is enabled the next time I try to capture I get te irq error mentioned here even if I stop the previous capture session. Clearly some sort of regression.

However, when the capture software uses ipu_prp_enc.c driver, the behaviour is OK. Both cameras can be used but not at the same time of course. It seems that the only way we can get 2 cameras at the same time without having to modified the release software is by using ipu_csi_enc.c (driver) for one camera and  ipu_prp_enc.c (driver) for the other. Is there a way to do this with gstreamer?

I have not managed to get the cameras to work yet with the latest kernel, but the display artifacts when 2 camera are used are an issue. I found that if I swap the two cameras around the artifacts are less prominent. Is the fact that on the SabreSD board there is a MIPI and a CSI camera used which do not have the same interface that causes the artifacts?

[edison_fernande]

Hi,

Regarding your Gstreamer question, if you take a look at the mfw_v4lsrc element you'll see a property called input. Setting that property to 0 or 1 you are able to use CSI -> MEM or CSI -> IC -> MEM. As i mentioned before, you can do it with a pipeline like this one:

gst-launch -e mfw_v4lsrc input=0 ! 'video/x-raw-yuv,width=176,height=144' ! vpuenc codec=6 ! matroskamux ! filesink location=test0.mkv mfw_v4lsrc device=/dev/video1 input=1 ! vpuenc codec=6 ! matroskamux ! filesink location=test1.mkv

That way one capture is using ipu_csi_enc and the other one is using prp_enc. With that pipeline i'm able to capture from both cameras at the same time but it still has some interference, mainly the one coming from the ov5642 (see test0.mkv in my previous message).

Hope this helps.

Regards,

Edison

[makislivadas]

Hi,

I tried this script and it seem to behave like you described. One of the video shows artifact.

However, tried to modify the script to allow me to display while capturing and that does not work for me. In particular setting input=0 does not work when capture and display is selected in gstreamer as follows:

gst-launch mfw_v4lsrc device=/dev/video0 input=0 ! mfw_v4lsink

So it appears that capture and display with two cameras and two display may no work without hacking the drivers at the moment.
Regards,

Makis

[rogerio_silva]

Hi,

I finished the driver update to enable both cameras at the same time. It is decribed on this link: Displaying two cameras at the same time on i.MX6Q

Rgds

Rogerio

[jwatts]

Roger, I am trying to reproduce the above dual-camera success on the Sabre SDP (our own hardware must eventually going to support 4 simultaneous inputs). The link you posted gives me an "Unauthorized" error. Are the contents under NDA?

[rogerio_silva]

Hi Josh,

No need NDA. I removed this page because we noticed flickering when enabling both cameras at the same time.

When I find the problem I'll publish it again.

How do you plan to connect 4 cameras at the same time? Using MIPI?

[jwatts]

Ultimately yes, but until our custom hardware is design I'm using the Sabre SDP to do proof-of-concept work, so for now I'm stuck using one parallel input and one MIPI input. The latter I hope to "virtualize" so the system thinks there are 3 devices on separate virtual channels, and then (manually) manipulate the imager's VC to get a handful of frames on each channel. This should theoretically prove at least 2x simultaneous capture and that I've correctly mapped all 4 channels.

I've just started to dig into the relevant driver source to see about adding support for additional channels: The above patch by edison.fernandez is definitely helpful in getting started, but I would greatly appreciate any additional insight/examples.

[rogerio_silva]

Hi Josh,

I'm sending attached the patch for dual camera. As I said, the cameras are flickering and I didn't check the root cause yet. This is a non official Freescale patch.

It was applied on kernel 3.10.17 and can be tested using the following unit test:

/unit_tests/mxc_v4l2_overlay.out -t 200 -di /dev/video19 & /unit_tests/mxc_v4l2_overlay.out -t 200 -fg -ot 200 -ol 200 -di /dev/video20 &

Best Regards,

Rogerio

[richogrady]

Hello Rogerio,

I've used this patch with some good behavior and some questions.  If I run it with overlay.out as you post with one foreground and 1 background, it works.  If I try to run both background it appears to work but I get flooded with IPU WARNINGS .  If I try to run both foreground, the first camera image seems to disappear and again I get flooded with IPU WARNINGS.  

   imx-ipuv3 imx-ipuv3.0: IPU Warning - IPU_INT_STAT_10 = 0x00000001

   or imx-ipuv3 imx-ipuv3.0: IPU Warning - IPU_INT_STAT_10 = 0x00000002

I've tried in 3.0.35 and in 4.1.15.  It seems like the video stuff hasnt evolved much.  How can I correct this behavior?

[rogerio_silva]

Hi Rich,

This is caused because V4L2 driver is trying to access twice the same resource (the background framebuffer) and it causes conflict.

To handle multiple videos in the same framebuffer, I would recommend to use gstreamer. Please, see the chapter "Video Composition" in the document "i.MX_Linux_User's_Guide.pdf" - This document is available in the linux documentation package:

https://www.nxp.com/webapp/Download?colCode=L4.1.15_2.0.0-LINUX-DOCS&Parent_nodeId=13376371545356958... 

Best regards,

Rogerio

[richogrady]

Rogerio,

Thanks for the feedback.  I've noticed 2 types of v4l2 camera drivers.  One kind is v4l2_int_device and v4l2_subdev.  Can either work with gstreamer?

[rogerio_silva]

Hi Rich,

v4l2_int_device was made to work on NXP kernel. It's recommend if you're using this kernel. I'm not sure if NXP kernel works with v4l2_subdev.

v4l2_subdev is the driver from mainline kernel (from kernel.org). 

Rgds

Rogerio

[jwatts]

A very belated thank you for the patch. I just got back to working on this, and it works as expected including the flickering, which appears to be a signal cross-talk issue between the digital MIPI traces being near the parallel traces (my $0.02). Once I had your patch working, I had to make additional changes to support our end-goal of 4 cameras. I've not yet tested the 3rd and 4th inputs, but the 1st and 2nd work just as well as with your original patch, so they all ought to work equally well. See attached for the updated patch. :smileyhappy: