i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

Jump to solution
40,582 Views
davemcmordie
Contributor III

Hello,

I have implemented an OV5647 camera board which connects to my i.MX6 quad Wandboard.  I am having trouble getting images from the driver and need some help figuring out why.  I have reviewed about a dozen related threads on this forum, and tried to carefully implement their suggestions.

Here are the steps I have taken so far:

  1. Started with working kernel (based on John Weber's Yocto Kernel) and board which is known to work with OV5640 (YUV sensor) in this configuration.  rootfs is Linaro armhf with desktop ubuntu (so I can use VLC and other known tools for testing).
  2. Made sure that the BSP sets GPR1 19 to 0 (select MIPI CSI interface to IPU0 CSI 0), and that the camera driver is detected by I2C.
  3. Built a driver, OV5647_mipi.c, based on OV5640_mipi.c with the following changes:
    1. Replaced modes (register programming code) with code appropriate to generate a 1280x960 30 fps preview on the OV5647
    2. Changed the chip_id high and low bytes to 0x56 and 0x47 respectively
    3. Changed ov5647_data.pix.pixelformat to V4L2_PIX_FMT_SBGGR10
    4. Added code to detect this and call mipi_csi2_set_datatype with MIPI_DT_RAW10
    5. Tried steps 3, 4 with GREY and RAW10
  4. Added support for V4L2_PIX_FMT_SBGGR10 and V4L2_PIX_FMT_GREY to
    1. drivers\media\video\mxc\capture\ipu_csi_enc.c
    2. drivers\media\video\mxc\capture\mxc_v4l2_capture.c
    3. drivers\mxc\ipu3\ipu_capture.c
    4. drivers\mxc\ipu3\ipu_common.c
    5. drivers\mxc\ipu3\ipu_param_mem.h
  5. Tested
    1. Sensor is correctly detected on I2C
    2. Driver reports MIPI clock stabilized okay
    3. Driver reports data being received from sensor


I have to guess that what is going on is that the image format is not being handled correctly, but I can't tell from the tracing.


Some specific questions:


  1. How can I verify that I am talking to the correct IPU, CSI port and virtual channel?
  2. Despite the driver's claim that it is receiving data, is it possible that this is a signal integrity issue?  I have not been able to successfully probe the lanes (I see a ton of common mode noise)
  3. Are there steps I have missed to support a 10 bit Bayer sensor just as grayscale?
  4. What are the likely reasons why it is not getting frame interrupts?
  5. Help!   :smileyhappy:


The following is what I get with DEBUG tracing on in the drivers:


[ 1160.505314] End of mxc_v4l2_g_fmt: crop_current widthxheight 1280 x 960

[ 1160.505352] In MVC:mxc_v4l_ioctl

[ 1160.505360] In MVC: mxc_v4l_do_ioctl c0405602

[ 1160.505373] In MVC:mxc_v4l_ioctl

[ 1160.505380] In MVC: mxc_v4l_do_ioctl c0405602

[ 1160.505458] In MVC:mxc_v4l_ioctl

[ 1160.505465] In MVC: mxc_v4l_do_ioctl c0445624

[ 1160.505472]    case default or not supported

[ 1160.505528] In MVC:mxc_v4l_ioctl

[ 1160.505535] In MVC: mxc_v4l_do_ioctl c02c563a

[ 1160.505540]    case VIDIOC_CROPCAP

[ 1160.505547] In MVC:mxc_v4l_ioctl

[ 1160.505552] In MVC: mxc_v4l_do_ioctl 4014563c

[ 1160.505561]    case VIDIOC_S_CROP

[ 1160.505567]    Cropping Input to ipu size 1280 x 960

[ 1160.505678] In MVC:mxc_v4l_ioctl

[ 1160.505686] In MVC: mxc_v4l_do_ioctl c0145608

[ 1160.505693]    case VIDIOC_REQBUFS

[ 1160.505697] In MVC:mxc_streamoff

[ 1160.505702] MVC: In mxc_free_frame_buf

[ 1160.505709] In MVC:mxc_allocate_frame_buf - size=152064

[ 1160.508739] In MVC:mxc_v4l_ioctl

[ 1160.508746] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508752]    case VIDIOC_QUERYBUF

[ 1160.508759] In MVC:mxc_v4l2_buffer_status

[ 1160.508904] In MVC:mxc_mmap

[ 1160.508911]    pgoff=0x184c0, start=0x45dbd000, end=0x45de3000

[ 1160.508933] In MVC:mxc_v4l_ioctl

[ 1160.508939] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508944]    case VIDIOC_QUERYBUF

[ 1160.508949] In MVC:mxc_v4l2_buffer_status

[ 1160.508966] In MVC:mxc_mmap

[ 1160.508972]    pgoff=0x18300, start=0x46549000, end=0x4656f000

[ 1160.508987] In MVC:mxc_v4l_ioctl

[ 1160.508992] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508997]    case VIDIOC_QUERYBUF

[ 1160.509002] In MVC:mxc_v4l2_buffer_status

[ 1160.509013] In MVC:mxc_mmap

[ 1160.509019]    pgoff=0x18340, start=0x46700000, end=0x46726000

[ 1160.509033] In MVC:mxc_v4l_ioctl

[ 1160.509039] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.509044]    case VIDIOC_QUERYBUF

[ 1160.509049] In MVC:mxc_v4l2_buffer_status

[ 1160.509058] In MVC:mxc_mmap

[ 1160.509064]    pgoff=0x19f00, start=0x467ed000, end=0x46813000

[ 1160.509085] In MVC:mxc_v4l_ioctl

[ 1160.509092] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509097]    case VIDIOC_QBUF

[ 1160.509104] In MVC:mxc_v4l_ioctl

[ 1160.509110] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509115]    case VIDIOC_QBUF

[ 1160.509121] In MVC:mxc_v4l_ioctl

[ 1160.509126] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509131]    case VIDIOC_QBUF

[ 1160.509138] In MVC:mxc_v4l_ioctl

[ 1160.509143] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509148]    case VIDIOC_QBUF

[ 1160.509154] In MVC:mxc_v4l_ioctl

[ 1160.509160] In MVC: mxc_v4l_do_ioctl 40045612

[ 1160.509166]    case VIDIOC_STREAMON

[ 1160.509170] In MVC:mxc_streamon

[ 1160.509175] IPU:In prp_enc_enabling_tasks

[ 1160.509981] In prp_enc_setup

[ 1160.509988] YUV420

[ 1160.510026] eba 184c0000

[ 1160.510033] eba 18300000

[ 1160.510230] In MVC:mxc_poll

[ 1160.510243] In MVC:mxc_v4l_ioctl

[ 1160.510250] In MVC: mxc_v4l_do_ioctl c0445611

[ 1160.510256]    case VIDIOC_DQBUF

[ 1160.510261] In MVC:mxc_v4l_dqueue

[ 1170.504999] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[ 1170.511966] In MVC:mxc_poll

[ 1170.511983] In MVC:mxc_v4l_ioctl

[ 1170.511991] In MVC: mxc_v4l_do_ioctl c0445611

[ 1170.511997]    case VIDIOC_DQBUF

[ 1170.512001] In MVC:mxc_v4l_dqueue

[ 1180.504959] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[ 1180.511757] In MVC:mxc_poll

[ 1180.511773] In MVC:mxc_v4l_ioctl

[ 1180.511783] In MVC: mxc_v4l_do_ioctl c0445611

[ 1180.511789]    case VIDIOC_DQBUF

[ 1180.511794] In MVC:mxc_v4l_dqueue

[ 1190.504954] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

Labels (5)
1 Solution
9,405 Views
davemcmordie
Contributor III

[Edit Oct 24 2014: After receiving various messages from other users about what patches we have used, I have attached the relevant patches in a zip file.  These contain the MT9M031 driver, OV5647 driver and changes to the freescale kernel to (1) enable Greyscale/Raw capture and (2) allow correctly synch'd capture from a MIPI channel other than channel 0]

I finally got it working a couple of days ago.  I had a couple of problems but basically I hit and resolved the following issues:

  1. No useful error prints on dmesg -> add #define DEBUG before #include <device.h>, or add CFLAGS_ipu_common.o := -DDEBUG to the Makefile for ipu_common.c (as an example).
  2. MIPI from the camera -- I had the wrong register settings-- I was missing the setting that told the sensor to start streaming MIPI data.  The symptom here was 0x300/0x330 on the MIPI status register, and both error registers clear, but no data on the bus with the scope.  To see data on my fairly noisy Wandboard I had to measure at 10 ns/div in differential mode between the pairs (5GS/s scope, 1.5GHz probes).
  3. Eliminating CSI->IC->MEM path in mxc_v4l2_capture-- I just took out the code that made this an option.  Eventually I will add code to automatically remove the option when Bayer/Grey, but for now this works.
  4. Switching to 8 bit sensor output.  Shouldn't have to do this, but I was unable to get the image size right with 10 bit data (I was using 2 * width * height).  This seems to have been a big part of the missing EOF interrupt.
  5. Grey worked, Bayer didn't.  Not sure why, but I couldn't get Bayer working.  When I added support for Grey, I started getting images.  Bayer should work, but doesn't seem to for me.
  6. Made the capture client request Grey images.  If the capture client was requesting the image in a different format (default was UYVY or something), then the image would appear with the wrong format-- clearly having IPU colourspace processing done.

At this point I am getting Bayer patter data in memory and will soon look into using the graphics coprocessor or IPU to handle demosaicing.  When the code is a little cleaner I will also post my kernel patch and driver file.

View solution in original post

28 Replies
1,361 Views
venkatesh_pvsm
Contributor II

Hi Dave,

Sub:Ov9712 camera with Imx6 Bayer format support

         i have read your threads regarding the Bayer format support for imx6 .I have used imx6 solo processor with ov9712 omni vision camera in parallel port interface .

Ov9712 1MP  camera  does not have ISP and supports output formats (10-bit) raw RGB data, our camera connection in hardware side sensor->imx6 solo processor 8 bit (CSIO_Data12-CSIO_Data19).How can i achieve  this  conversion from camera  RGB bayer datas output   to YUV422 format .IPU does not support the bayer format so i need to convert bayer data  to YUV422 data before giving this data to IPU unit.

   Please suggest your ideas how to access this bayer data format in imx6 solo processor.

Thanks by,

Venkat

0 Kudos
1,361 Views
timokokkonen
Contributor I

Thanks Dave,

I can now see data flowing through the pipeline in the GREY mode. Although I don't think I am even close yet in having the data through correctly. And the behaviour is not very consistent with respect to the different capturing test applications I used (gstreamer, v4l2 capture example, Freescale v4l2 unit tests). But clearly I am making big progress here now. Thanks again!

-Timo

0 Kudos
1,361 Views
davemcmordie
Contributor III

More info (I've managed to enable some IPU tracing):

[  264.365866] imx-ipuv3 imx-ipuv3.0: CSI_SENS_CONF = 0x00001B00

[  264.365875] imx-ipuv3 imx-ipuv3.0: CSI_ACT_FRM_SIZE = 0x03BF04FF

[  265.948470] Grey/Generic

[  265.948485] imx-ipuv3 imx-ipuv3.0: init channel = 19

[  265.948508] imx-ipuv3 imx-ipuv3.0: resizing from 960 -> 960 pixels, downsize=0, resize=1.0 (reg=8192)

[  265.948527] imx-ipuv3 imx-ipuv3.0: initializing idma ch 20 @ ea8c0500

[  265.948550] imx-ipuv3 imx-ipuv3.0: ch 20 word 0 - 00000000 00000000 00000000 E0002800 000EFC9F

[  265.948563] imx-ipuv3 imx-ipuv3.0: ch 20 word 1 - 02D80000 005B0000 20C3C000 00027FC0 00000000

[  265.948573] imx-ipuv3 imx-ipuv3.0: PFS 0x6,

[  265.948582] imx-ipuv3 imx-ipuv3.0: BPP 0x5,

[  265.948591] imx-ipuv3 imx-ipuv3.0: NPB 0xf

[  265.948598] imx-ipuv3 imx-ipuv3.0: FW 1279,

[  265.948606] imx-ipuv3 imx-ipuv3.0: FH 959,

[  265.948614] imx-ipuv3 imx-ipuv3.0: EBA0 0x16c00000

[  265.948622] imx-ipuv3 imx-ipuv3.0: EBA1 0x16c00000

[  265.948631] imx-ipuv3 imx-ipuv3.0: Stride 2559

[  265.948638] imx-ipuv3 imx-ipuv3.0: scan_order 0

[  265.948646] imx-ipuv3 imx-ipuv3.0: uv_stride 0

[  265.948656] imx-ipuv3 imx-ipuv3.0: u_offset 0x0

[  265.948664] imx-ipuv3 imx-ipuv3.0: v_offset 0x0

[  265.948672] imx-ipuv3 imx-ipuv3.0: Width0 0+1,

[  265.948679] imx-ipuv3 imx-ipuv3.0: Width1 0+1,

[  265.948688] imx-ipuv3 imx-ipuv3.0: Width2 0+1,

[  265.948695] imx-ipuv3 imx-ipuv3.0: Width3 0+1,

[  265.948703] imx-ipuv3 imx-ipuv3.0: Offset0 0,

[  265.948712] imx-ipuv3 imx-ipuv3.0: Offset1 0,

[  265.948720] imx-ipuv3 imx-ipuv3.0: Offset2 0,

[  265.948727] imx-ipuv3 imx-ipuv3.0: Offset3 0

[  265.948741] eba 18c00000

[  265.948749] eba 16000000

[  265.948756] Calling enc_enable_csi

[  265.948763] Enabling ipu 0 with csi 0

[  265.949043] In MVC:mxc_poll

[  265.949063] In MVC:mxc_v4l_ioctl

[  265.949072] In MVC: mxc_v4l_do_ioctl c0445611

[  265.949080]    case VIDIOC_DQBUF

[  265.949084] In MVC:mxc_v4l_dqueue

[  275.947279] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[  275.955480] In MVC:mxc_poll

[  275.955501] In MVC:mxc_v4l_ioctl

[  275.955510] In MVC: mxc_v4l_do_ioctl c0445611

[  275.955516]    case VIDIOC_DQBUF

[  275.955520] In MVC:mxc_v4l_dqueue

[  285.955111] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[  780.248845] IPU_CONF =0x100006A4
[  780.248850] IDMAC_CONF =0x0000002F
[  780.248856] IDMAC_CHA_EN1 =0x00900000
[  780.248862] IDMAC_CHA_EN2 =0x00000000
[  780.248868] IDMAC_CHA_PRI1 =0x18800001
[  780.248875] IDMAC_CHA_PRI2 =0x00000000
[  780.248880] IDMAC_BAND_EN1 =0x00000000
[  780.248886] IDMAC_BAND_EN2 =0x00000000
[  780.248893] IPU_CHA_DB_MODE_SEL0 =0x00100000
[  780.248899] IPU_CHA_DB_MODE_SEL1 =0x00000000
[  780.248906] IPU_CHA_TRB_MODE_SEL0 =0x00800000
[  780.248913] IPU_CHA_TRB_MODE_SEL1 =0x00000000
[  780.248919] DMFC_WR_CHAN =0x00000090
[  780.248925] DMFC_WR_CHAN_DEF =0x202020F6
[  780.248931] DMFC_DP_CHAN =0x000096CA
[  780.248937] DMFC_DP_CHAN_DEF =0x2020F6F6
[  780.248943] DMFC_IC_CTRL =0x00000002
[  780.248949] IPU_FS_PROC_FLOW1 =0x00000000
[  780.248954] IPU_FS_PROC_FLOW2 =0x00000000
[  780.248961] IPU_FS_PROC_FLOW3 =0x00000000
[  780.248967] IPU_FS_DISP_FLOW1 =0x00000000
[  780.248973] IPU_VDIC_VDI_FSIZE =0x00000000
[  780.248978] IPU_VDIC_VDI_C =0x00000000
[  780.248984] IPU_IC_CONF =0x00000001

This tells me that (1) I am setting the format and size for the IPU registers correctly (Step A) 4. in the "Some Experience when enabling MIPI.." guide.

Still no frames, and no error message telling me what is failing.

There is one more clue however.  I can only attempt to capture once from the device.  The second time I capture, I get the following error and have to reboot to start capturing again:

[  440.222233] IPU:In prp_enc_enabling_tasks

[  440.230557] imx-ipuv3 imx-ipuv3.0: handler already installed on irq 20

This suggests that the IPU is the problem, and that it is not getting cleaned up properly.

Can anyone tell me how to dump CPU registers from the command line, in order to inspect the state of all these registers?  Or is that just something I have to code into (or enable) in the kernel?

0 Kudos
1,361 Views
davemcmordie
Contributor III

I can now get frames (although not actually correct width/height, etc), with a little bit of acrobatics:


  1. I set the format of the driver on line 1622 of the driver file to V4L2_PIX_FMT_UYVY.
  2. Compile the driver
  3. Unload and reload the capture drivers
  4. Attempt capture -> it fails
  5. Set the pixel format back to V4L2_PIX_FMT_SBGGR10, recompile, reload drivers
  6. Attempt capture -> it works

When it is working, I get "Warning IPU_INT_STAT_5 = 1", which seems to mean that the EOF interrupt is already set.  This would make sense since I am now getting frames with incorrect vertical sync (chopped images).

Once again, if anyone with knowledge of the IPU and Bayer / Generic capture can comment, I would be most grateful!  I am more than a little surprised at the lack of support from Freescale on this, especially given that I intend to share both my driver and tips on supporting Bayer and Generic capture.

0 Kudos
1,361 Views
jimmychan
NXP TechSupport
NXP TechSupport

We have an i.MX6 porting guide for Linux BSP. Please read the Chapter 7 for the CSI porting details. Hope this information can help you.

0 Kudos
1,361 Views
davemcmordie
Contributor III

Thanks Jimmy.

Have just read through the chapter and I don't see any significant mention of the MIPI interface. Don't I need to make sure the MIPI capture is working?

This guide would have been very helpful to me as I was implementing the driver. I had to figure out And guess at a bunch of things that it explains.

Do you have any thoughts on what would lead to the MIPI PHY having correct state and no errors, but no images coming through?

Dave

0 Kudos
1,361 Views
jimmychan
NXP TechSupport
NXP TechSupport
0 Kudos
1,361 Views
davemcmordie
Contributor III

Yes, I have reviewed that in detail, verifying register settings all the way through.  It doesn't help with figuring out why I am not getting frame interrupts.  From what I can tell it could be any of the following:

1. Incorrectly configured CSI2IPU gasket

2. Bayer image format not supported, or special magic needed to get it copied into memory (there are many threads on this topic and I have reviewed them all carefully.

3. MIPI connection not working correctly (seems not to be the issue since error registers are clear and state is 0x300)

4. Camera not correctly configured

It would be very helpful to have some feedback on which of these are likely to be problematic, based on the dmesg trace above.

One thing: even though the MIPI PHY state is 0x300, I have not been able to see any data on my scope, and there are high levels of noise throughout the wandboard system.  Could it be that the PHY state is okay, but in fact the camera is not sending any data?  Also, I never see the PHY state change to 0x330, which I would expect if either of the data lanes goes into stop mode.

0 Kudos