i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

[davemcmordie]

Hello,

I have implemented an OV5647 camera board which connects to my i.MX6 quad Wandboard.  I am having trouble getting images from the driver and need some help figuring out why.  I have reviewed about a dozen related threads on this forum, and tried to carefully implement their suggestions.

Here are the steps I have taken so far:

1. Started with working kernel (based on John Weber's Yocto Kernel) and board which is known to work with OV5640 (YUV sensor) in this configuration.  rootfs is Linaro armhf with desktop ubuntu (so I can use VLC and other known tools for testing).
2. Made sure that the BSP sets GPR1 19 to 0 (select MIPI CSI interface to IPU0 CSI 0), and that the camera driver is detected by I2C.
3. Built a driver, OV5647_mipi.c, based on OV5640_mipi.c with the following changes:
   1. Replaced modes (register programming code) with code appropriate to generate a 1280x960 30 fps preview on the OV5647
   2. Changed the chip_id high and low bytes to 0x56 and 0x47 respectively
   3. Changed ov5647_data.pix.pixelformat to V4L2_PIX_FMT_SBGGR10
   4. Added code to detect this and call mipi_csi2_set_datatype with MIPI_DT_RAW10
   5. Tried steps 3, 4 with GREY and RAW10
4. Added support for V4L2_PIX_FMT_SBGGR10 and V4L2_PIX_FMT_GREY to
   1. drivers\media\video\mxc\capture\ipu_csi_enc.c
   2. drivers\media\video\mxc\capture\mxc_v4l2_capture.c
   3. drivers\mxc\ipu3\ipu_capture.c
   4. drivers\mxc\ipu3\ipu_common.c
   5. drivers\mxc\ipu3\ipu_param_mem.h
5. Tested
   1. Sensor is correctly detected on I2C
   2. Driver reports MIPI clock stabilized okay
   3. Driver reports data being received from sensor

I have to guess that what is going on is that the image format is not being handled correctly, but I can't tell from the tracing.

Some specific questions:

1. How can I verify that I am talking to the correct IPU, CSI port and virtual channel?
2. Despite the driver's claim that it is receiving data, is it possible that this is a signal integrity issue?  I have not been able to successfully probe the lanes (I see a ton of common mode noise)
3. Are there steps I have missed to support a 10 bit Bayer sensor just as grayscale?
4. What are the likely reasons why it is not getting frame interrupts?
5. Help!   :smileyhappy:
   

The following is what I get with DEBUG tracing on in the drivers:

[ 1160.505314] End of mxc_v4l2_g_fmt: crop_current widthxheight 1280 x 960

[ 1160.505352] In MVC:mxc_v4l_ioctl

[ 1160.505360] In MVC: mxc_v4l_do_ioctl c0405602

[ 1160.505373] In MVC:mxc_v4l_ioctl

[ 1160.505380] In MVC: mxc_v4l_do_ioctl c0405602

[ 1160.505458] In MVC:mxc_v4l_ioctl

[ 1160.505465] In MVC: mxc_v4l_do_ioctl c0445624

[ 1160.505472]    case default or not supported

[ 1160.505528] In MVC:mxc_v4l_ioctl

[ 1160.505535] In MVC: mxc_v4l_do_ioctl c02c563a

[ 1160.505540]    case VIDIOC_CROPCAP

[ 1160.505547] In MVC:mxc_v4l_ioctl

[ 1160.505552] In MVC: mxc_v4l_do_ioctl 4014563c

[ 1160.505561]    case VIDIOC_S_CROP

[ 1160.505567]    Cropping Input to ipu size 1280 x 960

[ 1160.505678] In MVC:mxc_v4l_ioctl

[ 1160.505686] In MVC: mxc_v4l_do_ioctl c0145608

[ 1160.505693]    case VIDIOC_REQBUFS

[ 1160.505697] In MVC:mxc_streamoff

[ 1160.505702] MVC: In mxc_free_frame_buf

[ 1160.505709] In MVC:mxc_allocate_frame_buf - size=152064

[ 1160.508739] In MVC:mxc_v4l_ioctl

[ 1160.508746] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508752]    case VIDIOC_QUERYBUF

[ 1160.508759] In MVC:mxc_v4l2_buffer_status

[ 1160.508904] In MVC:mxc_mmap

[ 1160.508911]    pgoff=0x184c0, start=0x45dbd000, end=0x45de3000

[ 1160.508933] In MVC:mxc_v4l_ioctl

[ 1160.508939] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508944]    case VIDIOC_QUERYBUF

[ 1160.508949] In MVC:mxc_v4l2_buffer_status

[ 1160.508966] In MVC:mxc_mmap

[ 1160.508972]    pgoff=0x18300, start=0x46549000, end=0x4656f000

[ 1160.508987] In MVC:mxc_v4l_ioctl

[ 1160.508992] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.508997]    case VIDIOC_QUERYBUF

[ 1160.509002] In MVC:mxc_v4l2_buffer_status

[ 1160.509013] In MVC:mxc_mmap

[ 1160.509019]    pgoff=0x18340, start=0x46700000, end=0x46726000

[ 1160.509033] In MVC:mxc_v4l_ioctl

[ 1160.509039] In MVC: mxc_v4l_do_ioctl c0445609

[ 1160.509044]    case VIDIOC_QUERYBUF

[ 1160.509049] In MVC:mxc_v4l2_buffer_status

[ 1160.509058] In MVC:mxc_mmap

[ 1160.509064]    pgoff=0x19f00, start=0x467ed000, end=0x46813000

[ 1160.509085] In MVC:mxc_v4l_ioctl

[ 1160.509092] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509097]    case VIDIOC_QBUF

[ 1160.509104] In MVC:mxc_v4l_ioctl

[ 1160.509110] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509115]    case VIDIOC_QBUF

[ 1160.509121] In MVC:mxc_v4l_ioctl

[ 1160.509126] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509131]    case VIDIOC_QBUF

[ 1160.509138] In MVC:mxc_v4l_ioctl

[ 1160.509143] In MVC: mxc_v4l_do_ioctl c044560f

[ 1160.509148]    case VIDIOC_QBUF

[ 1160.509154] In MVC:mxc_v4l_ioctl

[ 1160.509160] In MVC: mxc_v4l_do_ioctl 40045612

[ 1160.509166]    case VIDIOC_STREAMON

[ 1160.509170] In MVC:mxc_streamon

[ 1160.509175] IPU:In prp_enc_enabling_tasks

[ 1160.509981] In prp_enc_setup

[ 1160.509988] YUV420

[ 1160.510026] eba 184c0000

[ 1160.510033] eba 18300000

[ 1160.510230] In MVC:mxc_poll

[ 1160.510243] In MVC:mxc_v4l_ioctl

[ 1160.510250] In MVC: mxc_v4l_do_ioctl c0445611

[ 1160.510256]    case VIDIOC_DQBUF

[ 1160.510261] In MVC:mxc_v4l_dqueue

[ 1170.504999] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[ 1170.511966] In MVC:mxc_poll

[ 1170.511983] In MVC:mxc_v4l_ioctl

[ 1170.511991] In MVC: mxc_v4l_do_ioctl c0445611

[ 1170.511997]    case VIDIOC_DQBUF

[ 1170.512001] In MVC:mxc_v4l_dqueue

[ 1180.504959] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[ 1180.511757] In MVC:mxc_poll

[ 1180.511773] In MVC:mxc_v4l_ioctl

[ 1180.511783] In MVC: mxc_v4l_do_ioctl c0445611

[ 1180.511789]    case VIDIOC_DQBUF

[ 1180.511794] In MVC:mxc_v4l_dqueue

[ 1190.504954] ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0

[levlvovsky]

Dave,

I am trying to make ov8865 sensor to work with wandboard dual by modifying the ov5640_mipi driver. All the I2C communication seems to be OK, then I get the same "ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0" error. Can you please clarify for me couple of thing from this post?

1) How can you see that "Driver reports data being received from sensor" ?

2) What's the "MIPI status register" ?

3) What' the setting to tell sensor to start streaming the MIPI data? Is it the 0x4202 register?

Thank you,

Lev.

[davemcmordie]

Hi Lev,

I believe this means that we got 0x300 or 0x330 from the MIPI status register (I am going from memory here and it has been a long time).  These values mean the MIPI bus is set up correctly and packets are being received on it.  The MIPI status register is a register (documented in the i.MX6Q reference manual in the chapter on the MIPI to CSI gasket) which tells you whether the MIPI bus is able to communicate with your sensor or not.

Basically you have to tell your sensor to start sending frames using the I2C commands, and then you should start receiving data on the MIPI bus.  Once you are receiving data correctly you have to make sure the data format and size are correct and that the correct capture channel is configured (for me this was CSI->MEM, not CSI->IPU->MEM (again going from memory, but something like that-- it had to be direct to memory since I wasn't decoding YUV).

The register to start streaming depends on your sensor.  Contact your OmniVision rep for support.  I didn't have any direct sales contact with OmniVision, and they still sent me the datasheets and provided register settings to get the OV5647 up and running.

If you're feeling lost, I was too.  It took me a while to get my head around the data path, but once I did there were only two or three things to sort out.  There is a somewhat helpful checklist on this forum:

https://community.freescale.com/docs/DOC-94312

Hope that helps,

Dave

[vijaikumar]

Hi Dave McMordie /​ levlvovsky                   

                         Is there any driver for ov8865, which you know of ? We are planning to use ov8865 module with i.MX6Q processor.

[davemcmordie]

Hi Vijai,

I have no idea if this device is supported as we do not use it. I would

suggest you post generally in a new topic to ask the question if you have

not already done so. If support is not offered, you will need register

settings from Omnivision’s support department and another driver for a

related sensor as a starting point. From there it is a messy road of

reading the datasheet and register reference, tweaking the driver, testing

and debugging.

Best,

Dave

From: Vijai Kumar

Sent: Tuesday, August 4, 2015 1:10 AM

To: Dave McMordie <mcmordie@viionsystems.com>

Subject: You have been mentioned by Vijai Kumar in Re: i.MX6 OV5647 Bayer

sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

in Freescale Community

<http://jiveon.jivesoftware.com/mpss/c/iAA/PDcDAA/t.1pa/1sgFa3hhRMSf8pZgEF3etQ/h0/eqhG5v9o4WV1pCmWaB03cUjBPgRj-2F3GoJUaXSZ409bQ-3D>

You have been mentioned

by Vijai Kumar

<http://jiveon.jivesoftware.com/mpss/c/iAA/PDcDAA/t.1pa/1sgFa3hhRMSf8pZgEF3etQ/h1/eqhG5v9o4WV1pCmWaB03cUjBPgRj-2F3GoJUaXSZ409bRN-2FArDB1Dbz5vXUH-2BStRQn-2BONs1jeq5q4DHqLr40tzcMsi9vTZvKL7uhw2OMi-2Bqng-3D>

*in

Re: i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue

timeout enc_counter 0) in Freescale Community* - View Vijai Kumar's

reference to you

<http://jiveon.jivesoftware.com/mpss/c/iAA/PDcDAA/t.1pa/1sgFa3hhRMSf8pZgEF3etQ/h2/eqhG5v9o4WV1pCmWaB03cUjBPgRj-2F3GoJUaXSZ409bTFPK2mrQBb99hPyShlgUemYgMLvNbG1ra5FN11uHtPyRB-2B30puStpr9Ysa0beN7B8-3D>

[levlvovsky]

I got the sensor to send image data. For some reason I get data only when I set the sensor output to 8bit and declare image format as  V4L2_PIX_FMT_GREY. If I declare format as  V4L2_PIX_FMT_SBGGR8 bayer (which is right for that sensor in 8bit setting) then I get nothing. The data is also corrupted - rather than getting one big image I get 16 small images instead.

What the difference between CSI->MEM and CSI->IPU->MEM? I tried changing CSI->IPU->MEM to CSI->MEM but didn't notice any change. I think the Ov8865 sensor gives output only in the raw bayer format. 

Thank you,

Lev.

[davemcmordie · Accepted solution]

[Edit Oct 24 2014: After receiving various messages from other users about what patches we have used, I have attached the relevant patches in a zip file.  These contain the MT9M031 driver, OV5647 driver and changes to the freescale kernel to (1) enable Greyscale/Raw capture and (2) allow correctly synch'd capture from a MIPI channel other than channel 0]

I finally got it working a couple of days ago.  I had a couple of problems but basically I hit and resolved the following issues:

1. No useful error prints on dmesg -> add #define DEBUG before #include <device.h>, or add CFLAGS_ipu_common.o := -DDEBUG to the Makefile for ipu_common.c (as an example).
2. MIPI from the camera -- I had the wrong register settings-- I was missing the setting that told the sensor to start streaming MIPI data.  The symptom here was 0x300/0x330 on the MIPI status register, and both error registers clear, but no data on the bus with the scope.  To see data on my fairly noisy Wandboard I had to measure at 10 ns/div in differential mode between the pairs (5GS/s scope, 1.5GHz probes).
3. Eliminating CSI->IC->MEM path in mxc_v4l2_capture-- I just took out the code that made this an option.  Eventually I will add code to automatically remove the option when Bayer/Grey, but for now this works.
4. Switching to 8 bit sensor output.  Shouldn't have to do this, but I was unable to get the image size right with 10 bit data (I was using 2 * width * height).  This seems to have been a big part of the missing EOF interrupt.
5. Grey worked, Bayer didn't.  Not sure why, but I couldn't get Bayer working.  When I added support for Grey, I started getting images.  Bayer should work, but doesn't seem to for me.
6. Made the capture client request Grey images.  If the capture client was requesting the image in a different format (default was UYVY or something), then the image would appear with the wrong format-- clearly having IPU colourspace processing done.

At this point I am getting Bayer patter data in memory and will soon look into using the graphics coprocessor or IPU to handle demosaicing.  When the code is a little cleaner I will also post my kernel patch and driver file.

[timokokkonen]

Hi Dave,

Thanks for sharing the details of your work. I am working on to get a camera sensor working on a Wandboard Dual. The sensor appears to be somewhat similar to OV5647, at least in that sense that it is sending out the data over MIPI CSI-2 in raw Bayer 8 or 10bit format. I have now confirmed that I have the MIPI_CSI working properly as the DPHY status register is alternating between 0x300 and 0x330 during capture and the error bits are all clear. However, other than that, I get no data through the IPU pipeline. Looking at your description the reason for that is clear.

Therefore, I would be very much interested in seeing your code changes you have done. Have published them anywhere yet? Even if they are not cleaned up yet, I think those would be quite useful. Maybe I could somehow participate with fixing things as I appear to be facing the same problem here too.

Thanks!

[DraganOstojic]

Timo, did you check that virtual channel is properly set on the ov5647 side? I had the same outcome as you did working with ov9740 and the reason was that imx6 software expects vc=1 while ov9740 default is vc=0.

[timokokkonen]

The virtual channel should be fine as I am getting the data all the way through to the gstreamer buffers and I am able save the captured image on a file.

Now the problem is that apparently the data that gets through is in the raw bayer format but the pixelformat information doesn't come correctly with the image stream. The data gets through if I configure the pixel format to V4L2_PIX_FMT_GREY, but that's not what the image data is really. If I configure the sensor and CSI2 into RAW10 mode, the captured image it looks like there are in fact four frames instead of one. That would make sense if the data that gets through is in bayer format but it is being interpreted as YUV or RGB. One reason for that is obviously the V4L2_PIX_FMT_GREY which I am using. Probably I need to add support for V4L2_PIX_FMT_SRGGB8 and V4L2_PIX_FMT_SRGGB10 at least.

But I haven't quite figured out yet how the data flows through the layers, so I am not there yet. I will let you know when I have some progress.

[davemcmordie]

It sounds like you are well on the way if you are receiving the full

resolution bayer format images. I was not able to get the RG8/RG10

versions working—like you it returned an image which it had tried to colour

convert, and the result was a 4x4 grid of subimages. Since there I will

need to debayer the images in software anyhow, I left it at that. The

tracing I did do confirmed the correct CPMEM settings for each raw mode.

There are other threads in which some people report being able to capture

in RAW, but it was not clear to me how they got it working.

From: Timo Kokkonen

Sent: October-04-13 5:45 AM

To: Dave McMordie

Subject: Re: - i.MX6 OV5647 Bayer sensor driver (ERROR:

v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

<https://community.freescale.com/>

i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue

timeout enc_counter 0)

reply from Timo

Kokkonen<https://community.freescale.com/people/timokokkonen?et=watches.email.thread>in

i.MX Community - View the full

discussion<https://community.freescale.com/message/354072?et=watches.email.thread#354072>

[meflo]

Hi,

I also am getting a 4x4 grid of smaller images instead of one normal image.

But my sensor is outputting YUV422.

Any idea what might be happening?

Thanks

[levlvovsky]

Do you see correct colors? Are the small images stable? - Because if you look at YUV422 as grey then you see the full size Y image followed by U and V 1/4 size each.  

[meflo]

Hi,

thanks for your reply.

Here's how it looks like

https://www.dropbox.com/s/8c2ww10qmk2s7h6/20150129_102941.jpg?dl=0

The images are stable, not scrolling, but you can see some slight color issues (seems to be a bit greenish) and the fact that it actually displays the frames in the space of a grid of 2 x 2. Sometimes it displays the entire 4 frames, sometimes like in the picture it displays some of the frames full and the others with some hsync and vsync issues.

[meflo]

So that is using V4L2_PIX_FMT_UYVY in my driver.

If I use V4L2_PIX_FMT_YUV420, I get a 2x4 grid

Subsequently, my driver enabled mipi using datatype:

mipi_csi2_set_datatype(mipi_csi2_info, MIPI_DT_YUV422);

[meflo]

Dump of registers:

IPUx_CSI0_SENS_CONF is 0x00008A32

IPUx_CSI0_ACT_FRM_SIZE is 0x023F02CF

Apparently IPUx_CSI0_SENS_CONF value shows data width is 8 bits. As I am receiving YUV422, shouldn't it read 16 bits instead?

[levlvovsky]

I would think it's 12bit for the YUV422

[meflo]

It's 16 bits: 8 bits Y, 8 bits U, 8 bits Y, 8 bits V ---> 2 pixels (one Y for each of the 2 pixels, while U and V are shared between the 2 pixels)

So question remains: IPUx_CSI0_SENS_CONF should reflect through bits 14-11 the data format of the received data or is it something internal to CSI?

CSI0_DATA_WIDTH has the description - "Data width. This field defines the number of bits per color."

Not clear if it's the width of the received data or the width it will be sent internally to IPU.

[meflo]

Found this in the RM of imx6:

37.4.3.2.2

High-speed serial interface - MIPI (Mobile Industry Processor

Interface).

In MIPI interface two values arrive in each cycle. Each value is 8 bit wide, which means

16 MSB bits of the data bus input are treated, while 4 LSB bits are ignored.

When working in this mode, the CSI can handle up to 4 streams of data. Each stream is

identified with DI (data identifier) that includes the virtual channel and the data type of

this stream. Each stream that is handled is defined in registers MIPI_DI0-3. Only the

main stream (MIPI_DI0) can be sent to all destination units while the other streams are

sent only to the SMFC as generic data.

In this mode SENS_DATA_FORMAT and DATA_WIDTH registers are ignored, since

this information is coming to the CSI via the MCT_DI bus.

[ilkodossev]

The only Bayer-related FOURCCs which the IPU Capture Driver would recognize are

#define IPU_PIX_FMT_GENERIC fourcc('I', 'P', 'U', '0') /*!< IPU Generic Data */

#define IPU_PIX_FMT_GENERIC_32 fourcc('I', 'P', 'U', '1') /*!< IPU Generic Data */

#define IPU_PIX_FMT_GENERIC_16 fourcc('I', 'P', 'U', '2') /*!< IPU Generic Data */

Formats "V4L2_PIX_FMT_SBGGR10" or "V4L2_PIX_FMT_SBGGR16" are not going to work.

Try 0x32555049 instead in your driver -- it is "IPU2" and will get data stream going.

[davemcmordie]

Hi Timo,

My changes to the kernel can be found here:

https://github.com/mcmordie/linux/compare/johnweber:wandboard_imx_3.0.35_4.0.0...wandboard_imx_3.0.3...

I don’t have my camera driver up in a public place yet, but anyway that

won’t help you.

Basically there were substantial changes to enable grayscale mode and a

more minor change to get the MIPI clock rate right. It was also important

to use the correct image capture path (CSI->MEM) and to request the GRAY

format when capturing. Otherwise the drivers try to handle the colourspace

conversion and fail.

HTH,

D

From: Timo Kokkonen

Sent: October-02-13 9:26 AM

To: Dave McMordie

Subject: Re: - i.MX6 OV5647 Bayer sensor driver (ERROR:

v4l2 capture: mxc_v4l_dqueue timeout enc_counter 0)

<https://community.freescale.com/>

i.MX6 OV5647 Bayer sensor driver (ERROR: v4l2 capture: mxc_v4l_dqueue

timeout enc_counter 0)

reply from Timo

Kokkonen<https://community.freescale.com/people/timokokkonen?et=watches.email.thread>in

i.MX Community - View the full

discussion<https://community.freescale.com/message/353698?et=watches.email.thread#353698>