IMX8MM EVK OV7251 MIPI integration

[will_goodman]

Hi,

I'm trying to get an ov7251 sensor working on the MIPI CSI interface of the IMX8MM EVK. We have a custom breakout board that exactly matches the connector pinout of the ov5640 board available from NXP, and I have edited the imx8mm-evk.dts as follows (using the device tree binding document supplied with the mainline ov7251 driver):

&i2c3 {

   clock-frequency = <100000>;
   pinctrl-names = "default";
   pinctrl-0 = <&pinctrl_i2c3>;
   status = "okay";

   ov7251: ov7251@60  {        
        compatible = "ovti,ov7251";
        reg = <0x60>;
        status = "okay";
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_csi_pwn>, <&pinctrl_csi_rst>;                
        clocks = <&clk IMX8MM_CLK_CLKO1>;                            
        clock-names = "xclk";
        clock-frequency = <24000000>;
        enable-gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;        //same as the ov5640
        //these are all hardwired on, 0 here seems to assign dummy registers ok
        vdddo-supply = <0>;
        vdda-supply = <0>;
        vddd_supply = <0>;
        
        //assume these can be culled if not required
        assigned-clocks = <&clk IMX8MM_CLK_CLKO1>;
        assigned-clock-parents = <&clk IMX8MM_CLK_24M>;
        assigned-clock-rates = <24000000>;
        csi_id = <0>;
        pwn-gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;                    
        mclk = <24000000>;
        mclk_source = <0>;        
        port {
            ov7251_mipi1_ep: endpoint {
                clock-lanes = <1>;                                                        
                data-lanes = <2>;
                remote-endpoint = <&mipi1_sensor_ep>;
            };
        };
    };

};

&mipi_csi_1 {
    #address-cells = <1>;
    #size-cells = <0>;
    status = "okay";
    port {
        mipi1_sensor_ep: endpoint@1 {
            remote-endpoint = <&ov7251_mipi1_ep>;
            data-lanes = <2>;
            csis-hs-settle = <13>;
            csis-clk-settle = <2>;
            csis-wclk;
        };
        
        csi1_mipi_ep: endpoint@2 {
            remote-endpoint = <&csi1_ep>;
        };
    };
};

The driver has been enabled in imx_v8_defconfig, and dmesg shows the sensor is successfully being probed and initialised during boot, with a final message of:

mxc_mipi-csi 32e30000.mipi_csi: Registered sensor subdevice: ov7251 2-0060

Running gst-device-monitor-1.0 does not show the interface however. Checking dmesg after running the device monitor command shows the following:

mxc_mipi-csi 32e30000.mipi_csi: format not match

I added a printk to the mipi_csis_enum_mbus_code() fn in mxc_mipi_csi.c and this returns a code of 301021992, but I am not sure why this is invalid.

I have tried adding an entry to the formats struct in mx6s_capture.c:

{
        .name        = "RAW-10",
        .fourcc        = V4L2_PIX_FMT_Y10,
        .pixelformat    = V4L2_PIX_FMT_Y10,
        .mbus_code    = MEDIA_BUS_FMT_Y10_1X10,
        .bpp        = 1,
    }

I have also tried fourcc and pixelformat set to V4L2_PIX_FMT_Y10P, but the "format not match" error persists.

I'm not sure if I'm barking up the wrong tree here. Perhaps I should I be trying to create an equivalent of the ov5640_mipi_v2.c driver for the ov7251, in addition to the one found in the i2c folder?

Any pointers to helpful documentation regarding the MIPI subsystems would also be appreciated.

Many thanks,

Will

[joanxie]

I don't have ov7251 to test for you, so far we tested this successfully

{       

.name        = "RAWRGB10 (SBGGR10)",       

.fourcc        = V4L2_PIX_FMT_SBGGR10,       

.pixelformat    = V4L2_PIX_FMT_SBGGR10,
.mbus_code    = MEDIA_BUS_FMT_SBGGR10_1X10,       

.bpp        = 2,   }

do you mean you use this in the mx6s_capture.c failed?

for raw10 capturing in the imx8mm, we have tested command is:

For raw10

v4l2-ctl --set-parm=30
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=BG10 --stream-mmap --stream-count=1 --stream-to=raw10_1920x1080.raw

[will_goodman]

Hi Joan,

I have copied your additional format table entry exactly into mx6s_capture.c, but the error persists:

v4l2 v4l2_calls.c:546:gst_v4l2_subscribe_event: Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'

I am using gstreamer to interface with V4L2 as that is the only media streaming package included in the default multimedia build (I did try installing v4l2-ctl a while back via apt, but the upstream version broke quite a few dependencies for the board, maybe I will have more luck if I try an earlier release).

What I would really like to know is if I need to implement a capture driver with the same interfaces as the ov5640_mipi_v2.c driver?

In chapter 14 of the i.MX porting guide there is an explanation of what functional interfaces to implement in a MIPI driver for the IMX6 series (that does not appear to match the implementation in ov5640_mipi_v2.c). 

In chapter 15 there is absolutely no information on how to support cameras on the IM7 or 8 series, apart from a pointer to the Capture Overview chapter of the i.MX linux reference manual.

Table 6.5 from that chapter lists files for the that are not present in the multimedia build for the i.MX8MM. The files that are in build appear to be the ones used by the 'i.MX6 IPU V4L2 plugin', so I can only assume that the i.MX8MM contains the i.MX6 IPU, but this doesn't appear to be documented anywhere.

Can you clarify what MIPI IP I should actually be trying to interface with?

Thanks,

Will 

[will_goodman]

Can anybody offer any update on this? I created a support ticket detailing what I have achieved so far but was given a generic response, with no indication of whether a platform driver specific to the target sensor needs to be implemented on top of the mainline i2c driver.

Calling gst-device-monitor-1.0 with debug level 3 set gives the following error:

v4l2 v4l2_calls.c:546:gst_v4l2_subscribe_event: Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'

[joanxie]

for raw10, try to use :

 .name  = "RAWRGB12 (SGRBG10)",
 .fourcc  = V4L2_PIX_FMT_SGRBG10,
 .pixelformat = V4L2_PIX_FMT_SGRBG10,
 .mbus_code = MEDIA_BUS_FMT_SRGGB12_1X10,
.bpp  = 1,

[will_goodman]

Hi Joan,

Thanks for the suggestion. I forgot to mention the OV7251 is a black and white sensor, and the mainline i2c driver uses MEDIA_BUS_FMT_Y10_1X10. Is it possible to convert formats like this? When I use your suggested format as is I get the following errors:

unknown mbus:0x200a

mx6s-csi 32e20000.csi1_bridge: mbus (0x0000200a) invalid 

I found another format table in mxc_mipi_csi.c, to which I have added:

{

         .code = MEDIA_BUS_FMT_Y10_1X10,

         .fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,

         .data_alignment = 10,

}

I am unclear on the correct data flow here. I assume the i2c driver interfaces directly with the camera for configuration and power, but am unsure if it can directly interface with the mipi platform drivers or if it needs another interfacing driver such as the ov_5640_mipi_v2.c file which is included in the demo multimedia image build. 

I'm going to rake back through the NXP documentation to see if I can find anything that describes the requirements, but again, if anybody can point me to any documentation that's worth reading I would be grateful.

Will 

[joanxie]

how about now? did you try the code as I mentioned in the mx6s_capture.c? I don't think you can copy the code from mxc_mipi_csi.c to the mx6s_capture.c directly

[will_goodman]

Hi again Joan,

I have replaced the media bus format in the ov7251 driver with both SRGGB12_1X12 and SRGGB12_1X10, and filled in the relevant entries in both mxc_mipi_csi and mx6s_capture so that V4L2 doesn't report any format mismatch.

In all cases (even when using MEDIA_BUS_FMT_Y10_1X10), when probing the device with gst-device-monitor (debug level 3) I get the following error message:

v4l2 v4l2_calls.c:546:gst_v4l2_subscribe_event: Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'

Have I missed any useful information on how to debug this further? 

The MIPI CSI documentation provided with the kernel porting guide seems to be extremely sparse (I may have missed something, apologies if I have).

In response to a support ticket, NXP technical support recommended that I contact Omnivision as 'it must be similar to the OV5640', but from what I can tell there is a sort of wishbone driver that was written at NXP/Freescale to allow the 5640 to communicate successfully with the Samsung MIPI IP integrated into the IMX8MM, of which Omnivision are unlikely to have any information.

[joanxie]

how did you get gst_v4l2_subscribe_event: Cannot subscribe V4L2_EVENT_SOURCE_CHANGE or V4L2_EVENT_EOS event for device '/dev/video0'? if you use gstreamer, don't forget add format in the command

[will_goodman]

Hi Joan,

Following this up:

The error I described above occurs whenever Gstreamer attempts to access /dev/video0.

I have found Gstreamer doesn't appear to be capable of converting RAW formats, so I am now capturing straight to disk using v4l2-ctl.

To get any buffers to arrive I have to set up the data path in a very strange way, and the received data is, unsurprisingly, malformed. 

First I changed all bus format references in the driver from Y10_1X10 to SBGGR10_1X10.

Then in mxc_mipi_csi.c I added a new entry to the format table:

{

    .code = MEDIA_BUS_FMT_SBGGR10_1X10,

    .fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW10,

    .data_alignment = 16,

}

And finally in mx6s_capture.c

{

    .name = "RAWRGB10 (BG10)",

    .fourcc = V4L2_PIX_FMT_SBGGR10,

    .pixelformat = V4L2_PIX_FMT_SBGGR10,

    .mbus_code = MEDIA_BUS_FMT_SBGGR10_1X10,

    .bpp = 2,

}

This didn't initially work, no buffers were being received and v4l2-ctl was just hung waiting.

I then changed the .data_alignment property to 32 and the .fmt_reg to MIPI_CSIS_ISPCFG_FMT_RAW8, and successfully received a buffer. Although the image is definitely not clear or correct, raw data shows a lighter image when the camera is pointing at a lit white object and a darker one if pointing at a dark object.

Can anybody provide any suggestion about how to debug this further? I have tried turning on debugging for all the relevant modules, but I can't seem to find the cause for buffers not being received when the MIPI module is given the correct configuration, and I cannot analyse the bus with the analysers that I have available to me as they are not capable of sampling fast enough.

Any help appreciated.

[will_goodman]

Another update: I have spoken to the Omnivision FAE and they have supplied the correct PLL settings to switch the mainline driver to RAW8 output, which it appears Gstreamer is capable of converting to sRGB.

The main problem now is that I never receive an EOF in this configuration. 

The final messages from Gstreamer before hanging (with GST_DEBUG=5) can be seen in the attached log file. I assume the PHY is not receiving the full 307200 bytes necessary for a frame, but I can't seem to get any useful debug information from the lower layers of the drivers, such as if any bytes have been received at all.

Can anybody point me to the relevant code?

Thanks,

Will

[will_goodman · Accepted solution]

Got it! There was some confusion about how lanes are identified in the device tree. The 7251 driver specifies that to use 2 lanes you supply an array <1 2> where the integers are ignored, the number of elements are what is important. I guess this is not the case with the NXP kernel, as changing data-lanes=<2> to <1> got us going.

Unfortunately the bayer2rgb gstreamer plugin is incredibly laggy, but that's a different can of worms. Thanks for your help!

[will_goodman]

Hi Joan,

I tried your suggested change and got the error (as above):

unknown mbus:0x200a

mx6s-csi 32e20000.csi1_bridge: mbus (0x0000200a) invalid 

I think this is because the I2C driver specifies MEDIA_BUS_FMT_Y10_1X10 for the OV7251. 

Should I maybe replace that in the I2C driver?

Thanks,

Will