IMX8M Mini New MIPI-CSI Camera Driver Integration

[matthewmader]

Hello,

I have a Sony IMX219 camera that I am trying to integrate with the IMX8M Mini.  I have modeled my driver on the OV5640 driver provided the linux-imx repo.  However, I cannot seem the camera to stream video.  I run the following command: 

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480 ! autovideosink sync=false

and get the following error:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:../../../../git/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)

I have tried variation on the above command but always get the same output.

When I query the camera via the v4l2-ctl tools I get the following responses:

v4l2-ctl --list-devices

i.MX6S_CSI (platform:32e20000.csi1_bridge):
/dev/video0

v4l2-ctl --list-formats-ext

ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'BA81' (8-bit Bayer BGBG/GRGR)
Size: Discrete 3280x2464
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1640x1232
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)

v4l2-ctl --all

Driver Info:
Driver name : mx6s-csi
Card type : i.MX6S_CSI
Bus info : platform:32e20000.csi1_bridge
Driver version : 5.4.3
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 0
Video input : 0 (Camera: ok)
Format Unknown (00000000): <= Not sure why this is 'Unknown'
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 0, Height 0
Default : Left 0, Top 0, Width 0, Height 0
Pixel Aspect: 1/1
Crop: Left 0, Top 0, Width 0, Height 0
Selection: crop, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: crop_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: crop_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_padded, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: native_size, Left 0, Top 0, Width 0, Height 0, Flags:
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0

Below is what my DTS file looks like:

/* IMX219 Camera */
imx219@10 {
    status = "okay";
    compatible = "sony,imx219";
    reg = <0x10>;

    port {
        imx219_ep: endpoint {
            remote-endpoint = <&mipi1_sensor_ep>;
            clock-lanes = <0>;
            data-lanes = <1 2>;
            clock-noncontinuous;
            link-frequencies = /bits/ 64 <456000000>;
      };
    };
};

&csi1_bridge {
    fsl,mipi-mode;
    status = "okay";
    port {
      csi1_ep: endpoint {
        remote-endpoint = <&csi1_mipi_ep>;
      };
    };
};

&mipi_csi_1 {
    #address-cells = <1>;
    #size-cells = <0>;
    status = "okay";

    port {
      mipi1_sensor_ep: endpoint@1 {
        remote-endpoint = <&imx219_ep>;
        data-lanes = <2>;
        csis-hs-settle = <13>; <= Not sure what this field does
        csis-clk-settle = <2>; <= Not sure what this field does
        csis-wclk; <= Not sure what this field does
      };

      csi1_mipi_ep: endpoint@2 {
        remote-endpoint = <&csi1_ep>;
      };
    };
};

[udhayamoorthi]

can you pls share your whole dtsi , mipi-csi,csir driver codes?

[matthewmader · Accepted solution]

I am not an expert with gstreamer or v4l2, however with Weidong's comment above about the eLCDIF not natively supporting RAW8/RAW10, I revised my gst-launch-1.0 command to the following and it worked (mostly)!

gst-launch-1.0 -v v4l2src ! "video/x-bayer,format=bggr,width=640,height=480,framerate=30/1" ! bayer2rgb ! videoconvert ! autovideosink sync=false

I am getting a very dark image that is off color, but that is probably some other defect in my driver.  However this is enough progress to me to mark this question as closed!

[dhruvinrajpura]

Hello @matthewmader 

I am using same command but getting error, giving you configurations as well in logs

root@imx93-11x11-lpddr4x-evk:~# gst-launch-1.0 -v v4l2src ! "video/x-bayer,format=bggr,width=640,height=480,framerate=30/1" ! bayer2rgb ! videoconvert ! autovideosink sync=false
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
WARNING: from element /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0: Failed to initialize egl: EGL_NOT_INITIALIZED
Additional debug info:
/usr/src/debug/gstreamer1.0-plugins-base/1.24.7.imx/ext/gl/gstglimagesink.c(1140): _ensure_gl_setup (): /GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
/usr/src/debug/gstreamer1.0/1.24.7.imx/libs/gst/base/gstbasesrc.c(3177): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.003832912
Setting pipeline to NULL ...
Freeing pipeline ...
root@imx93-11x11-lpddr4x-evk:~# 



root@imx93-11x11-lpddr4x-evk:~# media-ctl -p
Media controller API version 6.6.23

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial          
bus info        platform:42800000.bus:camera
hw revision     0x0
driver version  6.6.23

Device topology
- entity 1: mxc_isi.0 (16 pads, 2 links, 0 routes)
            type V4L2 subdev subtype Unknown flags 0
	pad0: Sink
		<- "mxc-mipi-csi2.0":4 [ENABLED]
	pad1: Sink
	pad2: Sink
	pad3: Sink
	pad4: Sink
	pad5: Sink
	pad6: Sink
	pad7: Sink
	pad8: Sink
	pad9: Sink
	pad10: Sink
	pad11: Sink
	pad12: Source
		-> "mxc_isi.0.capture":0 [ENABLED]
	pad13: Source
	pad14: Source
	pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
	pad0: Sink
		<- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx219 2-0010":0 [ENABLED,IMMUTABLE]
	pad1: Sink
	pad2: Sink
	pad3: Sink
	pad4: Source
		-> "mxc_isi.0":0 [ENABLED]
	pad5: Source
	pad6: Source
	pad7: Source

- entity 31: imx219 2-0010 (1 pad, 1 link, 0 routes)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
	pad0: Source
		[stream:0 fmt:SRGGB10_1X10/3280x2464 field:none colorspace:raw xfer:none quantization:full-range
		 crop.bounds:(8,8)/3280x2464
		 crop:(8,8)/3280x2464]
		-> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

root@imx93-11x11-lpddr4x-evk:~# 



root@imx93-11x11-lpddr4x-evk:~# v4l2-ctl --list-devices
FSL Capture Media Device (platform:42800000.bus:camera):
	/dev/media0

mxc-isi-cap_v1 (platform:4ae40000.isi:cap_devic):
	/dev/video0

root@imx93-11x11-lpddr4x-evk:~# 

[jhpark_vine]

Can you share how to integrate MipI-csi camera driver?

1. how to configure mipi-csi dphy clock settings?

2. patches of ov5640 camera driver and mx6s_capture  

3. is this work on linux or android ?

[weidong_sun]

Hello matthewmader ,

See below, please!
1. i.MMX8MM MIPI CSI doesn't support capturing RAW10 data for camera by default.
2. i.MX8MM doesn't support ISP, so RAW10 data can't be displayed directly, eLCDIF needs YUV or RGB format to display camera image.
3. About SoftISP we provide from customer
Actually we provide softisp for customer, but the softisp needs OpenCL, and i.MX8MM doesn't support opencl, so softisp can't run on i.MX8MM platfor.
4. About customer's application for RAW data.
---customer should write soft ISP code by herself.
---customer should add these code to mx6s_capture.c to support capturing RAW10 data.(Try it , please!)

.......

+ {
+.name = "RAWRGB10 (SBGGR10)",
+.fourcc = V4L2_PIX_FMT_SBGGR10,
+.pixelformat = V4L2_PIX_FMT_SBGGR10,
+.mbus_code = MEDIA_BUS_FMT_SBGGR10_1X10,
+.bpp = 1,
+}

case V4L2_PIX_FMT_YUV32:
case V4L2_PIX_FMT_SBGGR8:
+case V4L2_PIX_FMT_SBGGR10:
width = pix->width;
break;

.......

Hope these information is helpful for you.
Have a nice day!
B.R,
weidong

[Bhavesh_Bhadricha]

hi weidong,

As per imx8m mini reference manual, RAW format ,pg. 4019 , CSI2 support RAW10.
kindly let us know the reason for mentioning CSI2 not supporting RAW10.
kindly correct if I mis understood your reply

[Sen_tian1]

Hello

在imx8mm上使用uvc camera跑v4l2官方的sample没有问题，但是使用ov5640的camera就跑不通，会报select timeout的错误。

https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/capture.c.html

v4l2-ctl --all

Driver Info:
Driver name : mx6s-csi
Card type : i.MX6S_CSI
Bus info : platform:32e20000.csi1_bridge
Driver version : 5.4.3
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 0
Video input : 0 (Camera: ok)
Format Unknown (00000000): <= Not sure why this is 'Unknown'
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 0, Height 0
Default : Left 0, Top 0, Width 0, Height 0
Pixel Aspect: 1/1
Crop: Left 0, Top 0, Width 0, Height 0
Selection: crop, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: crop_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: crop_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_default, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_bounds, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: compose_padded, Left 0, Top 0, Width 0, Height 0, Flags:
Selection: native_size, Left 0, Top 0, Width 0, Height 0, Flags:
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0

crop，selection的size都为0，是这个的原因嘛？

有什么建议或可以提供一些解决的方向嘛？

十分感谢。

[wjboylan]

Hello @weidong_sun,

I am also using imx219 with imx8mm mini but using Linux Kernel 5.10.35.  I don't see a mx6s_capture.c file to add support for capturing RAW10 data.  Attached are all the imx media files that are available in 5.10.35.  Which file should I update?

Thank you, 

Will 

[udhayamoorthi]

Did you take proper image?

[matthewmader]

Thank you Weidong for your reply,

However I am not attempting to use the RAW10 format yet.  Instead I am using RAW8 which should be natively supported the mx6s_capture driver but I will add your suggestions for when we decided to move to RAW10. 

It is good to know that regardless of RAW10 or RAW8 it needs to be converted to RGB for the LCDIF.  I was unaware of this restriction.

I am not sure I understand your comment about SoftISP?  I am not attempting to use SoftISP.

Thank you for your help.