How to enable media device on imx8mm?

jerry_liu · ‎08-22-2023

Hello,

I want to enable media devices for camera sensors on imx8mm.

Then I found the topic related to that:

https://community.nxp.com/t5/i-MX-Processors/imx8mm-mipi-csi-capture-with-tc358743-hdmi-gt-mipi-not-...

Environment: Yocto Kirkstone 5.15.71

I modified imx8mm.dtsi:

csi1_bridge: csi1_bridge@32e20000 {
				compatible = "fsl,imx8mm-csi", "fsl,imx7-csi";
				reg = <0x32e20000 0x1000>;
				interrupts = <GIC_SPI 16 IRQ_TYPE_LEVEL_HIGH>;
				clocks = <&clk IMX8MM_CLK_CSI1_ROOT>;
				clock-names = "mclk";
				power-domains = <&dispmix_pd>;
				status = "disabled";

				port {
					csi_in: endpoint {
						remote-endpoint = <&imx8mm_mipi_csi_out>;
					};
				};
			};

mipi_csi: mipi-csi@32e30000 {
				compatible = "fsl,imx8mm-mipi-csi2", "fsl,imx8mm-mipi-csi";
				reg = <0x32e30000 0x1000>;
				interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
				assigned-clocks = <&clk IMX8MM_CLK_CSI1_CORE>,
						  <&clk IMX8MM_CLK_CSI1_PHY_REF>;
				assigned-clock-parents = <&clk IMX8MM_SYS_PLL2_1000M>,
							  <&clk IMX8MM_SYS_PLL2_1000M>;
				clock-frequency = <500000000>;
				clocks = <&clk IMX8MM_CLK_DISP_APB_ROOT>,
					 <&clk IMX8MM_CLK_CSI1_ROOT>,
					 <&clk IMX8MM_CLK_CSI1_PHY_REF>,
					 <&clk IMX8MM_CLK_DISP_AXI_ROOT>;
				clock-names = "pclk", "wrap", "phy", "axi";
				bus-width = <4>;
				power-domains = <&mipi_pd>;
				status = "disabled";

				ports {
					#address-cells = <1>;
					#size-cells = <0>;

					port@0 {
						reg = <0>;
					};

					port@1 {
						reg = <1>;
						imx8mm_mipi_csi_out: endpoint {
							remote-endpoint = <&csi_in>;
						};
					};
				};
			};

and modified sensor dtsi below that:

...

fragment@1 {
		target = <&mipi_csi>;
		__overlay__ {
			status = "okay";
			#address-cells = <1>;
			#size-cells = <0>;
			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				port@0 {
					reg = <0>;
					imx8mm_mipi_csi_in: endpoint {
						remote-endpoint = <&ov5640_mipi1_ep>;
						data-lanes = <1 2>;
					};
				};
			};
		};
	};

...
fragment@2 {
		target = <&i2c2>;
		__overlay__ {
			#address-cells = <1>;
			#size-cells = <0>;
			status = "okay";

...
			ov5640_mipi: ov5640_mipi@3c {
				compatible = "ovti,ov5640";
				reg = <0x3c>;
				status = "okay";
				pinctrl-names = "default";
				pinctrl-0 = <&pinctrl_csi_pwn>, <&pinctrl_csi_rst>;
				clocks = <&clk IMX8MM_CLK_CLKO1>;
				clock-names = "xclk";
				assigned-clocks = <&clk IMX8MM_CLK_CLKO1>;
				assigned-clock-parents = <&clk IMX8MM_CLK_24M>;
				assigned-clock-rates = <24000000>;
				csi_id = <0>;
				powerdown-gpios = <&pca9555_a21 2 GPIO_ACTIVE_HIGH>;
				reset-gpios = <&gpio1 5 GPIO_ACTIVE_LOW>;
				mclk = <24000000>;
				mclk_source = <0>;
				mipi_csi;
				nvmem = <&ov5640_otp_0>;
				nvmem-names = "calib-data";
				port {
					ov5640_mipi1_ep: endpoint {
						remote-endpoint = <&imx8mm_mipi_csi_in>;
					};
				};
			};
		};

However, I cannot ENABLE the sensor source pad to the csi sink pad:

# media-ctl -p
Media controller API version 5.15.71

Media device information
------------------------
driver          imx7-csi
model           imx-media
serial
bus info        platform:32e20000.csi1_bridge
hw revision     0x0
driver version  5.15.71

Device topology
- entity 1: csi (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "csis-32e30000.mipi-csi":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "csi capture":0 [ENABLED,IMMUTABLE]

- entity 4: csi capture (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "csi":1 [ENABLED,IMMUTABLE]

- entity 10: csis-32e30000.mipi-csi (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
                <- "ov5640 1-003c":0 []
        pad1: Source
                [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
                -> "csi":0 [ENABLED,IMMUTABLE]

- entity 15: ov5640 1-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
        pad0: Source
                -> "csis-32e30000.mipi-csi":0 []

Why it cannot link between sensor entity and csi entity and how to resolve it?

rashichouhan · ‎08-28-2023

Hi,

It seems like you've made the necessary modifications to the device tree and subdevice configurations to enable the camera sensor on imx8mm. However, you're facing issues with establishing the link between the sensor entity and the CSI entity. You can try this, maybe it helps-

You must ensure that the remote endpoint connections are correctly specified in the device tree. In your sensor's device tree entry, verify that the remote-endpoint property of the sensor's source pad matches the sink pad of the CSI entity and vice versa.

In addition to this, inspect the media controller API output to establish a connection, the output of media-ctl -p indicates the current state of the media devices and their connections. Check if there are any issues reported, and make sure that the entities and links are properly enabled and connected.

The URL you shared somewhere help you. https://community.nxp.com/t5/i-MX-Processors/imx8mm-mipi-csi-capture-with-tc358743-hdmi-gt-mipi-not-... mlops course

Thanks

joanxie · ‎08-23-2023

do you mind sharing logfile and result from "v4l2-ctl --list-device"?

jerry_liu · ‎08-23-2023

Hi joanxei,

Sure! here is my log:

# v4l2-ctl --list-device
imx-capture (platform:32e20000.csi1_bridge):
        /dev/video0
        /dev/media0

joanxie · ‎08-24-2023

pls upload your boot up full logfile too

jerry_liu · ‎08-24-2023

Hi Joanxie,

Attached file is the boot log.

I can link manually the pads between sensor and csi, but I still cannot get the sensor format by v4l2-ctl.

First run the command 'media-ctl -p':

# media-ctl -p
Media controller API version 5.15.71

Media device information
------------------------
driver imx7-csi
model  imx-media
serial
bus info        platform:32e20000.csi1_bridge
hw revision     0x0
driver version  5.15.71

Device topology
- entity 1: csi (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "imx7-mipi-csis.0":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "csi capture":0 [ENABLED,IMMUTABLE]

- entity 4: csi capture (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "csi":1 [ENABLED,IMMUTABLE]

- entity 10: imx7-mipi-csis.0 (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "ov5640 1-003c":0 []
        pad1: Source
                [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "csi":0 [ENABLED,IMMUTABLE]

- entity 15: ov5640 1-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
        pad0: Source
                -> "imx7-mipi-csis.0":0 []

Then run the command to link pads:

# media-ctl --link "'ov5640 1-003c':0->'imx7-mipi-csis.0':0[1]"

Check the linked result:

# media-ctl -p
Media controller API version 5.15.71

Media device information
------------------------
driver imx7-csi
model imx-media
serial
bus info        platform:32e20000.csi1_bridge
hw revision     0x0
driver version  5.15.71

Device topology
- entity 1: csi (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "imx7-mipi-csis.0":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:UYVY8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "csi capture":0 [ENABLED,IMMUTABLE]

- entity 4: csi capture (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "csi":1 [ENABLED,IMMUTABLE]

- entity 10: imx7-mipi-csis.0 (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                <- "ov5640 1-003c":0 [ENABLED]
        pad1: Source
                [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "csi":0 [ENABLED,IMMUTABLE]

- entity 15: ov5640 1-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
        pad0: Source
                -> "imx7-mipi-csis.0":0 [ENABLED]

But I still cannot get the formats from sensor by v4l2-ctl:

# v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'UYVY' (UYVY 4:2:2)
                Size: Continuous 1x1 - 65535x65535
        [1]: 'YUYV' (YUYV 4:2:2)
                Size: Continuous 1x1 - 65535x65535
        [2]: 'YU12' (Planar YUV 4:2:0)
                Size: Continuous 1x1 - 65535x65535
        [3]: 'YV12' (Planar YVU 4:2:0)
                Size: Continuous 1x1 - 65535x65535
        [4]: '422P' (Planar YUV 4:2:2)
                Size: Continuous 1x1 - 65535x65535
        [5]: 'NV12' (Y/CbCr 4:2:0)
                Size: Continuous 1x1 - 65535x65535
        [6]: 'NV16' (Y/CbCr 4:2:2)
                Size: Continuous 1x1 - 65535x65535
        [7]: 'RGBP' (16-bit RGB 5-6-5)
                Size: Continuous 1x1 - 65535x65535
        [8]: 'RGB3' (24-bit RGB 8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [9]: 'BGR3' (24-bit BGR 8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [10]: 'BX24' (32-bit XRGB 8-8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [11]: 'XR24' (32-bit BGRX 8-8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [12]: 'RX24' (32-bit XBGR 8-8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [13]: 'XB24' (32-bit RGBX 8-8-8-8)
                Size: Continuous 1x1 - 65535x65535
        [14]: 'BA81' (8-bit Bayer BGBG/GRGR)
                Size: Continuous 1x1 - 65535x65535
        [15]: 'GBRG' (8-bit Bayer GBGB/RGRG)
                Size: Continuous 1x1 - 65535x65535
        [16]: 'GRBG' (8-bit Bayer GRGR/BGBG)
                Size: Continuous 1x1 - 65535x65535
        [17]: 'RGGB' (8-bit Bayer RGRG/GBGB)
                Size: Continuous 1x1 - 65535x65535
        [18]: 'BYR2' (16-bit Bayer BGBG/GRGR)
                Size: Continuous 1x1 - 65535x65535
        [19]: 'GB16' (16-bit Bayer GBGB/RGRG)
                Size: Continuous 1x1 - 65535x65535
        [20]: 'GR16' (16-bit Bayer GRGR/BGBG)
                Size: Continuous 1x1 - 65535x65535
        [21]: 'RG16' (16-bit Bayer RGRG/GBGB)
                Size: Continuous 1x1 - 65535x65535
        [22]: 'GREY' (8-bit Greyscale)
                Size: Continuous 1x1 - 65535x65535
        [23]: 'Y10 ' (10-bit Greyscale)
                Size: Continuous 1x1 - 65535x65535
        [24]: 'Y12 ' (12-bit Greyscale)
                Size: Continuous 1x1 - 65535x65535

joanxie · ‎08-28-2023

what formats do you need test? what's result when you use gstreamer？let me reproduce this on my board firslty

jerry_liu · ‎08-30-2023

Hi joanxie,

any update on that issue?

jerry_liu · ‎08-28-2023

Hi joanxie,

We use ov5640 sensor and modify mpi csi driver(mx6s-csi -> imx7-mipi-csis) and sensor driver (mxc/ov5640_mipi.c -> i2c/ov5640.c) to generate media devices:

we hope it shows these formats with imx7-mipi-csis driver not mx6s-csi:

# v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.067s (15.000 fps)

and it shows the supported resolution by gstreamer like:

# gst-device-monitor-1.0
...
Device found:

        name  : i.MX6S_CSI
        class : Video/Source
        caps  : video/x-raw, format=YUY2, width=2592, height=1944, pixel-aspect-ratio=1/1, framerate=15/1
                video/x-raw, format=YUY2, width=1920, height=1080, pixel-aspect-ratio=1/1, framerate=30/1
                video/x-raw, format=YUY2, width=1280, height=720, pixel-aspect-ratio=1/1, framerate=30/1
                video/x-raw, format=YUY2, width=720, height=480, pixel-aspect-ratio=1/1, framerate=30/1
                video/x-raw, format=YUY2, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1
                video/x-raw, format=YUY2, width=320, height=240, pixel-aspect-ratio=1/1, framerate=30/1
                video/x-raw(format:Interlaced), format=YUY2, width=320, height=240, pixel-aspect-ratio=1/1, framerate=30/1, interlace-mode=alternate
        properties:
                udev-probed = true
                device.bus_path = platform-32e20000.csi1_bridge
                sysfs.path = /sys/devices/platform/soc@0/32c00000.bus/32e20000.csi1_bridge/video4linux/video0
                device.subsystem = video4linux
                device.product.name = i.MX6S_CSI
                device.capabilities = :capture:
                device.api = v4l2
                device.path = /dev/video0
                v4l2.device.driver = mx6s-csi
                v4l2.device.card = i.MX6S_CSI
                v4l2.device.bus_info = platform:32e20000.csi1_bridge
                v4l2.device.version = 331591 (0x00050f47)
                v4l2.device.capabilities = 2216689665 (0x84200001)
                v4l2.device.device_caps = 69206017 (0x04200001)
        gst-launch-1.0 v4l2src ! ...

joanxie · ‎09-08-2023

I don't know why you change all of drivers, obviously these issue are related to your driver, since you use your own drivers, you need debug by yourself, I suggest that you can add more print in the driver, then to check if your mipi csi create correct link from camera to mipi csi, for nxp, imx8mm uses this mipi csi driver

"https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/drivers/media/platform/mxc/capture/mxc_mipi_csi.c...

you can compare yours with this, you can focus on these codes

/* Attach sensors linked to csi receivers */
for_each_available_child_of_node(parent, node) {
if (of_node_cmp(node->name, "port"))
continue;

/* The csi node can have only port subnode. */
port = of_get_next_child(node, NULL);
if (!port)
continue;
rem = of_graph_get_remote_port_parent(port);
of_node_put(port);
if (rem == NULL) {
v4l2_info(&state->v4l2_dev,
"Remote device at %s not found\n",
port->full_name);
return -1;
}

you also can check your dts settings, refer to your information, link from mipi csi to csi is ok, but ov5640 to mipi csi failed,

ports {
					#address-cells = <1>;
					#size-cells = <0>;

					port@0 {
						reg = <0>;
					};

					port@1 {
						reg = <1>;
						imx8mm_mipi_csi_out: endpoint {
							remote-endpoint = <&csi_in>;
						};

check mipi csi get proper information from your dts settings

How to enable media device on imx8mm?

How to enable media device on imx8mm?

i.MX 8M | i.MX 8M Mini | i.MX 8M Nano

Linux

Multimedia