Adding support for Camera sensor IMX708

[gk123]

I am working on adding support for the Raspberry Pi Cam3 IMX708 camera sensor to the imx93 based board provided by Karo, QSBase93.

https://karo-electronics.github.io/docs/getting-started/qsbase93/quickstart-qsbase93.html

I have followed the following steps to add the driver to the board support package:

1. Git clone the kernel source code
2. Add imx708.c to the folder drivers/media/i2c
3. Add the sensor in the make file and k config
4. Select the sensor as a module in the menu config
5. Save the config as the new def config

For the above I used the below link as a reference

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-change-Linux-Kernel-configuration...

I am now trying to add the camera sensor to the device tree so I simply added a node for the imx708 sensor in the device tree source file for imx219 which is added as a overlay to the device.

I have attached the modified device tree source file for your reference.

However when I check in the menu config the driver is not included already and the device tree file changes revert to the original state.

Can you please help me figure out how I am to modify the device tree to include the imx708 camera sensor as an inbuilt module?

[jimmychan]

You could add "obj-y += imx708.o" directly in the drivers/media/i2c/Makefile.

[gk123]

Thankyou for your reply!

I added a node for the imx708 in the dts file for the baseboard instead of the overlay for imx219.

I am using the same pins that are mentioned for imx219 in the dts file I attached earlier.

I get an error regarding the probe function that it is deferred. This is the error I get when I do not apply the imx219 overlay.

When I do apply the overlay then I get an error regarding the regulators

[ 0.398679] imx708 1-001a: supply VANA1 not found, using dummy regulator

[ 0.398806] imx708 1-001a: supply VANA2 not found, using dummy regulator

[ 0.398949] imx708 1-001a: supply VDDL not found, using dummy regulator

I am unable to understand what the issue might be. Is it possible I missed out on something?

[jimmychan]

So, you remove the VANA, VDIG, VDDL supply setting in imx219?

[gk123]

Yes I tried removing the regulators declarations and the assignment for the imx219 sensor. 

I removed all the pin configurations as well. But only the 1.1V regulator is found and the rest are not being set. 

I tried declaring regulators with different names as well just incase there was conflict as the regulators in the imx219 overlay had the same name but I got the same error. 

Is there anything I might be missing out in the dts? 

[gk123]

The changes I make to the main dts file I don't think are being applied or something. 

How can I ensure that my changes are being applied? 

[jimmychan]

After you change the dts file, you re-compile the code again.  Then the new dtb file come out. Then you copy the dtb file to the FAT partition. For example, when you board boot up, there is a folder something like "mmcblk#p1" (where # is a number for different SDHC device SD card or EMMC) under /run/media/. This is the emmc's FAT partition mounted to the system. You could copy the new dtb file to there to replace the old one.

[gk123]

Thankyou for your reply! I couldn't find the directory you mentioned but managed to apply changes to the dts file using devtool. 

The issue regarding the supply regulators is solved however the camera is not being detected. 

I keep getting the following error in the dmesg log

imx708 1 -001a: failed to read chip id 708

imx708 : probe of 1 -001a failed with error -5

[jimmychan]

Seems it cannot read the register of IMX708 via I2C. You may check the pins connection, I2C signals, power supply to IMX708, etc..

[gk123]

I checked the connections and they seem to be right.  I am using the same pins that were mentioned for the imx219 camera.
I have attached the dts file for your reference. 

[jimmychan]

You have separate two &lpi2c2.  It is better to put them together.

&lpi2c2 {

#address-cells = <1>;
#size-cells = <0>;

clock-frequency = <400000>;
pinctrl-names = "default", "gpio";
pinctrl-0 = <&pinctrl_lpi2c2>;
pinctrl-1 = <&pinctrl_lpi2c2_gpio>;
scl-gpios = <&gpio1 2 (GPIO_ACTIVE_HIGH | GPIO_OPEN_DRAIN)>;
sda-gpios = <&gpio1 3 (GPIO_ACTIVE_HIGH | GPIO_OPEN_DRAIN)>;
status = "okay";

imx708: sensor@1a {
compatible = "sony,imx708";
reg = <0x1a>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_imx708_reset>;
reset-gpios = <&gpio4 27 GPIO_ACTIVE_HIGH>;
clock-names = "xclk";
clocks = <&clk IMX93_CLK_24M>;
assigned-clocks = <&clk IMX93_CLK_24M>;
assigned-clock-rates = <24000000>;
csi_id = <0>;
VANA1-supply = <&reg_2v8>; /* 2.8v */
VANA2-supply = <&reg_1v8>; /* 1.8v */
VDIG-supply = <&reg_1v1>; /* 1.1v */
VDDL-supply = <&reg_1v8>; /* 1.8v */

rotation = <180>;
orientation = <2>;

mclk = <24000000>;
mclk_source = <0>;
mipi_csi;
status = "okay";

port {
imx708_out: endpoint {
remote-endpoint = <&mipi_csi_in>;
data-lanes = <1 2>;
clock-noncontinuous;
link-frequencies = /bits/ 64 <450000000>;
};
};
};

};

[gk123]

Thankyou for your reply I got the camera working!

The camera however requires a library libcap. 

I am unable to include the library in the Yocto build. I tried adding

IMAGE_INSTALL:append = " libcap"
The recipe is including the library in the rootfs image.
Is there anything else required that I might be missing?

[jimmychan]

Adding the IMAGE_INSTALL:append = " libcap" in the conf/local.conf. And then build your whole image again.  Do you find the library in your target board's rootfs?

There is recipe for libcap in the sources/poky/meta/recipes-support/libcap/. You may check do_install that where is the library install to the rootfs. 

[gk123]

The device is detected and registered however in the i2c adapter it is shown as uu in the address 1a and pm_runtime_enable unbalanced error is shown.

Is there any change in the driver code I might be missing? 

[csenatore]

Hello,

I would be interested to know if you have got the imx708 Camera running.
I am also trying to get the camera running, unfortunately with no success.

I am using the sensor driver from the RPI kernel.

I have defined the following nodes in the devicetree:

&lpi2c2 {
	clock-frequency = <400000>;
	imx708: camera-sensor@1a {
		compatible = "sony,imx708";
		reg = <0x1a>;

		clocks = <&clk IMX93_CLK_24M>;
		clock-names = "inclk";
		assigned-clocks = <&clk IMX93_CLK_CAM_PIX>,
					<&clk IMX93_CLK_MIPI_PHY_CFG>;
		assigned-clock-parents = <&clk IMX93_CLK_VIDEO_PLL>,
						<&clk IMX93_CLK_24M>;
		assigned-clock-rates = <140000000>, <24000000>;

		reset-gpios = <&expander 14 GPIO_ACTIVE_HIGH>;

		port {
			imx708_ep: endpoint {
				remote-endpoint = <&csi_ep>;
				data-lanes = <1 2>;
				clock-noncontinuous;
				link-frequencies = /bits/ 64 <450000000>;
			};
		};
	};
};

&cameradev {
	status = "okay";
};

&isi_0 {
	status = "okay";

	cap_device {
		status = "okay";
	};
};

&mipi_csi {
	status = "okay";
	port {
		csi_ep: endpoint {
			remote-endpoint = <&imx708_ep>;
			data-lanes = <2>;
			cfg-clk-range = <28>;
			hs-clk-range = <0x29>;
			bus-type = <4>;
		};
	};
};

The sensor driver is also successfully probed:

[    1.905972] imx708 1-001a: supply vana1 not found, using dummy regulator
[    1.911470] imx708 1-001a: supply vana2 not found, using dummy regulator
[    1.916901] imx708 1-001a: supply vdig not found, using dummy regulator
[    1.922229] imx708 1-001a: supply vddl not found, using dummy regulator
[    1.937442] imx708 1-001a: camera module ID 0x0301

Missing function definitions had to be added to the sensor driver so that a link between ISI -> Mipi-CSI -> and sensor device can be established:

diff --git a/drivers/media/i2c/imx708.c b/drivers/media/i2c/imx708.c
index 29b1d4479c..17f5877487 100644
--- a/drivers/media/i2c/imx708.c
+++ b/drivers/media/i2c/imx708.c
@@ -6,6 +6,8 @@
  * Based on Sony imx477 camera driver
  * Copyright (C) 2020 Raspberry Pi Ltd
  */
+
+#define DEBUG 1
 #include <asm/unaligned.h>
 #include <linux/clk.h>
 #include <linux/delay.h>
@@ -1752,9 +1754,15 @@ static int imx708_identify_module(struct imx708 *imx708)
 	return 0;
 }
 
+static int imx708_s_power(struct v4l2_subdev *sd, int on)
+{
+	return 0;
+}
+
 static const struct v4l2_subdev_core_ops imx708_core_ops = {
 	.subscribe_event = v4l2_ctrl_subdev_subscribe_event,
 	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
+	.s_power = imx708_s_power,
 };
 
 static const struct v4l2_subdev_video_ops imx708_video_ops = {
@@ -1972,6 +1980,18 @@ static int imx708_check_hwcfg(struct device *dev, struct imx708 *imx708)
 	return ret;
 }
 
+
+static int imx708_link_setup(struct media_entity *entity,
+			      const struct media_pad *local,
+			      const struct media_pad *remote, u32 flags)
+{
+	return 0;
+}
+
+static const struct media_entity_operations imx708_sd_media_ops = {
+	.link_setup = imx708_link_setup,
+};
+
 static int imx708_probe(struct i2c_client *client)
 {
 	struct device *dev = &client->dev;
@@ -2038,6 +2058,7 @@ static int imx708_probe(struct i2c_client *client)
 	imx708->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
 			    V4L2_SUBDEV_FL_HAS_EVENTS;
 	imx708->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	imx708->sd.entity.ops = &imx708_sd_media_ops;
 
 	/* Initialize source pads */
 	imx708->pad[IMAGE_PAD].flags = MEDIA_PAD_FL_SOURCE;

The media driver reports this when booting:

[    2.337659] mxc-md 42800000.bus:camera: subdev_notifier_complete
[    2.337664] mxc-md 42800000.bus:camera: mxc_isi.0 entity is found
[    2.337668] mxc-md 42800000.bus:camera: mxc_isi.0.capture entity is found
[    2.337673] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[    2.342989] mxc-md 42800000.bus:camera: mxc_isi.0 entity is found
[    2.342994] mxc-md 42800000.bus:camera: mxc-mipi-csi2.0 entity is found
[    2.342999] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[    2.348128] mxc-md 42800000.bus:camera: mxc-mipi-csi2.0 entity is found
[    2.348133] mx8-img-md: created link [imx708] => [mxc-mipi-csi2.0]
[    2.353009] mxc-md 42800000.bus:camera: mxc_md_create_links

The device nodes /dev/video0 and /dev/media0 were also created.

however, as soon as i try to get an image with gst-launch and v4l2src, the pipeline is cancelled.
The cause becomes clear through dmesg:

mxc_isi.0: set remote fmt fail!

A set_fmt() function of the src device is called in the mxc_isi driver. In this case, the set_fmt function of the mxc-mipi-csi2 driver. This returns -EINVAL when called.

I would be interested to know which changes are still necessary.

[gk123]

Unfortunately I was unable to get the camera to work.  The camera did start however no data was being captured and it always created a 0 byte jpg file. I have attached my g streamer-launch logs below

I would suggest checking the supported formats by reviewing your ISI driver (drivers/media/platform/NXP/IMX-ISI) and your sensor driver.

I did have to define one of the formats in the driver as it was not mentioned in the code. I came across a patch that mentioned the definition of the value.

Also consider cross checking the frequencies in the device tree and the requirements of the imx708 sensor.

Hope this helps.
If you do get the camera do let me know.