topic Re: Adding support for Camera sensor IMX708 in i.MX Processors

Adding support for Camera sensor IMX708

gk123 — Mon, 28 Oct 2024 02:13:36 GMT

I am working on adding support for the Raspberry Pi Cam3 IMX708 camera sensor to the imx93 based board provided by Karo, QSBase93.

https://karo-electronics.github.io/docs/getting-started/qsbase93/quickstart-qsbase93.html

I have followed the following steps to add the driver to the board support package:

Git clone the kernel source code
Add imx708.c to the folder drivers/media/i2c
Add the sensor in the make file and k config
Select the sensor as a module in the menu config
Save the config as the new def config

For the above I used the below link as a reference

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-change-Linux-Kernel-configuration-file-in-Yocto-Project/ta-p/1112097

I am now trying to add the camera sensor to the device tree so I simply added a node for the imx708 sensor in the device tree source file for imx219 which is added as a overlay to the device.

I have attached the modified device tree source file for your reference.

However when I check in the menu config the driver is not included already and the device tree file changes revert to the original state.

Can you please help me figure out how I am to modify the device tree to include the imx708 camera sensor as an inbuilt module?

Re: Adding support for Camera sensor IMX708

jimmychan — Tue, 29 Oct 2024 03:21:27 GMT

You could add "obj-y += imx708.o" directly in the drivers/media/i2c/Makefile.

Re: Adding support for Camera sensor IMX708

gk123 — Tue, 29 Oct 2024 17:21:29 GMT

Thankyou for your reply!

I added a node for the imx708 in the dts file for the baseboard instead of the overlay for imx219.

I am using the same pins that are mentioned for imx219 in the dts file I attached earlier.

I get an error regarding the probe function that it is deferred. This is the error I get when I do not apply the imx219 overlay.

When I do apply the overlay then I get an error regarding the regulators

[ 0.398679] imx708 1-001a: supply VANA1 not found, using dummy regulator

[ 0.398806] imx708 1-001a: supply VANA2 not found, using dummy regulator

[ 0.398949] imx708 1-001a: supply VDDL not found, using dummy regulator

I am unable to understand what the issue might be. Is it possible I missed out on something?

Re: Adding support for Camera sensor IMX708

jimmychan — Mon, 04 Nov 2024 07:10:07 GMT

So, you remove the VANA, VDIG, VDDL supply setting in imx219?

Re: Adding support for Camera sensor IMX708

gk123 — Mon, 04 Nov 2024 12:36:28 GMT

Yes I tried removing the regulators declarations and the assignment for the imx219 sensor.

I removed all the pin configurations as well. But only the 1.1V regulator is found and the rest are not being set.

I tried declaring regulators with different names as well just incase there was conflict as the regulators in the imx219 overlay had the same name but I got the same error.

Is there anything I might be missing out in the dts?

Re: Adding support for Camera sensor IMX708

gk123 — Mon, 04 Nov 2024 12:41:14 GMT

The changes I make to the main dts file I don't think are being applied or something.

How can I ensure that my changes are being applied?

Re: Adding support for Camera sensor IMX708

jimmychan — Tue, 05 Nov 2024 02:01:30 GMT

After you change the dts file, you re-compile the code again. Then the new dtb file come out. Then you copy the dtb file to the FAT partition. For example, when you board boot up, there is a folder something like "mmcblk#p1" (where # is a number for different SDHC device SD card or EMMC) under /run/media/. This is the emmc's FAT partition mounted to the system. You could copy the new dtb file to there to replace the old one.

Re: Adding support for Camera sensor IMX708

gk123 — Wed, 06 Nov 2024 07:17:47 GMT

Thankyou for your reply! I couldn't find the directory you mentioned but managed to apply changes to the dts file using devtool.

The issue regarding the supply regulators is solved however the camera is not being detected.

I keep getting the following error in the dmesg log

imx708 1 -001a: failed to read chip id 708

imx708 : probe of 1 -001a failed with error -5

Re: Adding support for Camera sensor IMX708

jimmychan — Wed, 06 Nov 2024 08:47:38 GMT

Seems it cannot read the register of IMX708 via I2C. You may check the pins connection, I2C signals, power supply to IMX708, etc..

Re: Adding support for Camera sensor IMX708

gk123 — Mon, 11 Nov 2024 02:26:01 GMT

I checked the connections and they seem to be right. I am using the same pins that were mentioned for the imx219 camera.
I have attached the dts file for your reference.

Re: Adding support for Camera sensor IMX708

jimmychan — Wed, 13 Nov 2024 02:46:11 GMT

You have separate two &lpi2c2. It is better to put them together.

&lpi2c2 {

#address-cells = <1>;
#size-cells = <0>;

clock-frequency = <400000>;
pinctrl-names = "default", "gpio";
pinctrl-0 = <&pinctrl_lpi2c2>;
pinctrl-1 = <&pinctrl_lpi2c2_gpio>;
scl-gpios = <&gpio1 2 (GPIO_ACTIVE_HIGH | GPIO_OPEN_DRAIN)>;
sda-gpios = <&gpio1 3 (GPIO_ACTIVE_HIGH | GPIO_OPEN_DRAIN)>;
status = "okay";

imx708: sensor@1a {
compatible = "sony,imx708";
reg = <0x1a>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_imx708_reset>;
reset-gpios = <&gpio4 27 GPIO_ACTIVE_HIGH>;
clock-names = "xclk";
clocks = <&clk IMX93_CLK_24M>;
assigned-clocks = <&clk IMX93_CLK_24M>;
assigned-clock-rates = <24000000>;
csi_id = <0>;
VANA1-supply = <&reg_2v8>; /* 2.8v */
VANA2-supply = <&reg_1v8>; /* 1.8v */
VDIG-supply = <&reg_1v1>; /* 1.1v */
VDDL-supply = <&reg_1v8>; /* 1.8v */

rotation = <180>;
orientation = <2>;

mclk = <24000000>;
mclk_source = <0>;
mipi_csi;
status = "okay";

port {
imx708_out: endpoint {
remote-endpoint = <&mipi_csi_in>;
data-lanes = <1 2>;
clock-noncontinuous;
link-frequencies = /bits/ 64 <450000000>;
};
};
};

};

Re: Adding support for Camera sensor IMX708

gk123 — Wed, 13 Nov 2024 05:08:09 GMT

Thankyou for your reply I got the camera working!

The camera however requires a library libcap.

I am unable to include the library in the Yocto build. I tried adding

IMAGE_INSTALL:append = " libcap"
The recipe is including the library in the rootfs image.
Is there anything else required that I might be missing?

Re: Adding support for Camera sensor IMX708

jimmychan — Wed, 13 Nov 2024 06:41:34 GMT

Adding the IMAGE_INSTALL:append = " libcap" in the conf/local.conf. And then build your whole image again. Do you find the library in your target board's rootfs?

There is recipe for libcap in the sources/poky/meta/recipes-support/libcap/. You may check do_install that where is the library install to the rootfs.

Re: Adding support for Camera sensor IMX708

gk123 — Wed, 13 Nov 2024 10:44:30 GMT

The device is detected and registered however in the i2c adapter it is shown as uu in the address 1a and pm_runtime_enable unbalanced error is shown.

Is there any change in the driver code I might be missing?

Re: Adding support for Camera sensor IMX708

csenatore — Mon, 20 Jan 2025 09:45:30 GMT

Hello,

I would be interested to know if you have got the imx708 Camera running.
I am also trying to get the camera running, unfortunately with no success.

I am using the sensor driver from the RPI kernel.

I have defined the following nodes in the devicetree:

&lpi2c2 { clock-frequency = <400000>; imx708: camera-sensor@1a { compatible = "sony,imx708"; reg = <0x1a>; clocks = <&clk IMX93_CLK_24M>; clock-names = "inclk"; assigned-clocks = <&clk IMX93_CLK_CAM_PIX>, <&clk IMX93_CLK_MIPI_PHY_CFG>; assigned-clock-parents = <&clk IMX93_CLK_VIDEO_PLL>, <&clk IMX93_CLK_24M>; assigned-clock-rates = <140000000>, <24000000>; reset-gpios = <&expander 14 GPIO_ACTIVE_HIGH>; port { imx708_ep: endpoint { remote-endpoint = <&csi_ep>; data-lanes = <1 2>; clock-noncontinuous; link-frequencies = /bits/ 64 <450000000>; }; }; }; }; &cameradev { status = "okay"; }; &isi_0 { status = "okay"; cap_device { status = "okay"; }; }; &mipi_csi { status = "okay"; port { csi_ep: endpoint { remote-endpoint = <&imx708_ep>; data-lanes = <2>; cfg-clk-range = <28>; hs-clk-range = <0x29>; bus-type = <4>; }; }; };

The sensor driver is also successfully probed:

[ 1.905972] imx708 1-001a: supply vana1 not found, using dummy regulator [ 1.911470] imx708 1-001a: supply vana2 not found, using dummy regulator [ 1.916901] imx708 1-001a: supply vdig not found, using dummy regulator [ 1.922229] imx708 1-001a: supply vddl not found, using dummy regulator [ 1.937442] imx708 1-001a: camera module ID 0x0301

Missing function definitions had to be added to the sensor driver so that a link between ISI -> Mipi-CSI -> and sensor device can be established:

diff --git a/drivers/media/i2c/imx708.c b/drivers/media/i2c/imx708.c index 29b1d4479c..17f5877487 100644 --- a/drivers/media/i2c/imx708.c +++ b/drivers/media/i2c/imx708.c @@ -6,6 +6,8 @@ * Based on Sony imx477 camera driver * Copyright (C) 2020 Raspberry Pi Ltd */ + +#define DEBUG 1 #include <asm/unaligned.h> #include <linux/clk.h> #include <linux/delay.h> @@ -1752,9 +1754,15 @@ static int imx708_identify_module(struct imx708 *imx708) return 0; } +static int imx708_s_power(struct v4l2_subdev *sd, int on) +{ + return 0; +} + static const struct v4l2_subdev_core_ops imx708_core_ops = { .subscribe_event = v4l2_ctrl_subdev_subscribe_event, .unsubscribe_event = v4l2_event_subdev_unsubscribe, + .s_power = imx708_s_power, }; static const struct v4l2_subdev_video_ops imx708_video_ops = { @@ -1972,6 +1980,18 @@ static int imx708_check_hwcfg(struct device *dev, struct imx708 *imx708) return ret; } + +static int imx708_link_setup(struct media_entity *entity, + const struct media_pad *local, + const struct media_pad *remote, u32 flags) +{ + return 0; +} + +static const struct media_entity_operations imx708_sd_media_ops = { + .link_setup = imx708_link_setup, +}; + static int imx708_probe(struct i2c_client *client) { struct device *dev = &client->dev; @@ -2038,6 +2058,7 @@ static int imx708_probe(struct i2c_client *client) imx708->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS; imx708->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR; + imx708->sd.entity.ops = &imx708_sd_media_ops; /* Initialize source pads */ imx708->pad[IMAGE_PAD].flags = MEDIA_PAD_FL_SOURCE;

The media driver reports this when booting:

[ 2.337659] mxc-md 42800000.bus:camera: subdev_notifier_complete [ 2.337664] mxc-md 42800000.bus:camera: mxc_isi.0 entity is found [ 2.337668] mxc-md 42800000.bus:camera: mxc_isi.0.capture entity is found [ 2.337673] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture] [ 2.342989] mxc-md 42800000.bus:camera: mxc_isi.0 entity is found [ 2.342994] mxc-md 42800000.bus:camera: mxc-mipi-csi2.0 entity is found [ 2.342999] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0] [ 2.348128] mxc-md 42800000.bus:camera: mxc-mipi-csi2.0 entity is found [ 2.348133] mx8-img-md: created link [imx708] => [mxc-mipi-csi2.0] [ 2.353009] mxc-md 42800000.bus:camera: mxc_md_create_links

The device nodes /dev/video0 and /dev/media0 were also created.

however, as soon as i try to get an image with gst-launch and v4l2src, the pipeline is cancelled.
The cause becomes clear through dmesg:

mxc_isi.0: set remote fmt fail!

A set_fmt() function of the src device is called in the mxc_isi driver. In this case, the set_fmt function of the mxc-mipi-csi2 driver. This returns -EINVAL when called.

I would be interested to know which changes are still necessary.

Re: Adding support for Camera sensor IMX708

gk123 — Tue, 21 Jan 2025 16:29:24 GMT

Unfortunately I was unable to get the camera to work. The camera did start however no data was being captured and it always created a 0 byte jpg file. I have attached my g streamer-launch logs below

I would suggest checking the supported formats by reviewing your ISI driver (drivers/media/platform/NXP/IMX-ISI) and your sensor driver.

I did have to define one of the formats in the driver as it was not mentioned in the code. I came across a patch that mentioned the definition of the value.

Also consider cross checking the frequencies in the device tree and the requirements of the imx708 sensor.

Hope this helps.
If you do get the camera do let me know.