IMX8MP ISP camera autofocus

[VoVan]

Hello. In last yocto kirkstone (Linux 5.15.32-2.0.0) uses kernel-module-isp-vvcam 4.2.2.18.0. This isp has an example for working with a sensor AR1335 and autofocus functions are used there. But in Linux there is no mention of this sensor(no dts). We need to test the autofocus performance on this platform, in particular the lens adjustment. For this, as I understand it, in dts you need a description of the sensor and lens. My dts:

/dts-v1/;

#include "imx8mp-evk.dts"

&i2c2 {
/delete-node/ov5640_mipi@3c;

imx258_0: imx258_mipi@1a {
compatible = "sony,imx258";
reg = <0x1a>;
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_csi0_pwn>, <&pinctrl_csi0_rst>, <&pinctrl_csi_mclk>;
clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
clock-names = "mclk";
clock-frequency = <24000000>;
rotation = <180>;
orientation = <2>;
assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
assigned-clock-rates = <24000000>;
csi_id = <0>;
resetb-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
lens-focus = <&dw9714>;
port {
imx258_mipi_0_ep: endpoint {
data-lanes = <1 2 3 4>;
clock-lanes = <0>;
max-pixel-frequency = /bits/ 64 <500000000>;
remote-endpoint = <&mipi_csi0_ep>;
};
};
};

dw9714: dac@c {
compatible = "dongwoon,dw9714";
reg = <0x0c>;
};
};

&i2c3 {
clock-frequency = <100000>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c3>;
status = "okay";

ov5640_1: ov5640_mipi@3c {
status = "disabled";
};
};

&cameradev {
status = "okay";
};

&isi_0 {
status = "disabled";
};

&isi_1 {
status = "disabled";
};

&isp_0 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_0 {
status = "okay";

port@0 {
endpoint {
remote-endpoint = <&imx258_mipi_0_ep>;
data-lanes = <4>;
csis-hs-settle = <16>;
};
};
};

&mipi_csi_1 {
status = "disabled";
};

Sensor work fine, but how to describe autofocus correctly in dts?? ISP driver need lens id and name.

Can you write the correct dts for example for the ar1335 sensor lens?

Thank you

[jheaffey]

Hi,

@malik_cisse Would you be able to share your sensor ISP driver? The example OS08a20 driver provided in the ISP repo has blank implementations for the IsiFocus functions (see Re: iMX8M Plus ISP on Android - how to enable AutoFocus function ? - NXP Community)

[malik_cisse]

Hi VoVan,

did you succeed making DW9714 VCM work with ar1335 ?
Would you share the DTS and driver?
Thanks,

[VoVan]

I made autofocus based on an imx258 camera and a dw9714 lens. For this I wrote my own driver(kernel module) for imx258 sensor and placed it to vvcam/v4l2/sensor/imx258 path. I also wrote userspace driver imx258 for isp and place it to isp-imx/units/isi/drv/imx258. Added dewarp json files and xml tuning files for this sensor. Driver for dw9714 is in the kernel, i just add it in kernel config. I can’t post the sources for the sensor imx258, but I can post the dts. I wrote drivers for the sensor imx258 based on ar1335 (this sensor has support for working with a lens).

&i2c2 {

imx258_0: imx258_mipi@1a {
compatible = "sony,imx258-pdaf";
reg = <0x1a>;
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_csi0_pwn>, <&pinctrl_csi0_rst>, <&pinctrl_csi_mclk>;
clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
clock-names = "mclk";
clock-frequency = <24000000>;
rotation = <180>;
orientation = <2>;
assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
assigned-clock-rates = <24000000>;
csi_id = <0>;
resetb-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
lens-focus = <&dw9714>;
port {
imx258_mipi_0_ep: endpoint {
data-lanes = <1 2 3 4>;
clock-lanes = <0>;
max-pixel-frequency = /bits/ 64 <500000000>;
remote-endpoint = <&mipi_csi0_ep>;
};
};
};

dw9714: dw9714@c {
compatible = "dongwoon,dw9714";
id = <9714>;
reg = <0x0c>;
};

};

&cameradev {
status = "okay";
};

&isi_0 {
status = "disabled";
};

&isi_1 {
status = "disabled";
};

&isp_0 {
status = "okay";
};

&isp_1 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_0 {
status = "okay";
port@0 {
endpoint {
remote-endpoint = <&imx258_mipi_0_ep>;
data-lanes = <4>;
csis-hs-settle = <16>;
};
};
};

[malik_cisse]

Thank you for the feedback VoVan,
You had already sent the DTS file earlier.
For the DW9714 I have also seen that this driver ist part of the stock Linux kernel. However for the i.MX8MP ISP to steer the VMC, there are additional API IOCTLs to be added.

Obviously in your case, there is no Autofocus implemented, is it?
BR, Malik

[VoVan]

I also had to rewrite the dw9714 driver in the kernel to make it compatible with isp. I used driver dw9790 as the base(vvcam/v4l2/focus/vcm_dw9790). I attach driver for dw9714 lens

[malik_cisse]

Thank you VoVan, 

I tried your modified driver and it works. ISP logic is moving the VCM motor automatically according to image content.
Thank you for the code. This saved me at least one day of work.
The resulting autofocus reacts not very smoothly though as can you can see in attached video. One can hear the VCM motor clicking very quickly and adjust the focus in an instant.
Is this also the case for your setup or did you manage to have smooth motor movements and thus smooth auto-focus ?
Thx

[VoVan]

The imx8 isp is responsible for focusing the image by analyzing the input data. It would be very good if the ISP had support for PDAF technology, but this chip does not and will not have it. In XML i set for autofocus:
<af enable="true">
<algorithm oneshot="false">2</algorithm>
</af>

[VoVan]

Thus, during the kernel loading process, dw9714 is detected via i2c and the device |dev|v4l-subdev is created to work with this lens.

Next in the isp driver, the OpenMotorDevice function detects this device and uses an ioctl to focus it.

It's simple, good luck in job

[malik_cisse]

ah, this is very interesting. I am going to test it right now.
Thank you

[VoVan]

We have already done autofocus and checked its operation(sensor: IMX258 lens:dw9714). Works fine, as for contrast autofocus. Is it planned to add support for PDAF (Phase Detection Autofocus) to ISP?

[joanxie]

don't find such plan

[joanxie]

auto focus doesn't defined in the dts file, should support in the ISP v4l2 API, you can refer to the document as below:

"https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces..."

for more detailed information, pls contact professional service support team

"https://www.nxp.com/support/support/nxp-engineering-services/professional-support-for-processors-and..."