Camera Driver Guide for i.MX 93 EVK

[Navee_nK]

Hi,

I am trying to integrate my AR1335 sensor (which has its own ISP) with the i.MX93-EVK. I have written the driver under drivers/media/platform/imx8/ar1335.c and am able to load the driver and generate the video node. I have attached the dmesg and media-ctl logs for your reference.
However, I am unable to list my controls when I run v4l2-ctl -d 0 -l or --list-ctrls. I’ve tried changing the video device, but nothing seems to work.
In my driver, I have registered a new function with:
static struct v4l2_subdev_core_ops ar1335_subdev_core_ops = {
.queryctrl = ar1335_queryctrl,
};
and registered it in v4l2-subdev.h with the following:
int (*queryctrl)(struct v4l2_subdev *sd, struct v4l2_queryctrl *qc);

I’m not sure where I’m going wrong. Could you help me understand the flow for getting these kernel calls working with my i.MX93-EVK kit?

Thanks,
Naveen

[joanxie]

what the result of "v4l2-ctl --list-device" ?

[Navee_nK]

Hi Joanxie,
Below is the result:
v4l2-ctl --list-device
FSL Capture Media Device (platform:42800000.bus:camera):
/dev/media0

mxc-isi-cap_v1 (platform:4ae40000.isi:cap_devic):
/dev/video0

Kindly help me understand the kernel flow of registering these properly.

Thanks,
Naveen.

[Navee_nK]

Hi,

Adding to the above points, I have also implemented the below changes in the kernel driver drivers/staging/media/imx/imx8-isi-cap.c
static const struct v4l2_ioctl_ops mxc_isi_capture_ioctl_ops = {
.vidioc_queryctrl = mxc_vidioc_queryctrl,
};

static int mxc_vidioc_queryctrl(struct file *file, void *fh,
struct v4l2_queryctrl *a)
{
struct mxc_isi_cap_dev *isi_cap = video_drvdata(file);
struct v4l2_subdev *sd;
int ret = 0;
printk("MXC Vidioc Queryctrl\n");
sd = mxc_get_remote_subdev(&isi_cap->sd, __func__);
printk("Subdevice: %p, type: %s\n", sd, sd->name);

if (!sd)
return -EINVAL;
ret = v4l2_subdev_call(sd, core, queryctrl, a);
if (ret < 0) {
printk("v4l2_subdev_call failed with error %d\n", ret);
}

And with the above implementation when given v4l2-ctl -d 0 -l command am getting the below mentioned logs in dmesg
[ 607.910646] MXC Vidioc Queryctrl
[ 607.910652] Subdevice: 0000000063e9cea4, type: mxc-mipi-csi2.0
[ 607.910658] v4l2_subdev_call failed with error -515

I don't know what else is missing, kindly help me with this. Below are some additional dmesg logs related to this

[ 13.901196] MXC Media Dev Probe Call
[ 13.901309] MXC ISI Subdev registered
[ 13.901313] MXC ISI Register Cap Device
[ 13.904724] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[ 13.904840] mx8-img-md: Registered sensor subdevice : ar1335 (1)
[ 13.904848] Inside MXC MD Create Links function
[ 13.904859] mx8-img-md: 1 created link [mxc_isi.0] => [mxc_isi.0.capture]
[ 13.904873] mx8-img-md: 3 created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ 13.904879] mx8-img-md: 5 created link [ar1335] => [mxc-mipi-csi2.0]
[ 13.904885] mxc-md 42800000.bus:camera: mxc_md_create_links


Thanks,
Naveen.

[joanxie]

I'm not sure why you add queryctrl, but did you add this in camera driver? refer to the error message, maybe your camera driver didn't enable queryctrl

[Navee_nK]

Hi @joanxie,

I have solved this issue by implementing few funcions in the kernel drivers and now this list controls works fine. But right now am facing issues with stream on call which is the default implementation. When I give v4l2-ctl -d 0 --stream-mmap I am facing the below kernel crash issue.
[ 696.215896] ------------[ cut here ]------------
[ 696.215903] WARNING: CPU: 0 PID: 675 at drivers/media/common/videobuf2/videobuf2-core.c:827 vb2_core_reqbufs+0x224/0x548
[ 696.215921] Modules linked in: crct10dif_ce polyval_ce polyval_generic snd_soc_fsl_asoc_card layerscape_edac_mod snd_soc_imx_audmux snd_soc_imx_card snd_soc_wm8962 snd_soc_fsl_micfil rtc_pcf2127 rtc_pcf2131 flexcan can_dev snd_soc_fsl_xcvr snd_soc_fsl_sai imx_pcm_dma snd_soc_fsl_utils imx8_media_dev(C) fuse
[ 696.215975] CPU: 0 PID: 675 Comm: v4l2-ctl Tainted: G WC 6.6.52-g5b10ef983033-dirty #24
[ 696.215982] Hardware name: NXP i.MX93 11X11 EVK board (DT)
[ 696.215986] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 696.215992] pc : vb2_core_reqbufs+0x224/0x548
[ 696.215997] lr : vb2_core_reqbufs+0x1e0/0x548
[ 696.216001] sp : ffff8000843e3b00
[ 696.216004] x29: ffff8000843e3b00 x28: ffff00000436b8b8 x27: 0000000000000000
[ 696.216014] x26: ffff8000825fe000 x25: ffff00000436a750 x24: 0000000000000001
[ 696.216023] x23: ffff8000843e3d38 x22: ffff00000436a790 x21: 0000000000000000
[ 696.216032] x20: ffff800081db7000 x19: ffff00000436a6e8 x18: 0000000000000000
[ 696.216041] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffffd7d962c8
[ 696.216050] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 696.216058] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 696.216067] x8 : ffff00000436a988 x7 : ffff8000824a4978 x6 : ffff00000436a080
[ 696.216076] x5 : 0000000000000001 x4 : ffff00000436a750 x3 : 0000000000000001
[ 696.216084] x2 : 0000000000000001 x1 : ffff8000843e3b60 x0 : 0000000000000000
[ 696.216093] Call trace:
[ 696.216097] vb2_core_reqbufs+0x224/0x548
[ 696.216102] vb2_ioctl_reqbufs+0xb0/0xe8
[ 696.216109] v4l_reqbufs+0x50/0x64
[ 696.216116] __video_do_ioctl+0x1a0/0x424
[ 696.216123] video_usercopy+0x330/0x684
[ 696.216129] video_ioctl2+0x18/0x24
[ 696.216135] v4l2_ioctl+0x3c/0x5c
[ 696.216140] __arm64_sys_ioctl+0xac/0xec
[ 696.216149] invoke_syscall+0x44/0x104
[ 696.216158] el0_svc_common.constprop.1+0xc0/0xe0
[ 696.216165] do_el0_svc+0x1c/0x28
[ 696.216171] el0_svc+0x3c/0xd4
[ 696.216178] el0t_64_sync_handler+0xa0/0xc4
[ 696.216183] el0t_64_sync+0x174/0x178
[ 696.216189] ---[ end trace 0000000000000000 ]---

Kindly help me with solving this.

Adding to that kindly help me with the correct understanding of this flow like when we give the v4l2-ctl user space call through which driver we will get the call at last to my camera driver. Then to understand this we need more debug prints on v4l2-ioctls and other v4l2 and csi and isi cap related function prints, kindly provide me the steps to enable these debug prints. I have tried Dynamic Debugging but that ain't giving me any dmesg logs.

Thanks,
Naveen.

[Navee_nK]

Hi @joanxie,

An update with my above reply,
When I start my stream by setting the formats properly I am not getting any kernel crashes. However am facing this below issue in the next step
[ 35.446028] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_streamon
[ 35.446060] mxc_isi.0: set remote fmt fail! -22
Which belongs to the below attached part of the code in the file imx8-isi-cap.c in the function
static int mxc_isi_source_fmt_init(struct mxc_isi_cap_dev *isi_cap)
ret = v4l2_subdev_call(src_sd, pad, set_fmt, NULL, &src_fmt);
if (ret < 0 && ret != -ENOIOCTLCMD) {
v4l2_err(&isi_cap->sd, "set remote fmt fail! %d\n", ret);
return ret;
}

Am using the format MEDIA_BUS_FMT_UYVY8_2X8 which I have added like src_fmt.format.code = MEDIA_BUS_FMT_UYVY8_2X8; before v4l2_subdev_call function and in imx8-isi-fmt.c file as well I have added the below part
{
.name = "UYVY-16",
.fourcc = V4L2_PIX_FMT_UYVY,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_YUV422_1P8P,
.memplanes = 1,
.colplanes = 1,
.mbus_code = MEDIA_BUS_FMT_UYVY8_2X8,
},{
.name = "YUYV-16",
.fourcc = V4L2_PIX_FMT_YUYV,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_YUV422_1P8P,
.memplanes = 1,
.colplanes = 1,
.align = 3,
.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8,
},
and my doubt now is in the file imx8-mipi-csi2-sam.c file in static const struct csis_pix_format mipi_csis_formats[] = {}
it is given
{
.code = MEDIA_BUS_FMT_UYVY8_1X16,
.fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
.data_alignment = 16,
}, {
.code = MEDIA_BUS_FMT_UYVY8_2X8,
.fmt_reg = MIPI_CSIS_ISPCFG_FMT_YCBCR422_8BIT,
.data_alignment = 16,
},

But in dwc-mipi-csi2.c only the below is given
{
.code = MEDIA_BUS_FMT_YUYV8_2X8,
.fmt_reg = 0x18,
},
There is no code for UYVY8 but it is there in the function disp_mix_gasket_config
case MEDIA_BUS_FMT_UYVY8_2X8:
case MEDIA_BUS_FMT_UYVY8_1X16:
case MEDIA_BUS_FMT_VYUY8_2X8:
fmt_val = DT_YUV422_8;

I need the fmt_reg for MEDIA_BUS_FMT_UYVY8_2X8 and MEDIA_BUS_FMT_UYVY8_1X16.
to set in static const struct csi2h_pix_format dwc_csi2h_formats[].

I think the above mentioned issue of dmesg log is because of this.
[ 35.446028] isi-capture 4ae40000.isi:cap_device: mxc_isi_cap_streamon
[ 35.446053] Subdevice in fmt: 00000000178376df, type: mxc-mipi-csi2.0
[ 35.446060] mxc_isi.0: set remote fmt fail! -22

Kindly help me to solve this issue.

Thanks,
Naveen

[joanxie]

if you need MEDIA_BUS_FMT_UYVY8_2X8 , did you change the "src_fmt.format.code= MEDIA_BUS_FMT_UYVY8_2X8" in the mxc_isi_source_fmt_init of mx8-isi-cap.c? refer to the error message, it seems you didn't set correct format in this function

[Navee_nK]

Hi @joanxie,

I have explicitly changed it to src_fmt.format.code = MEDIA_BUS_FMT_UYVY8_2X8; and even printed and checked and have attached the debug log print earlier and below is that and it is being correctly printed as 8198 d (0x2006 h)

[ 33.975283] Inside DWC MIPI CSI2 Get Fmt
[ 33.975299] pad 0 code 0 field 0
[ 33.975304] colorspace 0 width 0 height 0

[ 33.975309] AR1335 pad 0 code 8198 field 1
[ 33.975313] colorspace 8 width 640 height 480

[ 33.975317] Ret value 0
[ 33.975319] pad 0 code 8198 field 1
[ 33.975322] colorspace 8 width 640 height 480

[ 33.975326] csi2h_fmt code 8198 csi2h_fmt reg 25

[ 33.980369] Inside MXC isi cap stream on
[ 33.980389] Config Parm Entry
[ 33.980392] MXC ISI Source Fmt init
[ 33.980395] UYVY_2x8 8198
[ 33.980401] pad 4 code 8198 field 0
[ 33.980404] colorspace 8 width 640 height 480

[ 33.980408] dst_f->fmt->mbus_code: 8198
[ 33.980412] mxc_isi.0: set remote fmt fail! -2

[joanxie]

could you confirm what format your camera output? refer to the AR1335 data sheet, it seems ar1335 output raw data, and we have solution is that ar1335 plus external ISP chip, pls double confirm this from camera vendor

[Navee_nK]

Hi @joanxie,

We have an ISP in our Camera board so the output our board sends in UYVY only.

I have also tried getting few logs and below are they.

[ 33.975283] Inside DWC MIPI CSI2 Get Fmt
[ 33.975299] pad 0 code 0 field 0
[ 33.975304] colorspace 0 width 0 height 0

[ 33.975309] AR1335 pad 0 code 8198 field 1
[ 33.975313] colorspace 8 width 640 height 480

[ 33.975317] Ret value 0
[ 33.975319] pad 0 code 8198 field 1
[ 33.975322] colorspace 8 width 640 height 480

[ 33.975326] csi2h_fmt code 8198 csi2h_fmt reg 25

[ 33.980369] Inside MXC isi cap stream on
[ 33.980389] Config Parm Entry
[ 33.980392] MXC ISI Source Fmt init
[ 33.980395] UYVY_2x8 8198
[ 33.980401] pad 4 code 8198 field 0
[ 33.980404] colorspace 8 width 640 height 480

[ 33.980408] dst_f->fmt->mbus_code: 8198
[ 33.980412] mxc_isi.0: set remote fmt fail! -22
This is prints I have added in those functions

static int dwc_mipi_csi2_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *cfg, struct v4l2_subdev_format *format)
{
struct dwc_mipi_csi2_host *csi2h = sd_to_dwc_mipi_csi2h(sd);
struct v4l2_mbus_framefmt *mf = &csi2h->format;
struct media_pad *source_pad;
struct v4l2_subdev *sen_sd;
struct csi2h_pix_format const *csi2h_fmt;
int ret;

printk("Inside DWC MIPI CSI2 Get Fmt\n");
/* Get remote source pad */
source_pad = dwc_csi2_get_remote_sensor_pad(csi2h);
if (!source_pad) {
v4l2_err(&csi2h->sd, "%s, No remote pad found!\n", __func__);
return -EINVAL;
}

/* Get remote source pad subdev */
sen_sd = dwc_get_remote_subdev(csi2h, __func__);
if (!sen_sd) {
v4l2_err(&csi2h->sd, "%s, No remote subdev found!\n", __func__);
return -EINVAL;
}

format->pad = source_pad->index;

printk("pad %d code %d field %d\n", format->pad, format->format.code, format->format.field);
printk("colorspace %d width %d height %d\n\n", format->format.colorspace, format->format.width, format->format.height);

ret = v4l2_subdev_call(sen_sd, pad, get_fmt, NULL, format);
if (ret < 0) {
v4l2_err(&csi2h->sd, "%s, call get_fmt of subdev failed!\n", __func__);
return ret;
}
printk("Ret value %d\n", ret);

printk("pad %d code %d field %d\n", format->pad, format->format.code, format->format.field);
printk("colorspace %d width %d height %d\n\n", format->format.colorspace, format->format.width, format->format.height);
memcpy(mf, &format->format, sizeof(struct v4l2_mbus_framefmt));

csi2h_fmt = find_csi2h_format(mf->code);
if (!csi2h_fmt) {
printk("Not\n");
csi2h_fmt = &dwc_csi2h_formats[0];
mf->code = csi2h_fmt->code;
}
csi2h->csi2h_fmt = csi2h_fmt;
printk("csi2h_fmt code %d csi2h_fmt reg %d\n\n", csi2h->csi2h_fmt->code, csi2h->csi2h_fmt->fmt_reg);

return 0;
}

This is the prints I have added in my driver

static int ar1335_get_fmt(struct v4l2_subdev *sd,struct v4l2_subdev_state *state,struct v4l2_subdev_format *format)
{
struct i2c_client *client = v4l2_get_subdevdata(sd);
struct v4l2_mbus_framefmt *mf = &format->format;
struct ar1335 *sensor = to_ar1335_dev(client);
int ret = 0;

if (format->pad)
return -EINVAL;

format->format.code = sensor->fmt.code;
format->format.colorspace = sensor->fmt.colorspace;
format->format.field = V4L2_FIELD_NONE;

format->format.width = sensor->pix.width;
format->format.height = sensor->pix.height;
printk("AR1335 pad %d code %d field %d\n", format->pad, format->format.code, format->format.field);
printk("colorspace %d width %d height %d\n\n", format->format.colorspace, format->format.width, format->format.height);

return ret;

}

These are the dmesg logs we get when do stream on with the previously said command.
in imx8-isi-cap.c inside the above said error function am also implementing this
src_fmt.format.code = MEDIA_BUS_FMT_UYVY8_2X8;
before that v4l2_subdev_call.
in dwc-mipi-csi2.c file I have added the below
static const struct csi2h_pix_format dwc_csi2h_formats[] = {
 {
.code = MEDIA_BUS_FMT_UYVY8_2X8,
.fmt_reg = 0x19,
}, {
.code = MEDIA_BUS_FMT_UYVY8_1X16,
.fmt_reg = 0x1A,

},
I want to know the fmt_reg val for these as MEDIA_BUS_FMT_UYVY8_2X8 is my format. And by  this error  [33.980412] mxc_isi.0: set remote fmt fail! -22 where should I concentrate, either in dwc-mipi-csi2.c file as it is the subdev of imx8-isi-cap.c or should I debug on imx8-isi-cap.c.

I don't know what I am missing. Kindly help me to solve it.

Thanks,
Naveen.

[joanxie]

what ISP chip do you use? your information is limited, I don't know how your HW connection, you should connect camera with imx93 via this ISP chip, right, so you should add this chip in the dts file and use this ISP driver

[Navee_nK]

Hi @joanxie,

My camera board has internal ISP in it and the Hardware connections were made correct as we have got it working in verdin with imx8mp. Here the only issue with the kernel registration and the format not being set properly, which is driver side issue. I have also seen the similar kind of issue

mxc_isi.0: set remote fmt fail!

in this link https://community.nxp.com/t5/i-MX-Processors/Adding-support-for-Camera-sensor-IMX708/m-p/1988663 Kindly help me with the driver registration part like the reasons for this issue and how to solve it. The issue is not with the Hardware here.

Thanks,
Naveen.

[Navee_nK]

Hi @joanxie ,

The same driver now I have used with 6.1 kernel version and it worked fine. Kindly let me know the driver changes I need to make with respect to the kernel 6.6 and also let me know if you have any driver source tested by getting and setting the format and got stream. Kindly help me to solve this issue with the latest 6.6 kernel.

Thanks,
Naveen.

[joanxie]

until now, I didn't have your driver, could you give me patch of isi or any other drives you change based on the nxp source code, and this patch can work with 6.1 but failed with 6.6, so this driver failed with 6.6, the error message is "mxc_isi.0: set remote fmt fail!" right?

[Navee_nK]

Hi @joanxie,

There was an issue with my driver and I have resolved it and now I can able to successfully set the resolution, format and stream via v4l2-ctl. However, now I am facing another issue, that is when I tried to stream it via gst-launch(gstreamer), I can able to successfully stream the 1280x720 and 1920x1080 resolution, but when I tried to stream it with 3840x2160(4K) the streaming is not proper and it looks corrupted, attached the image of all the three resolutions output. Adding to that, below is my device tree entry in imx93-11x11-evk.dts file:

ar1335_mipi: ar1335_mipi@42 {
compatible = "econ,ar1335_ff";
reg = <0x42>;
clocks = <&cam_xtal>;
clock-names = "xclk";
clock-frequency = <24000000>;
camera_mipi_lanes = <0x2>;
camera-mipi-clk = <800>;
csi_id = <0>;
mclk = <24000000>;
mclk_source = <0>;
mipi_csi;
cam_xtal: cam-xtal {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <25000000>;
clock-output-names = "cam_xtal";
};
port {
ar0830_mipi_ep: endpoint {
remote-endpoint = <&mipi_csi_ep>;
data-lanes = <2>;
cfg-clk-range = <28>;
hs-clk-range = <0x2b>;
};
};
};
&mipi_csi {
status = "okay";
port {
mipi_csi_ep: endpoint {
remote-endpoint = <&ar0830_mipi_ep>;
data-lanes = <2>;
cfg-clk-range = <28>;
hs-clk-range = <0x2b>;
bus-type = <4>;
};
};
};

Note: The output from our camera module is correct as we have tested the same source with IMX8MPLUS 2 lane mode and we haven't faced any issues there.

I doubt the issue may be due to sampling, Kindly let me know how to handle the clocks here and let me know the overall mipi csi data rate supported by this imx93 evk which is very important for me to work further and what are the other parameters I need to check on. Kindly help me solve this issue as this is very critical for me.

Thanks,
Naveen.

[joanxie]

imx93 mipi csi can support up to 200Mhz clock, you check if you set the mipi csi clock as max clock already

[Navee_nK]

Hi @joanxie 

i) imx93 mipi csi can support up to 200Mhz clock. By this do you mean per lane or totally ? If totally then it can only support 400Mbps data rate on the whole ?

From the above images, we are using data rate as 1200Mbps common for all the three resolution and with that I can able to get 720 and 1080p but not 4K. Kindly let me know what should I do to resolve this. And also how to change the clocks on the platform side ?

ii) you check if you set the mipi csi clock as max clock already. How to check that in platform if it is set to max already ?

Thanks,
Naveen.

[joanxie]

1) IMX93 has one mipi csi which can support up to 200Mhz, for data rate, each lane can support up to 1.5G, total 2 data lane is 3Gbps, but in the dts, you just need the clock

2)for 4K support, you also need consider the ISI driver, for the function chain_buf which has 2k limitation,

https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/drivers/staging/media/imx/imx8-isi-hw.c#L145C12-L...

in this function, if (frm->o_width > ISI_2K)  to set 0,  because imx93 only one ISI couldn't support chain buffer

[Navee_nK]

Hi @joanxie,

Now, the 4K streaming issue has been resolved. Gone through the file https://github.com/nxp-imx/linux-imx/blob/lf-6.6.y/drivers/staging/media/imx/imx8-isi-hw.c#L145C12-L... and functions you have mentioned. Thanks.

Now, I have two doubts here. You said, IMX93 has one mipi csi which can support up to 200Mhz, for data rate, each lane can support up to 1.5G, total 2 data lane is 3Gbps, but in the dts, you just need the clock.

1. From your above reply kindly help me clarify what do you mean by when you said IMX93 has one mipi csi which can support up to 200Mhz. If data rate supported per lane is 1.5Gbps (1500 Mbps), then the clock supported per lane should be around 700Mhz right ? as MIPI function as double data rate. This is what is confusing me. Because I want to know so that I can modify my clock rate accordingly. Kindly help me clarify how this 200Mhz clock supports 1.5Gbps data rate. How this functions.

2. but in the dts, you just need the clock. -> what do you mean by this ? in dts what are the parameters need to be added regarding this clock. I have attached my dts entries earlier. Kindly tell me the clocks needed in the dts file, what are they ?

Thanks,
Naveen.

[marco_c_variscite]

Hi @Navee_nK 

Can you please clarify how did you resolve the 4K streaming issue? Did you modify the imx8-isi-hw.c driver to skip the if (frm->o_width > ISI_2K) test for imx93?

Thanks

Marco