topic Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK in i.MX Processors

Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Mon, 06 Dec 2021 22:32:51 GMT

Hello all,

I am working on porting an IMX378 camera driver to the i.MX 8MP EVK board. I am following the i.MX 8M Plus Camera and Display Guid, section 2 Camera Sensor Porting Guide.

I have followed section 2.7 Camera Sensor Driver in V4L2 mode and I get the following dmesg output:

6.440486] imx378: loading out-of-tree module taints kernel.
[ 6.449105] enter imx378_probe
[ 6.458998] enter imx378_retrieve_capture_properties
[ 6.466931] imx378 1-001a: supply DOVDD not found, using dummy regulator
[ 6.474198] imx378 1-001a: supply DVDD not found, using dummy regulator
[ 6.481092] imx378 1-001a: supply AVDD not found, using dummy regulator
[ 6.489336] enter imx378_regulator_enable
[ 6.493550] enter imx378_set_clk_rate
[ 6.497365] enter imx378_power_on
[ 6.500962] enter imx378_reset
[ 6.539481] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.558768] enter imx378_read_reg
[ 6.562991] enter imx378_read_reg
[ 6.590179] imx378_probe camera mipi imx378, is found
[ 6.648388] mx8-img-md: Registered sensor subdevice: imx378 1-001a (1)
[ 6.740810] enter imx378_link_setup
[ 6.744958] mx8-img-md: created link [imx378 1-001a] => [mxc-mipi-csi2.0]
[ 6.751831] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ 6.768425] enter viv_dwe_init_module
[ 6.772649] enter dwe_hw_probe
[ 6.778619] enter viv_isp_init_module
[ 6.783472] vvcam dewa74] vvcam isp driver registered
[ 7.206408] enter imx378_priv_ioctl (cmd = -2140645888)
[ 7.211671] enter imx378_query_capability

My media-ctl output is as follows:

root@imx8mpevk:~# media-ctl -p
[ 2002.167891] enter imx378_get_fmt
Media controller API version 5.10.52

Media device information
------------------------
driver mxc-md
model FSL Capture Media Device
serial 
bus info 
hw revision 0x0
driver version 5.10.52

Device topology
- entity 1: mxc-mipi-csi2.0 (8 pads, 1 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev0
pad0: Sink
<- "imx378 1-001a":0 [ENABLED,IMMUTABLE]
pad1: Sink
pad2: Sink
pad3: Sink
pad4: Source
pad5: Source
pad6: Source
pad7: Source

- entity 10: imx378 1-001a (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev1
pad0: Source
[fmt:SGBRG10_1X10/4056x3040 field:none]
-> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

root@imx8mpevk:~#

At this point I can see that the I2C communication is working, however I cannot for the life of me figure out how to tell V4L2 to start the stream so I can view it. When I list V4L2 devices I do not see a video source:

root@imx8mpevk:~# v4l2-ctl --list-devices
[ 2130.712823] enter imx378_priv_ioctl (cmd = -2140645888)
[ 2130.718106] enter imx378_query_capability
():
/dev/v4l-subdev0

(csi0):
/dev/v4l-subdev1

FSL Capture Media Device (platform:mxc-md):
/dev/media0

vsi_v4l2dec (platform:vsi_v4l2dec):
/dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
/dev/video0

What am I missing?

My DTS file is as below:

// SPDX-License-Identifier: (GPL-2.0+ OR MIT) /* * Copyright 2020 NXP * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2 * of the License, or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. */ /dts-v1/; #include "imx8mp-evk.dts" &i2c2 { /delete-node/ov5640_mipi@3c; imx378_0: imx378_mipi@1a { compatible = "sony,imx378"; reg = <0x1a>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_csi0_pwn>, <&pinctrl_csi0_rst>, <&pinctrl_csi_mclk>; clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>; clock-names = "csi_mclk"; assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>; assigned-clock-parents = <&clk IMX8MP_CLK_24M>; assigned-clock-rates = <24000000>; csi_id = <0>; pwn-gpios = <&gpio2 11 GPIO_ACTIVE_HIGH>; rst-gpios = <&gpio1 6 GPIO_ACTIVE_LOW>; mclk = <24000000>; mclk_source = <0>; status = "okay"; port { imx378_mipi_0_ep: endpoint { data-lanes = <1 2 3 4>; clock-lanes = <0>; max-pixel-frequency = /bits/ 64 <500000000>; remote-endpoint = <&mipi_csi0_ep>; }; }; }; }; &i2c3 { clock-frequency = <100000>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_i2c3>; status = "okay"; ov5640_1: ov5640_mipi@3c { status = "disabled"; }; }; &cameradev { status = "okay"; }; &isi_0 { status = "disabled"; }; &isi_1 { status = "disabled"; }; &isp_0 { status = "okay"; }; &dewarp { status = "okay"; }; &mipi_csi_0 { status = "okay"; clock-frequency = <500000000>; assigned-clock-rates = <500000000>; port@0 { endpoint { remote-endpoint = <&imx378_mipi_0_ep>; data-lanes = <4>; csis-hs-settle = <16>; }; }; }; &mipi_csi_1 { status = "disabled"; };

ISI driver, and VVCAM driver Attached

Thank you,

Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Wed, 08 Dec 2021 02:05:34 GMT

Hi @chrispeterson,

I suppose that you are porting the camera driver for being used with the on-chip ISP since I saw that you disabled the ISI interfaces. For visualizing the output on the display, it would be better to run one of the gstreamer commands as below (as indicated in the end of /opt/imx8-isp/bin/run.sh) :

# gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=1920,height=1080" ! queue ! imxvideoconvert_g2d ! waylandsink # gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=3840,height=2160" ! queue ! imxvideoconvert_g2d ! waylandsink # gst-launch-1.0 -v v4l2src device=/dev/video0 ! waylandsink

You can also stream the video out of the board via an Ethernet connection by following the instruction of following knowledge sharing :https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-8-GStreamer-User-Guide/ta-p/1098942

Or, you can also use the provided utility /opt/imx8-isp/bin/video_test to dump either RAW or processed (YUV) images/frames out.

Hope that helps,

Khang

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Wed, 08 Dec 2021 14:53:19 GMT

Hi @khang_letruong

Thank you for the response. I found that I can get a /dev/video2 device when I run

modprobe vvcam_video

I also get a couple more /dev/v4l-subdev* devices and another /dev/media1. The output of media-ctl is below

root@imx8mpevk:/opt/imx8-isp/bin# media-ctl -d 1 -p
Media controller API version 5.10.52

Media device information
------------------------
driver vvcam-video
model viv_media
serial 
bus info 
hw revision 0x0
driver version 5.10.52

Device topology
- entity 1: viv_v4l20 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "vvcam-isp.0":0 [ENABLED]

- entity 5: vvcam-dwe.0 (2 pads, 0 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev2
pad0: Source
pad1: Sink

- entity 8: vvcam-isp.0 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev3
pad0: Source
-> "viv_v4l20":0 [ENABLED]

So, at this point it looks like I have the camera connected to the CSI in media 0 and the ISP output connected to dev/video0. I don't see a way to connect the CSI to the ISP.

I attempted to run those commands you suggested and I get the following result:

root@imx8mpevk:/opt/imx8-isp/bin# ./isp_media_server CAMERA0 &
[1] 1223
root@imx8mpevk:/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video2 ! waylandsink
[ 171.538913] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 217.940400] enter imx378_priv_ioctl (cmd = -2140645888)
[ 217.951996] enter imx378_query_capability
[ 217.967758] enter isp_mi_stop
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

[ 223.359212] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 223.518907] viv_post_event: unsubscribed event id =14 type=0x08002000
e=0x08002000
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
[ 223.846933] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 224.014916] viv_post_event: unsubscribed event id =7 type=0x08002000
[ 224.178915] viv_post_event: unsubscribed event id =6 type=0x08002000
[ 224.342910] viv_post_event: unsubscribed event id =10 type=0x08002000
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../git/sys/v4l2/gstv4l2src.c(659): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:06.506453500
Setting pipeline to NULL ...
[ 224.538919] viv_post_event: unsubscribed event id =1 type=0x08002000
Freeing pipeline ...

I seems that isp_media_server is expecting a native driver, however the one I have is v4l2 based (which I defaulted to since I didn't see any of the vvcam/native folders that the porting guide mentions).

On another note, I found a different porting guide to what I was using (IMX8MPCSPUG.pdf) and section 6.2.6 mentions that I need to add some defines to the cmake command, however I am using the Yocto build system and I do not see a way to do this.

Further, section 6.2.4 mentions needing a "<sensor>.drv" file and I see n way of creating one.

Thank you,

Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Wed, 08 Dec 2021 17:24:02 GMT

Hi @chrispeterson ,

Firstly, the imx378.drv is important and it is used by the isp_media_server (or communicate with this later). Missing it is critical since there's no way for the isp_media_server to propagate the IOCTL calls to VVCAM kernel driver.

The IMX378.c that you created in isp-imx will serve to generate the imx378.drv. However, it might not be taken into account yet and you will need to add it via some relevant CMakelists.txt files.

Together with IMX378.c, you will ned an IMX378.xml for the calibration parameters,

Aside from the imx378.drv, there's will be libimx378.so, libimx378.so.1, libimx378.so.1.0.0 generated. Let's suppose that you build your isp-imx with ./build-all-isp.sh command, then the generated files would be found in build_output_release_partial/opt/imx8-isp/ and build_output_release_partial/usr/lib/. You will need to copy them to to the target board :

scp -r build_output_release_partial/opt/imx8-isp target@target-ip:/opt/ scp -r build_output_release_partial/usr/lib/* target@target-ip:/usr/lib/

Also, you will need to adapt the start_isp.sh script and run.sh script to take into account of loading of your newly generated imx378.drv and IMX378.xml as well as the vvcam kernel driver (imx378.ko) described below :

For lower level vvcam kernel module (imx378.ko generated from imx378_mipi.c), you will need to compile it and copy to target board under /lib/module/<kernel-version>/extra/sensor/imx378/ (you might need to create this folder if it does not exist).

On another note, I found a different porting guide to what I was using (IMX8MPCSPUG.pdf) and section 6.2.6 mentions that I need to add some defines to the cmake command, however I am using the Yocto build system and I do not see a way to do this.
Further, section 6.2.4 mentions needing a "<sensor>.drv" file and I see n way of creating one.

-- > You can modify the isp-imx recipe to add patch for your custom stuff of IMX378 described above. Or you can build it separately then scp-ing to the target board.

You can inspire the example of ov2775 which is a good example.

Regards,

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Thu, 09 Dec 2021 15:18:21 GMT

Hi @khang_letruong

Thank you very much for the help. I was generating the imx378.drv file and had the IMX378.xml file already, they were just not being copied over. I did have to modify run.sh to add my sensor.

As a side note I looked through the porting guide and none of this information is in there.

So, I am now getting further. I can start the isp_media_server with the properly and it appears to run. I get a /dev/video2 output device that I can run gst-laucher-1.0 or video_test on, however both of those fail no matter what settings I use. This appears to be a result of an error being returned by the start function of the ISP driver:

root@imx8mpevk:/opt/imx8-isp/bin#
root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx378
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx378"...
Starting isp_media_server with configuration file CAMERA0
[ 1575.105955] enter imx378_priv_ioctl (cmd = -2140645888)
[ 1575.111223] enter imx378_query_capability
[ 1575.163435] enter imx378_priv_ioctl (cmd = -2140645888)
[ 1575.168688] enter imx378_query_capability
loading [imx378.drv]...
[ 1575.190939] enter imx378_priv_ioctl (cmd = 261)
[ 1575.197381] enter imx378_query_supports
WARN : [DEWARP_HW] dw[ 1575.201767] enter imx378_priv_ioctl (cmd = -2140645888)
e type /dev/v4l-subdev0 0
[ 1575.209895] enter imx378_query_capability
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev1 1
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2
loading [imx378.drv]...
[ 1575.241807] enter imx378_priv_ioctl (cmd = 261)
[ 1575.248236] enter imx378_query_supports
loading [imx378.drv]...
[ 1575.252899] enter imx378_priv_ioctl (cmd = 261)
[ 1575.259324] enter imx378_query_supports
loading [imx378.drv]...
[ 1575.264312] enter imx378_priv_ioctl (cmd = 261)
[ 1575.270708] enter imx378_query_supports
[ 1575.285873] enter imx378_priv_ioctl (cmd = 261)
[ 1575.290449] enter imx378_query_supports
[ 1575.294366] enter imx378_priv_ioctl (cmd = 257)
[ 1575.298938] enter imx378_priv_ioctl (cmd = 260)
[ 1575.303495] enter imx378_get_clk
[ 1575.306798] enter imx378_priv_ioctl (cmd = 259)
[ 1575.311367] enter imx378_priv_ioctl (cmd = 256)
[ 1575.315970] enter imx378_priv_ioctl (cmd = 262)
[ 1575.320519] enter imx378_set_sensor_mode
[ 1575.324479] enter imx378_priv_ioctl (cmd = 263)
[ 1575.329027] enter imx378_get_sensor_mode
[ 1575.332983] enter imx378_priv_ioctl (cmd = 261)
[ 1575.337522] enter imx378_query_supports
isp input: 4056x3040 RAW10
isp output: 3840x2160 YUV422I
dewarp input: 3840x2160 YUV422I
dewarp output: 3840x2160 YUV422I
[ 1575.431535] enter isp_mi_stop
[ 1575.436987] enter imx378_set_fmt
[ 1575.440238] enter imx378_write_reg_arry
[ 1575.748238] enter imx378_get_format_code
[ 1575.752190] enter imx378_priv_ioctl (cmd = 285)
[ 1575.756739] enter imx378_set_test_pattern
[ 1575.760817] enter imx378_write_reg
[ 1575.764872] enter imx378_write_reg
[ 1575.768871] enter imx378_write_reg
[ 1575.772940] enter imx378_write_reg
[ 1575.776975] enter imx378_priv_ioctl (cmd = 279)
[ 1575.781529] enter imx378_set_fps
[ 1575.784774] enter imx378_write_reg
[ 1575.788775] enter imx378_write_reg
[ 1575.792785] enter imx378_write_reg
[ 1575.796770] enter imx378_write_reg
[ 1575.800766] enter imx378_priv_ioctl (cmd = 263)
[ 1575.805320] enter imx378_get_sensor_mode
[ 1575.809508] enter imx378_priv_ioctl (cmd = 274)
[ 1575.814056] enter imx378_set_exp
[ 1575.817397] enter imx378_write_reg
[ 1575.821409] enter imx378_write_reg
[ 1575.825409] enter imx378_write_reg
[ 1575.829436] enter imx378_write_reg
[ 1575.833441] enter imx378_priv_ioctl (cmd = 277)
[ 1575.837986] enter imx378_set_gain
[ 1575.841330] enter imx378_write_reg
[ 1575.845322] enter imx378_write_reg
[ 1575.849325] enter imx378_write_reg
[ 1575.853327] enter imx378_write_reg
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_engine_interface.cpp:2156: start() = 13(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:427: inputConnect() = 13(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:1359: cameraConnect() = 13(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_engine_interface.cpp:2187: streamingStart() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:1084: streamingStart() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:1639: previewStart() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_engine_interface.cpp:2198: streamingStop() = 12(RET_WRONG_STATE)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:1104: streamingStop() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/[ 1581.094935] enter imx378_priv_ioctl (cmd = 272)
[ 1581.099651] enter imx378_s_stream
[ 1581.103013] enter imx378_write_reg
camera/cam_operations.cpp:1652: p[ 1581.107345] enter imx378_priv_ioctl (cmd = 260)
reviewStop() = 12(ret)
[ERR] /h[ 1581.114106] enter imx378_get_clk
ome/nxa14866/data/verisilicon_sw_[ 1581.120208] enter imx378_priv_ioctl (cmd = 259)
isp/units/cam_device/source/camer[ 1581.127589] enter imx378_priv_ioctl (cmd = 257)
a/cam_engine_interface.cpp:757: cprocEnableSet() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:820: inputDisconnect() = 12(ret)
[ERR] /home/nxa14866/data/verisilicon_sw_isp/units/cam_device/source/camera/cam_operations.cpp:1372: cameraDisconnect() = 12(ret)

As shown above, all the proper configuration is being written to the camera and it is just about to start the stream when the ISP start function fails with a return code of 13.

Looking through the code, return code of 13 is

#define RET_INVALID_PARM 13 //!< invalid parameter

However I cannot find any information on what parameter is wrong.

Any thoughts?

-Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Thu, 09 Dec 2021 22:37:38 GMT

Hello@khang_letruong ,

Thank you very much for the help. I was able to get further now that I have copied over the .drv and .xml files and updated the run.sh.

However I am not getting a parameter error from within the start function in cam_engine_interface.cpp as shown below.

root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx378
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx378"...
Killing preexisting instances of video_test and isp_media_server:
PID TTY STAT TIME COMMAND 1033 pts/0 S+ 0:00 ./video_test -w 4048 -h 3040 -f RAW10 -t file -m 0 -d 2
[ 296.878947] viv_post_event: unsubscribed event id =4 type=0x08002000
Starting isp_media_server with configuration file CAMERA0
[ 297.038965] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 304.427751] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.439580] enter imx378_query_capability
[ 304.483156] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.488406] enter imx378_query_capability
loading [imx378.drv]...
[ 304.510213] enter imx378_priv_ioctl (cmd = 261)
[ 304.516684] enter imx378_query_supports
WARN : [DEWARP_HW] dw[ 304.521098] enter imx378_priv_ioctl (cmd = -2140645888)
e type /dev/v4l-subdev0 0
[ 304.529218] enter imx378_query_capability
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev1 1
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2
loading [imx378.drv]...[ 304.561912] enter imx378_priv_ioctl (cmd = 261)

[ 304.568089] enter imx378_query_supports
loading [imx378.drv]...[ 304.572873] enter imx378_priv_ioctl (cmd = 261)

[ 304.579125] enter imx378_query_supports
loading [imx378.drv]...[ 304.584259] enter imx378_priv_ioctl (cmd = 261)

[ 304.590577] enter imx378_query_supports
[ 304.606186] enter imx378_priv_ioctl (cmd = 261)
[ 304.610754] enter imx378_query_supports
[ 304.614608] enter imx378_priv_ioctl (cmd = 257)
[ 304.619173] enter imx378_priv_ioctl (cmd = 260)
[ 304.623720] enter imx378_get_clk
[ 304.626963] enter imx378_priv_ioctl (cmd = 259)
[ 304.631530] enter imx378_priv_ioctl (cmd = 256)
[ 304.636073] enter imx378_priv_ioctl (cmd = 262)
[ 304.640612] enter imx378_set_sensor_mode
[ 304.644553] enter imx378_priv_ioctl (cmd = 263)
[ 304.649109] enter imx378_get_sensor_mode
[ 304.653054] enter imx378_priv_ioctl (cmd = 261)
[ 304.657593] enter imx378_query_supports
isp input: 4048x3040 RAW10
isp output: 3840x2160 YUV422I
dewarp input: 3840x2160 YUV422I
dewarp output: 3840x2160 YUV422I
[ 304.751846] enter isp_mi_stop
[ 304.757250] enter imx378_set_fmt
[ 304.760529] enter imx378_write_reg_arry
[ 304.905955] enter imx378_get_format_code
[ 304.909943] enter imx378_priv_ioctl (cmd = 285)
[ 304.914513] enter imx378_set_test_pattern
[ 304.918557] enter imx378_write_reg
[ 304.922576] enter imx378_write_reg
[ 304.926560] enter imx378_write_reg
[ 304.930558] enter imx378_write_reg
[ 304.934583] enter imx378_priv_ioctl (cmd = 279)
[ 304.939124] enter imx378_set_fps
[ 304.942349] enter imx378_write_reg
[ 304.946360] enter imx378_write_reg
[ 304.950353] enter imx378_write_reg
[ 304.954342] enter imx378_write_reg
[ 304.958352] enter imx378_priv_ioctl (cmd = 263)
[ 304.962900] enter imx378_get_sensor_mode
camEngineConfig.type = 1[ 304.967415] enter imx378_priv_ioctl (cmd = 274)

camEngineConfig.pathConfig[0].w[ 304.973526] enter imx378_set_exp
idth = 3840
camEngineConfig.path[ 304.979670] enter imx378_write_reg
Config[0].height = 2160
camEngin[ 304.986491] enter imx378_write_reg
eConfig.pathConfig[0].mode = 4
c[ 304.992758] enter imx378_write_reg
amEngineConfig.pathConfig[0].layo[ 304.999031] enter imx378_write_reg
ut = 3
camEngineConfig.pathConfi[ 305.005318] enter imx378_priv_ioctl (cmd = 277)
g[0].alignMode = 0
camEngineConf[ 305.012126] enter imx378_set_gain
ig.pathConfig[1].width = 0
camEn[ 305.018323] enter imx378_write_reg
gineConfig.pathConfig[1].height =[ 305.025170] enter imx378_write_reg
0
camEngineConfig.pathConfig[1][ 305.031449] enter imx378_write_reg
.mode = 1
camEngineConfig.pathCo[ 305.037712] enter imx378_write_reg
nfig[1].layout = 3
camEngineConfig.pathConfig[1].alignMode = 0
camEngineConfig.pathConfig[2].width = 0
camEngineConfig.pathConfig[2].height = 0
camEngineConfig.pathConfig[2].mode = 1
camEngineConfig.pathConfig[2].layout = 3
camEngineConfig.pathConfig[2].alignMode = 0
camEngineConfig.pathConfig[3].width = 0
camEngineConfig.pathConfig[3].height = 0
camEngineConfig.pathConfig[3].mode = 1
camEngineConfig.pathConfig[3].layout = 3
camEngineConfig.pathConfig[3].alignMode = 65535
camEngineConfig.pathConfig[4].width = 0
camEngineConfig.pathConfig[4].height = 0
camEngineConfig.pathConfig[4].mode = 1
camEngineConfig.pathConfig[4].layout = 3
camEngineConfig.pathConfig[4].alignMode = 65535
camEngineConfig.data.sensor.hSensor = 281473366536816
camEngineConfig.data.sensor.hSubSensor = 0
camEngineConfig.data.sensor.hCamCalibDb = 281473366260096
camEngineConfig.data.sensor.ifSelect = 1
camEngineConfig.data.sensor.sampleEdge = 2
camEngineConfig.data.sensor.hSyncPol = 1
camEngineConfig.data.sensor.vSyncPol = 2
camEngineConfig.data.sensor.bayerPattern = 3
camEngineConfig.data.sensor.subSampling = 3
camEngineConfig.data.sensor.seqCCIR = 1
camEngineConfig.data.sensor.fieldSelection = 1
camEngineConfig.data.sensor.inputSelection = 2
camEngineConfig.data.sensor.dvpPinMapping = 0
camEngineConfig.data.sensor.mode = 4
camEngineConfig.data.sensor.acqWindow.hOffset = 0
camEngineConfig.data.sensor.acqWindow.vOffset = 0
camEngineConfig.data.sensor.acqWindow.width = 4048
camEngineConfig.data.sensor.acqWindow.height = 3040
camEngineConfig.data.sensor.outWindow.hOffset = 0
camEngineConfig.data.sensor.outWindow.vOffset = 0
camEngineConfig.data.sensor.outWindow.width = 4048
camEngineConfig.data.sensor.outWindow.height = 3040
camEngineConfig.data.sensor.isWindow.hOffset = 0
camEngineConfig.data.sensor.isWindow.vOffset = 0
camEngineConfig.data.sensor.isWindow.width = 4048
camEngineConfig.data.sensor.isWindow.height = 3040
camEngineConfig.data.sensor.mipiMode = 43
camEngineConfig.data.sensor.enable3D = 0
camEngineConfig.data.sensor.enableTestpattern = 0
camEngineConfig.data.sensor.flickerPeriod = 1
camEngineConfig.data.sensor.enableAfps = 0
camEngineConfig.data.sensor.szSensorName[256] =
camEngineConfig.data.sensor.szCsiName[256] =
camEngineConfig.data.sensor.csiFormat = 808534599
camEngineConfig.data.sensor.csiPad = 0
camEngineConfig.data.sensor.stitchingMode = 0
camEngineConfig.data.sensor.compress_curve.enable = 0
camEngineConfig.data.sensor.compress_curve.in_bit = 0
camEngineConfig.data.sensor.compress_curve.out_bit = 0
camEngineConfig.data.sensor.expand_curve.enable = 0
camEngineConfig.data.sensor.expand_curve.in_bit = 0
camEngineConfig.data.sensor.expand_curve.out_bit = 0
camEngineConfig.data.sensor.colorspace = 0
camEngineConfig.data.sensor.colorrange = 0
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2340: start() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:427: inputConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1359: cameraConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2371: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1084: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1639: previewStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2382: streamingStop() = 12(RET_WRONG_STATE)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1104: streamingStop() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/un[ 308.127902] enter imx378_priv_ioctl (cmd = 272)
its/cam_device/source/camera/cam_[ 308.133165] enter imx378_s_stream
operations.cpp:1652: previewStop([ 308.139233] enter imx378_write_reg
) = 12(ret)
[ERR] /usr/src/debu[ 308.146157] enter imx378_priv_ioctl (cmd = 260)
g/isp-imx/4.2.2.15.0-r0/isp-imx-4[ 308.152908] enter imx378_get_clk
.2.2.15.0/units/cam_device/source[ 308.158977] enter imx378_priv_ioctl (cmd = 259)
/camera/cam_engine_interface.cpp:[ 308.166368] enter imx378_priv_ioctl (cmd = 257)
757: cprocEnableSet() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:820: inputDisconnect() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1372: cameraDisconnect() = 12(ret)

As can be seen above, I am gettin an invalid parameter error (return of 13). So I modified the source to output the configuration and everything looks ok to me. I have no idea why it is failing at this step.

Do you have any idea what would cause the CamEngineStart (I cannot find source for this function) to return a parameter error?

Thank you again for the help,

-Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Thu, 09 Dec 2021 23:05:13 GMT

Hi @khang_letruong

Thank you very much for the suggestion, I got a lot further. I can see the camera get configured and looks like it should work, however during starting the ISP driver returns a parameter error as shown below.

root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx378
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx378"...
Killing preexisting instances of video_test and isp_media_server:
PID TTY STAT TIME COMMAND 1033 pts/0 S+ 0:00 ./video_test -w 4048 -h 3040 -f RAW10 -t file -m 0 -d 2
[ 296.878947] viv_post_event: unsubscribed event id =4 type=0x08002000
Starting isp_media_server with configuration file CAMERA0
[ 297.038965] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 304.427751] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.439580] enter imx378_query_capability
[ 304.483156] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.488406] enter imx378_query_capability
loading [imx378.drv]...
[ 304.510213] enter imx378_priv_ioctl (cmd = 261)
[ 304.516684] enter imx378_query_supports
WARN : [DEWARP_HW] dw[ 304.521098] enter imx378_priv_ioctl (cmd = -2140645888)
e type /dev/v4l-subdev0 0
[ 304.529218] enter imx378_query_capability
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev1 1
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2
loading [imx378.drv]...[ 304.561912] enter imx378_priv_ioctl (cmd = 261)

[ 304.568089] enter imx378_query_supports
loading [imx378.drv]...[ 304.572873] enter imx378_priv_ioctl (cmd = 261)

[ 304.579125] enter imx378_query_supports
loading [imx378.drv]...[ 304.584259] enter imx378_priv_ioctl (cmd = 261)

[ 304.590577] enter imx378_query_supports
[ 304.606186] enter imx378_priv_ioctl (cmd = 261)
[ 304.610754] enter imx378_query_supports
[ 304.614608] enter imx378_priv_ioctl (cmd = 257)
[ 304.619173] enter imx378_priv_ioctl (cmd = 260)
[ 304.623720] enter imx378_get_clk
[ 304.626963] enter imx378_priv_ioctl (cmd = 259)
[ 304.631530] enter imx378_priv_ioctl (cmd = 256)
[ 304.636073] enter imx378_priv_ioctl (cmd = 262)
[ 304.640612] enter imx378_set_sensor_mode
[ 304.644553] enter imx378_priv_ioctl (cmd = 263)
[ 304.649109] enter imx378_get_sensor_mode
[ 304.653054] enter imx378_priv_ioctl (cmd = 261)
[ 304.657593] enter imx378_query_supports
isp input: 4048x3040 RAW10
isp output: 3840x2160 YUV422I
dewarp input: 3840x2160 YUV422I
dewarp output: 3840x2160 YUV422I
[ 304.751846] enter isp_mi_stop
[ 304.757250] enter imx378_set_fmt
[ 304.760529] enter imx378_write_reg_arry
[ 304.905955] enter imx378_get_format_code
[ 304.909943] enter imx378_priv_ioctl (cmd = 285)
[ 304.914513] enter imx378_set_test_pattern
[ 304.918557] enter imx378_write_reg
[ 304.922576] enter imx378_write_reg
[ 304.926560] enter imx378_write_reg
[ 304.930558] enter imx378_write_reg
[ 304.934583] enter imx378_priv_ioctl (cmd = 279)
[ 304.939124] enter imx378_set_fps
[ 304.942349] enter imx378_write_reg
[ 304.946360] enter imx378_write_reg
[ 304.950353] enter imx378_write_reg
[ 304.954342] enter imx378_write_reg
[ 304.958352] enter imx378_priv_ioctl (cmd = 263)
[ 304.962900] enter imx378_get_sensor_mode
camEngineConfig.type = 1[ 304.967415] enter imx378_priv_ioctl (cmd = 274)

camEngineConfig.pathConfig[0].w[ 304.973526] enter imx378_set_exp
idth = 3840
camEngineConfig.path[ 304.979670] enter imx378_write_reg
Config[0].height = 2160
camEngin[ 304.986491] enter imx378_write_reg
eConfig.pathConfig[0].mode = 4
c[ 304.992758] enter imx378_write_reg
amEngineConfig.pathConfig[0].layo[ 304.999031] enter imx378_write_reg
ut = 3
camEngineConfig.pathConfi[ 305.005318] enter imx378_priv_ioctl (cmd = 277)
g[0].alignMode = 0
camEngineConf[ 305.012126] enter imx378_set_gain
ig.pathConfig[1].width = 0
camEn[ 305.018323] enter imx378_write_reg
gineConfig.pathConfig[1].height =[ 305.025170] enter imx378_write_reg
0
camEngineConfig.pathConfig[1][ 305.031449] enter imx378_write_reg
.mode = 1
camEngineConfig.pathCo[ 305.037712] enter imx378_write_reg
nfig[1].layout = 3
camEngineConfig.pathConfig[1].alignMode = 0
camEngineConfig.pathConfig[2].width = 0
camEngineConfig.pathConfig[2].height = 0
camEngineConfig.pathConfig[2].mode = 1
camEngineConfig.pathConfig[2].layout = 3
camEngineConfig.pathConfig[2].alignMode = 0
camEngineConfig.pathConfig[3].width = 0
camEngineConfig.pathConfig[3].height = 0
camEngineConfig.pathConfig[3].mode = 1
camEngineConfig.pathConfig[3].layout = 3
camEngineConfig.pathConfig[3].alignMode = 65535
camEngineConfig.pathConfig[4].width = 0
camEngineConfig.pathConfig[4].height = 0
camEngineConfig.pathConfig[4].mode = 1
camEngineConfig.pathConfig[4].layout = 3
camEngineConfig.pathConfig[4].alignMode = 65535
camEngineConfig.data.sensor.hSensor = 281473366536816
camEngineConfig.data.sensor.hSubSensor = 0
camEngineConfig.data.sensor.hCamCalibDb = 281473366260096
camEngineConfig.data.sensor.ifSelect = 1
camEngineConfig.data.sensor.sampleEdge = 2
camEngineConfig.data.sensor.hSyncPol = 1
camEngineConfig.data.sensor.vSyncPol = 2
camEngineConfig.data.sensor.bayerPattern = 3
camEngineConfig.data.sensor.subSampling = 3
camEngineConfig.data.sensor.seqCCIR = 1
camEngineConfig.data.sensor.fieldSelection = 1
camEngineConfig.data.sensor.inputSelection = 2
camEngineConfig.data.sensor.dvpPinMapping = 0
camEngineConfig.data.sensor.mode = 4
camEngineConfig.data.sensor.acqWindow.hOffset = 0
camEngineConfig.data.sensor.acqWindow.vOffset = 0
camEngineConfig.data.sensor.acqWindow.width = 4048
camEngineConfig.data.sensor.acqWindow.height = 3040
camEngineConfig.data.sensor.outWindow.hOffset = 0
camEngineConfig.data.sensor.outWindow.vOffset = 0
camEngineConfig.data.sensor.outWindow.width = 4048
camEngineConfig.data.sensor.outWindow.height = 3040
camEngineConfig.data.sensor.isWindow.hOffset = 0
camEngineConfig.data.sensor.isWindow.vOffset = 0
camEngineConfig.data.sensor.isWindow.width = 4048
camEngineConfig.data.sensor.isWindow.height = 3040
camEngineConfig.data.sensor.mipiMode = 43
camEngineConfig.data.sensor.enable3D = 0
camEngineConfig.data.sensor.enableTestpattern = 0
camEngineConfig.data.sensor.flickerPeriod = 1
camEngineConfig.data.sensor.enableAfps = 0
camEngineConfig.data.sensor.szSensorName[256] =
camEngineConfig.data.sensor.szCsiName[256] =
camEngineConfig.data.sensor.csiFormat = 808534599
camEngineConfig.data.sensor.csiPad = 0
camEngineConfig.data.sensor.stitchingMode = 0
camEngineConfig.data.sensor.compress_curve.enable = 0
camEngineConfig.data.sensor.compress_curve.in_bit = 0
camEngineConfig.data.sensor.compress_curve.out_bit = 0
camEngineConfig.data.sensor.expand_curve.enable = 0
camEngineConfig.data.sensor.expand_curve.in_bit = 0
camEngineConfig.data.sensor.expand_curve.out_bit = 0
camEngineConfig.data.sensor.colorspace = 0
camEngineConfig.data.sensor.colorrange = 0
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2340: start() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:427: inputConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1359: cameraConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2371: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1084: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1639: previewStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2382: streamingStop() = 12(RET_WRONG_STATE)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1104: streamingStop() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/un[ 308.127902] enter imx378_priv_ioctl (cmd = 272)
its/cam_device/source/camera/cam_[ 308.133165] enter imx378_s_stream
operations.cpp:1652: previewStop([ 308.139233] enter imx378_write_reg
) = 12(ret)
[ERR] /usr/src/debu[ 308.146157] enter imx378_priv_ioctl (cmd = 260)
g/isp-imx/4.2.2.15.0-r0/isp-imx-4[ 308.152908] enter imx378_get_clk
.2.2.15.0/units/cam_device/source[ 308.158977] enter imx378_priv_ioctl (cmd = 259)
/camera/cam_engine_interface.cpp:[ 308.166368] enter imx378_priv_ioctl (cmd = 257)
757: cprocEnableSet() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:820: inputDisconnect() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1372: cameraDisconnect() = 12(ret)

I modified the wrapper function to dump the config structure and I do not see anything that stands out that would cause a parameter error.

Any ideas?

Thank you,

Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Fri, 10 Dec 2021 20:23:09 GMT

Hi @khang_letruong

Thank you very much for the help and suggestions.

I just needed to modify the run.sh for the imx378 and add the step to copy the files over in the recipie.

I did run into an issue where the CamEngineStart() call was returning RET_INVALID_PARM. As it turns out there was something wrong with the IMX378.xml isp configuration - I copied the basler config and everything appears to work now.

If NXP people see this, it would be very helpful to have a step in the porting guide to modify the run.sh and add a copy of the .drv and .xml files.

Again, Thank you,

-Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Sat, 11 Dec 2021 02:42:50 GMT

Hi @chrispeterson ,

Might it be an issue of HW (or wrong/incidental register's setting) during the streaming?

BR,
K.

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Mon, 13 Dec 2021 16:12:33 GMT

Hi @khang_letruong,

Something weird is going on. The latest message from me was actually sent a day before the solution acceptance. When I posted the message it just disappeared. In the mean time I figured out the issue was my IMX378.xml file. (I changed to use the DAA3840_30MC_4K.xml file and then everything started working).

Thank you very much for the help.

-Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Fri, 25 Mar 2022 12:36:37 GMT

Hi @all,

I'm sorry for digging out this old thread, but I'm facing similar issues as @chrispeterson did and hopefully somebody of you can help me.

Currently I'm trying to connect a Sony IMX327 image sensor via MIPI-CSI2 to the imx8mp. Since the sensor has no ISP capabilities I want to use the internal ISP of the imx8mp. It seems like there is a problem with the data exchange between the MIPI-CSI2 core and the ISP.
What I did so far:

- Implemented a vvcam kernel driver for the IMX327 based on the nxp example for OV2775. In principle the same that @chrispeterson did.

- Changed the device-tree as follows:

imx327_0: imx327@1a { compatible = "sony,imx327"; reg = <0x1A>; csi_id = <0x00>; status = "okay"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_powergood &pinctrl_imager_io>; pg-3v3-gpio = <&gpio3 19 GPIO_ACTIVE_HIGH>; pg-1v2-gpio = <&gpio3 21 GPIO_ACTIVE_HIGH>; pg-1v8-gpio = <&gpio3 23 GPIO_ACTIVE_HIGH>; pg-2v8-gpio = <&gpio3 25 GPIO_ACTIVE_HIGH>; en-1v2-gpio = <&gpio3 20 GPIO_ACTIVE_HIGH>; en-1v8-gpio = <&gpio3 22 GPIO_ACTIVE_HIGH>; en-2v8-gpio = <&gpio3 24 GPIO_ACTIVE_HIGH>; sen-clk-en-gpio = <&gpio4 22 GPIO_ACTIVE_HIGH>; sen-rst-n-gpio = <&gpio4 25 GPIO_ACTIVE_HIGH>; port { imx327_ep_0: endpoint { data-lanes = <1 2 3 4>; clock-lanes = <0>; max-pixel-frequency = /bits/ 64 <266000000>; remote-endpoint = <&mipi_csi0_ep>; }; }; }; &mipi_csi_0 { status = "okay"; port@0 { reg = <0>; mipi_csi0_ep: endpoint { remote-endpoint = <&imx327_ep_0>; data-lanes = <4>; csis-hs-settle = <2>; // 4 csis-clk-settle = <0>; csis-wclk; }; }; }; &cameradev { status = "okay"; }; &isi_0 { status = "disabled"; }; &isp_0 { status = "okay"; }; &dewarp { status = "okay"; };

- Added imx327.drv based on OV2775 and imx327.xml based on DAA3840_30MC_4K.xml and changed resolution to 1920x1080. (Also the same as in the examples above)

- Changed run.sh script to support my drivers

After all these changes I can start the isp_media_server with start_isp.sh I see that my driver is loaded and the v4l-devices are generated properly. But I'm confused why there is no linking between the mipi and the isp. Can anybody confirm that this is correct?

RUN_SCRIPT=/opt/imx8-isp/bin/run.sh RUN_SCRIPT_PATH=/opt/imx8-isp/bin Trying configuration "imx327_1080p30"... Removing vvcam-isp... Removing vvcam-dwe... Removing vvcam-video... [ 806.113804] enter viv_video_exit_module Removing imx8-media-dev... [ 806.300133] : Unregistered all entities Removing ov2775... Removing os08a20... Removing basler-camera-driver-vvcam... Loading module imx327 ... imx327 imx327 20480 0 imx327 already loaded. Loading module imx8-media-dev ... imx8_media_dev [ 806.722106] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned. [ 806.738391] mxc_md_probe [ 806.741860] mx8-img-md: Registered sensor subdevice: imx327 1-001a (1) [ 806.748716] mx8-img-md: created link [imx327 1-001a] => [mxc-mipi-csi2.0] [ 806.755721] mxc-md 32c00000.bus:camera: mxc_md_create_links Loaded /lib/modules/5.10.52+g0fe40616142b/kernel/drivers/staging/media/imx/imx8-media-dev.ko Loading module vvcam-video ... [ 806.799057] enter imx327_priv_ioctl (cmd = -2140645888) [ 806.805286] enter imx327_query_capability vvcam_video [ 806.987356] enter viv_video_init_module Loaded /lib/modules/5.10.52+g0fe40616142b/extra/video/vvcam-video.ko Loading module vvcam-dwe ... [ 807.085842] enter isp_mi_stop vvcam_dwe vvcam_dwe 28672 1 vvcam-dwe already loaded. Loading module vvcam-isp ... vvcam_isp vvcam_isp 65536 1 vvcam-isp already loaded. Starting isp_media_server with configuration file CAMERA0

Then I tried to start the video with gstreamer and the provided program video_test, but I can't get any video data. (See the log below)

./video_test -w 1920 -h 1080 -f RAW12 -t 2 -m0 -d0 loading [imx327.drv]... Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) WARN : [DEWARP_HW] dwe type /dev/v4l-subdev0 0 WARN : [DEWARP_HW] dwe type /dev/v4l-subdev1 1 WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2 loading [imx327.drv]... Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) Video Test:caps supports:{ count = 1 { index = 0 width = 1920 height = 1080 fps = 30 hdr_mode = 0 } } loading [imx327.drv]... Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) loading [imx327.drv]... Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) Info - IMX327_IsiGetSensorIss (enter) Info - IMX327_IsiGetSensorIss (exit) Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) Info - IMX327_IsiCreateSensorIss (enter) Info - IMX327_IsiSensorSetPowerIss: (enter) Info - IMX327_IsiSensorSetPowerIss: set power 1 Info - IMX327_IsiSensorSetPowerIss: (exit) Info - IMX327_IsiSensorGetClkIss: (enter) Info - IMX327_IsiSensorGetClkIss: status:276111060 sensor_mclk:37125000 csi_max_pixel_clk:266000000 Info - IMX327_IsiSensorGetClkIss: (exit) Info - IMX327_IsiSensorSetClkIss: (enter) Info - IMX327_IsiSensorSetClkIss: status:1 sensor_mclk:37125000 csi_max_pixel_clk:266000000 Info - IMX327_IsiSensorSetClkIss: (exit) Info - IMX327_IsiResetSensorIss: (enter) Info - IMX327_IsiResetSensorIss: (exit) Info - IMX327_IsiSetSensorModeIss (enter) Info - IMX327_IsiSetSensorModeIss (exit) Info - IMX327_IsiCreateSensorIss (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetCapsIss (enter) Info - IMX327_IsiQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (enter) Info - IMX327_IsiHalQuerySensorIss (exit) Info - IMX327_IsiQuerySensorIss (exit) Info - IMX327_IsiGetCapsIss (exit) isp input: 1920x1080 RAW12 isp output: 1920x1080 YUV422I dewarp input: 1920x1080 YUV422I dewarp output: 1920x1080 YUV422I Info - IMX327_IsiSetupSensorIss (enter) Info - IMX327_IsiSetupSensorIss (exit) Info - IMX327_IsiSetTestPatternIss (enter) Info - IMX327_IsiSetTestPatternIss: test pattern enable[0] mode[0] Info - IMX327_IsiSetTestPatternIss: (exit) Info - IMX327_IsiSetSensorFpsIss: (enter) Info - IMX327_IsiSetSensorFpsIss: (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetAeInfoIss (enter) Info - IMX327_IsiGetAeInfoIss (exit) Info - IMX327_IsiGetSensorFpsIss: (enter) Info - IMX327_IsiGetSensorFpsIss: (exit) Info - IMX327_IsiGetAeStartExposureIs (enter) Info - IMX327_IsiGetAeStartExposureIs:get start exposure 36403200 Info - IMX327_IsiGetAeStartExposureIs: (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetAeInfoIss (enter) Info - IMX327_IsiGetAeInfoIss (exit) Info - IMX327_IsiSetIntegrationTimeIss (enter) Info - IMX327_IsiSetIntegrationTimeIss set linear exp 400 Info - IMX327_IsiSetIntegrationTimeIss (exit) Info - IMX327_IsiSetGainIss (enter) Info - IMX327_IsiSetGainIss set linear gain 3072 Info - IMX327_IsiSetGainIss (exit) Info - IMX327_IsiGetIntegrationTimeIss (enter) Info - IMX327_IsiGetIntegrationTimeIss (exit) Info - IMX327_IsiGetGainIss (enter) Info - IMX327_IsiGetGainIss (exit) #### CsiSetFormat 1920 1080 842090322 Info - IMX327_IsiGetSensorIspStatusIss: (enter) Info - IMX327_IsiGetSensorIspStatusIss: (exit) Info - IMX327_IsiGetSensorIspStatusIss: (enter) Info - IMX327_IsiGetSensorIspStatusIss: (exit) Info - IMX327_IsiGetSensorIspStatusIss: (enter) Info - IMX327_IsiGetSensorIspStatusIss: (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetSensorModeIss (enter) Info - IMX327_IsiGetSensorModeIss (exit) Info - IMX327_IsiGetSensorIspStatusIss: (enter) Info - IMX327_IsiGetSensorIspStatusIss: (exit) CamEngineEnableCproc: set val 1 1 1 1.10 -15 1.00 0.00 Info - IMX327_IsiSensorSetStreamingIss (enter) Info - IMX327_IsiSensorSetStreamingIss: set streaming 1 Info - IMX327_IsiSensorSetStreamingIss (exit)

I checked that the sensor is configured properly by measuring the traffic on all MIPI data/clock lanes. I read out the interrupt source register of the MIPI core and can see that the data is coming in until a FIFO overflow occures.

For testing purposes I then activated the ISI. With this configuration there are no FIFO overflows and I can see the MIPI Frame Counter running up.

Due to that I assume that there must be a problem with the MIPI-CSI <-> ISP data exchange.

Has anybody an idea what's going wrong here? Maybe somebody can post the logs of a successful try.
Any help is appreciated.

Thanks in advance!

Best regards,

Daniel

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Mon, 28 Mar 2022 11:16:31 GMT

In the meantime I had a look in the video_test application and added some debug prints. It seems like the "pipeline.warm.up" statement is failing.

INFO : [VIDEOTEST] VIDIOC_S_EXT_CTRLS: ID: 0x98f901 Size: 65536 {<id>:<pipeline.warm.up>} INFO : [V4l2Event] get json request: {"id":"pipeline.warm.up"} INFO : [VIDEOTEST] VIDIOC_S_EXT_CTRLS: { "result" : -1 } INFO : [VIDEOTEST] VIDIOC_G_EXT_CTRLS: { "result" : -1 }

Does somebody know why this can happen?

Best regards,

Daniel

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Mon, 28 Mar 2022 14:30:48 GMT

Hi @dh84 ,

You might try other values for following properties in dts, for example :

csis-hs-settle = <13>; // 4 csis-clk-settle = <2>;

Best Regards,

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

chrispeterson — Mon, 28 Mar 2022 14:44:56 GMT

@dh84

Have you tried enabling the isi?

&isi_0 { status = "okay"; };

-Chris

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Tue, 29 Mar 2022 05:25:20 GMT

Hi @chrispeterson,

thanks for your reply. Yes I've tried to activate the ISI.
If I try that and capture the image via the ISI, I can the see MIPI Frame counter is running up. So the data is taken from is mipi-csi and no overflows occure.
That's the reason I assume, that something is wrong with the connection of MIPI-CSI to the ISP.

Did you have to add a linking between the cores manually through media-ctl?

Best regards,

Daniel

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Tue, 29 Mar 2022 05:27:59 GMT

Hi @khang_letruong ,

If I change the values, "MIPI Frame Start Errors" occure. So I think my values are ok. My Mipi Clock is 112 MHz with a datarate of 224 MBit/s per lane.

Best regards,

Daniel

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Thu, 31 Mar 2022 13:45:37 GMT

Hi @chrispeterson and @khang_letruong.,

I've made small progress. If I enable the parallel mode in the mipi csi driver (imx8-mipi-csi2-sam.c) as follows, there are no MIPI Overflows anymore.

}, { .code = MEDIA_BUS_FMT_SRGGB12_1X12, .fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12, // .data_alignment = 16, .data_alignment = 32, }, };

I can see some image data behind the ISP with ~1 Frame / per minute and strange content.
@chrispeterson I think you've also used RAW12 pixelformat. Did you have to change anything in the csi driver?

Many thanks,

Daniel

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Fri, 01 Apr 2022 03:56:37 GMT

Hi @dh84 ,

ISP examples such as ov2775 and basler cameras are also 12-bit RAW but do not require the modification as you did in imx8-mipi-csi2-sam.c.

Regards,
K

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

khang_letruong — Fri, 01 Apr 2022 04:09:20 GMT

Btw, IMX327 supports both RAW10 and RAW12 outputs. So make sure that you have configured it to output RAW12.

RG,

Re: Creating a MIPI CSI Camera driver on i.MX 8MP EVK

dh84 — Tue, 05 Apr 2022 15:37:29 GMT

Hi @all,

just for your information. I the meantime I got my sensor running. Indeed it was an issue of the image sensor configuration. It was outputting a few more pixels than expected. That reached to hang up the csi receiver.

Thanks again for your help.

Best regards,

Daniel