Creating a MIPI CSI Camera driver on i.MX 8MP EVK

[chrispeterson]

Hello all,

I am working on porting an IMX378 camera driver to the i.MX 8MP EVK board.  I am following the i.MX 8M Plus Camera and Display Guid, section 2 Camera Sensor Porting Guide.

I have followed section 2.7 Camera Sensor Driver in V4L2 mode and I get the following dmesg output:

6.440486] imx378: loading out-of-tree module taints kernel.
[ 6.449105] enter imx378_probe
[ 6.458998] enter imx378_retrieve_capture_properties
[ 6.466931] imx378 1-001a: supply DOVDD not found, using dummy regulator
[ 6.474198] imx378 1-001a: supply DVDD not found, using dummy regulator
[ 6.481092] imx378 1-001a: supply AVDD not found, using dummy regulator
[ 6.489336] enter imx378_regulator_enable
[ 6.493550] enter imx378_set_clk_rate
[ 6.497365] enter imx378_power_on
[ 6.500962] enter imx378_reset
[ 6.539481] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.558768] enter imx378_read_reg
[ 6.562991] enter imx378_read_reg
[ 6.590179] imx378_probe camera mipi imx378, is found
[ 6.648388] mx8-img-md: Registered sensor subdevice: imx378 1-001a (1)
[ 6.740810] enter imx378_link_setup
[ 6.744958] mx8-img-md: created link [imx378 1-001a] => [mxc-mipi-csi2.0]
[ 6.751831] mxc-md 32c00000.bus:camera: mxc_md_create_links
[ 6.768425] enter viv_dwe_init_module
[ 6.772649] enter dwe_hw_probe
[ 6.778619] enter viv_isp_init_module
[ 6.783472] vvcam dewa74] vvcam isp driver registered
[ 7.206408] enter imx378_priv_ioctl (cmd = -2140645888)
[ 7.211671] enter imx378_query_capability

 My media-ctl output is as follows:

root@imx8mpevk:~# media-ctl -p
[ 2002.167891] enter imx378_get_fmt
Media controller API version 5.10.52

Media device information
------------------------
driver mxc-md
model FSL Capture Media Device
serial 
bus info 
hw revision 0x0
driver version 5.10.52

Device topology
- entity 1: mxc-mipi-csi2.0 (8 pads, 1 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev0
pad0: Sink
<- "imx378 1-001a":0 [ENABLED,IMMUTABLE]
pad1: Sink
pad2: Sink
pad3: Sink
pad4: Source
pad5: Source
pad6: Source
pad7: Source

- entity 10: imx378 1-001a (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev1
pad0: Source
[fmt:SGBRG10_1X10/4056x3040 field:none]
-> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

root@imx8mpevk:~#

 At this point I can see that the I2C communication is working, however I cannot for the life of me figure out how to tell V4L2 to start the stream so I can view it.   When I list V4L2 devices I do not see a video source:

root@imx8mpevk:~# v4l2-ctl --list-devices
[ 2130.712823] enter imx378_priv_ioctl (cmd = -2140645888)
[ 2130.718106] enter imx378_query_capability
():
/dev/v4l-subdev0

(csi0):
/dev/v4l-subdev1

FSL Capture Media Device (platform:mxc-md):
/dev/media0

vsi_v4l2dec (platform:vsi_v4l2dec):
/dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
/dev/video0

 What am I missing?

My DTS file is as below:

// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
 * Copyright 2020 NXP
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 */

/dts-v1/;

#include "imx8mp-evk.dts"

&i2c2 {
	/delete-node/ov5640_mipi@3c;

	imx378_0: imx378_mipi@1a {
		compatible = "sony,imx378";
		reg = <0x1a>;
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_csi0_pwn>, <&pinctrl_csi0_rst>, <&pinctrl_csi_mclk>;
		clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
		clock-names = "csi_mclk";
		assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
		assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
		assigned-clock-rates = <24000000>;
		csi_id = <0>;
		pwn-gpios = <&gpio2 11 GPIO_ACTIVE_HIGH>;
		rst-gpios = <&gpio1 6 GPIO_ACTIVE_LOW>;
		mclk = <24000000>;
		mclk_source = <0>;
		status = "okay";

		port {
			imx378_mipi_0_ep: endpoint {
				data-lanes = <1 2 3 4>;
				clock-lanes = <0>;
				max-pixel-frequency = /bits/ 64 <500000000>;
				remote-endpoint = <&mipi_csi0_ep>;
			};
		};

	};
};

&i2c3 {
	clock-frequency = <100000>;
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_i2c3>;
	status = "okay";

	ov5640_1: ov5640_mipi@3c {
		status = "disabled";
	};
};

&cameradev {
	status = "okay";
};

&isi_0 {
	status = "disabled";
};

&isi_1 {
	status = "disabled";
};

&isp_0 {
	status = "okay";
};

&dewarp {
	status = "okay";
};

&mipi_csi_0 {
	status = "okay";
	clock-frequency = <500000000>;
	assigned-clock-rates = <500000000>;

	port@0 {
		endpoint {
			remote-endpoint = <&imx378_mipi_0_ep>;
			data-lanes = <4>;
			csis-hs-settle = <16>;			
		};
	};
};

&mipi_csi_1 {
	status = "disabled";
};

ISI driver, and VVCAM driver Attached

Thank you,

Chris

[malik_cisse]

Hi Chris,
Are you still working on the imx378 imx8mp ISP integration?

I managed to make 4K@34FPS work with your code and now I would like to update to 12MP.
Let me know if you are interested in sharing some insights.
BR, M.Cisse

[Amal_Antony3331]

Hi @chrispeterson 

I'm trying to integrate the IMX219 on the i.MX8MP. Now I'm able to get the stream and the issue with stream is FPS that I'm getting is 15 for 480p. When tried with the 144p, it increases to 20FPS. But setting 30FPS on the driver code.

Also there is a fish eye issue on the stream. Is it due to the xml file configuration?

VVCAM driver is attached for the reference. can you please guide on this.

Any help is appreciated.

Regards

Charles

[dh84]

Hi @all,

I'm sorry for digging out this old thread, but I'm facing similar issues as @chrispeterson did and hopefully somebody of you can help me. 

Currently I'm trying to connect a Sony IMX327 image sensor via MIPI-CSI2 to the imx8mp. Since the sensor has no ISP capabilities I want to use the internal ISP of the imx8mp. It seems like there is a problem with the data exchange between the MIPI-CSI2 core and the ISP.
What I did so far:

- Implemented a vvcam kernel driver for the IMX327 based on the nxp example for OV2775. In principle the same that @chrispeterson did.

- Changed the device-tree as follows:

imx327_0: imx327@1a {
	compatible = "sony,imx327";
	reg = <0x1A>;
	csi_id = <0x00>;
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_powergood &pinctrl_imager_io>;
	pg-3v3-gpio = <&gpio3 19 GPIO_ACTIVE_HIGH>;
	pg-1v2-gpio = <&gpio3 21 GPIO_ACTIVE_HIGH>;
	pg-1v8-gpio = <&gpio3 23 GPIO_ACTIVE_HIGH>;
	pg-2v8-gpio = <&gpio3 25 GPIO_ACTIVE_HIGH>;		
	en-1v2-gpio = <&gpio3 20 GPIO_ACTIVE_HIGH>;
	en-1v8-gpio = <&gpio3 22 GPIO_ACTIVE_HIGH>;
	en-2v8-gpio = <&gpio3 24 GPIO_ACTIVE_HIGH>;
	sen-clk-en-gpio = <&gpio4 22 GPIO_ACTIVE_HIGH>;
	sen-rst-n-gpio  = <&gpio4 25 GPIO_ACTIVE_HIGH>;
	port {
		imx327_ep_0: endpoint {
			data-lanes = <1 2 3 4>;
	 		clock-lanes = <0>;
			max-pixel-frequency = /bits/ 64 <266000000>;
	 		remote-endpoint = <&mipi_csi0_ep>;
		};
	};
};

&mipi_csi_0 {
	status = "okay";

	port@0 {
		reg = <0>;
		mipi_csi0_ep: endpoint {
			remote-endpoint = <&imx327_ep_0>;
			data-lanes = <4>;
			csis-hs-settle = <2>;  // 4
			csis-clk-settle = <0>;
			csis-wclk;
		};
	};
};

&cameradev {
	status = "okay";
};

&isi_0 {
	status = "disabled";
};

&isp_0 {
	status = "okay";
};

&dewarp {
	status = "okay";
};

- Added imx327.drv based on OV2775 and imx327.xml based on DAA3840_30MC_4K.xml and changed resolution to 1920x1080. (Also the same as in the examples above)

- Changed run.sh script to support my drivers

After all these changes I can start the isp_media_server with start_isp.sh I see that my driver is loaded and the v4l-devices are generated properly. But I'm confused why there is no linking between the mipi and the isp. Can anybody confirm that this is correct?

RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx327_1080p30"...
Removing vvcam-isp...
Removing vvcam-dwe...
Removing vvcam-video...
[  806.113804] enter viv_video_exit_module
Removing imx8-media-dev...
[  806.300133] : Unregistered all entities
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module imx327 ...
imx327
imx327                 20480  0
imx327 already loaded.
Loading module imx8-media-dev ...
imx8_media_dev
[  806.722106] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[  806.738391] mxc_md_probe
[  806.741860] mx8-img-md: Registered sensor subdevice: imx327 1-001a (1)
[  806.748716] mx8-img-md: created link [imx327 1-001a] => [mxc-mipi-csi2.0]
[  806.755721] mxc-md 32c00000.bus:camera: mxc_md_create_links
Loaded /lib/modules/5.10.52+g0fe40616142b/kernel/drivers/staging/media/imx/imx8-media-dev.ko 
Loading module vvcam-video ...
[  806.799057] enter imx327_priv_ioctl (cmd = -2140645888)
[  806.805286] enter imx327_query_capability
vvcam_video
[  806.987356] enter viv_video_init_module
Loaded /lib/modules/5.10.52+g0fe40616142b/extra/video/vvcam-video.ko 
Loading module vvcam-dwe ...
[  807.085842] enter isp_mi_stop
vvcam_dwe
vvcam_dwe              28672  1
vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
vvcam_isp              65536  1
vvcam-isp already loaded.
Starting isp_media_server with configuration file CAMERA0

Then I tried to start the video with gstreamer and the provided program video_test, but I can't get any video data. (See the log below)

./video_test -w 1920 -h 1080 -f RAW12 -t 2 -m0 -d0
loading [imx327.drv]...
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev0 0 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev1 1 
WARN   : [DEWARP_HW] dwe type /dev/v4l-subdev2 2 
loading [imx327.drv]...
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
Video Test:caps supports:{
	count = 1
	{
	index    = 0
	width    = 1920
	height   = 1080
	fps      = 30
	hdr_mode = 0
	}
}
loading [imx327.drv]...
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
loading [imx327.drv]...
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
Info - IMX327_IsiGetSensorIss (enter)
Info - IMX327_IsiGetSensorIss (exit)
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
Info - IMX327_IsiCreateSensorIss (enter)
Info - IMX327_IsiSensorSetPowerIss: (enter)
Info - IMX327_IsiSensorSetPowerIss: set power 1
Info - IMX327_IsiSensorSetPowerIss: (exit)
Info - IMX327_IsiSensorGetClkIss: (enter)
Info - IMX327_IsiSensorGetClkIss: status:276111060 sensor_mclk:37125000 csi_max_pixel_clk:266000000
Info - IMX327_IsiSensorGetClkIss: (exit)
Info - IMX327_IsiSensorSetClkIss: (enter)
Info - IMX327_IsiSensorSetClkIss: status:1 sensor_mclk:37125000 csi_max_pixel_clk:266000000
Info - IMX327_IsiSensorSetClkIss: (exit)
Info - IMX327_IsiResetSensorIss: (enter)
Info - IMX327_IsiResetSensorIss: (exit)
Info - IMX327_IsiSetSensorModeIss (enter)
Info - IMX327_IsiSetSensorModeIss (exit) 
Info - IMX327_IsiCreateSensorIss (exit)
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetCapsIss (enter) 
Info - IMX327_IsiQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (enter) 
Info - IMX327_IsiHalQuerySensorIss (exit)
Info - IMX327_IsiQuerySensorIss (exit)
Info - IMX327_IsiGetCapsIss (exit)
isp  input: 1920x1080 RAW12
isp output: 1920x1080 YUV422I
dewarp   input: 1920x1080 YUV422I
dewarp  output: 1920x1080 YUV422I
Info - IMX327_IsiSetupSensorIss (enter)
Info - IMX327_IsiSetupSensorIss (exit)
Info - IMX327_IsiSetTestPatternIss (enter)
Info - IMX327_IsiSetTestPatternIss: test pattern enable[0] mode[0]
Info - IMX327_IsiSetTestPatternIss: (exit)
Info - IMX327_IsiSetSensorFpsIss: (enter)
Info - IMX327_IsiSetSensorFpsIss: (exit)
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetAeInfoIss (enter)
Info - IMX327_IsiGetAeInfoIss (exit)
Info - IMX327_IsiGetSensorFpsIss: (enter)
Info - IMX327_IsiGetSensorFpsIss: (exit)
Info - IMX327_IsiGetAeStartExposureIs (enter)
Info - IMX327_IsiGetAeStartExposureIs:get start exposure 36403200
Info - IMX327_IsiGetAeStartExposureIs: (exit)
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetAeInfoIss (enter)
Info - IMX327_IsiGetAeInfoIss (exit)
Info - IMX327_IsiSetIntegrationTimeIss (enter)
Info - IMX327_IsiSetIntegrationTimeIss set linear exp 400 
Info - IMX327_IsiSetIntegrationTimeIss (exit)
Info - IMX327_IsiSetGainIss (enter)
Info - IMX327_IsiSetGainIss set linear gain 3072
Info - IMX327_IsiSetGainIss (exit)
Info - IMX327_IsiGetIntegrationTimeIss (enter)
Info - IMX327_IsiGetIntegrationTimeIss (exit)
Info - IMX327_IsiGetGainIss (enter)
Info - IMX327_IsiGetGainIss (exit)
#### CsiSetFormat 1920 1080 842090322
Info - IMX327_IsiGetSensorIspStatusIss: (enter)
Info - IMX327_IsiGetSensorIspStatusIss: (exit)
Info - IMX327_IsiGetSensorIspStatusIss: (enter)
Info - IMX327_IsiGetSensorIspStatusIss: (exit)
Info - IMX327_IsiGetSensorIspStatusIss: (enter)
Info - IMX327_IsiGetSensorIspStatusIss: (exit)
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetSensorModeIss (enter)
Info - IMX327_IsiGetSensorModeIss (exit) 
Info - IMX327_IsiGetSensorIspStatusIss: (enter)
Info - IMX327_IsiGetSensorIspStatusIss: (exit)
CamEngineEnableCproc: set val  1 1 1 1.10 -15 1.00 0.00
Info - IMX327_IsiSensorSetStreamingIss (enter)
Info - IMX327_IsiSensorSetStreamingIss: set streaming 1
Info - IMX327_IsiSensorSetStreamingIss (exit) 

I checked that the sensor is configured properly by measuring the traffic on all MIPI data/clock lanes. I read out the interrupt source register of the MIPI core and can see that the data is coming in until a FIFO overflow occures. 

For testing purposes I then activated the ISI. With this configuration there are no FIFO overflows and I can see the MIPI Frame Counter running up. 

Due to that I assume that there must be a problem with the MIPI-CSI <-> ISP data exchange. 

Has anybody an idea what's going wrong here? Maybe somebody can post the logs of a successful try.
Any help is appreciated.

Thanks in advance!

Best regards,

Daniel

[dh84]

Hi @all,

just for your information. I the meantime I got my sensor running. Indeed it was an issue of the image sensor configuration. It was outputting a few more pixels than expected. That reached to hang up the csi receiver.

Thanks again for your help.

Best regards,

Daniel

[dh84]

Hi @chrispeterson and @khang_letruong.,

I've made small progress. If I enable the parallel mode in the mipi csi driver (imx8-mipi-csi2-sam.c) as follows, there are no MIPI Overflows anymore. 

	}, {
		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
		.fmt_reg = MIPI_CSIS_ISPCFG_FMT_RAW12,
//		.data_alignment = 16,
		.data_alignment = 32,		
	},
};

I can see some image data behind the ISP with ~1 Frame / per minute and strange content. 
@chrispeterson I think you've also used RAW12 pixelformat. Did you have to change anything in the csi driver?

Many thanks,

Daniel

[khang_letruong]

Hi @dh84 ,

ISP examples such as ov2775 and basler cameras are also 12-bit RAW but do not require the modification as you did in imx8-mipi-csi2-sam.c.

Regards,
K

[khang_letruong]

Btw, IMX327 supports both RAW10 and RAW12 outputs. So make sure that you have configured it to output RAW12.

RG,

[dh84]

In the meantime I had a look in the video_test application and added some debug prints. It seems like the "pipeline.warm.up" statement is failing.

INFO   : [VIDEOTEST] VIDIOC_S_EXT_CTRLS: ID: 0x98f901 Size: 65536 {<id>:<pipeline.warm.up>}
INFO   : [V4l2Event] get json request: {"id":"pipeline.warm.up"}
INFO   : [VIDEOTEST] VIDIOC_S_EXT_CTRLS: {
	"result" : -1
}
INFO   : [VIDEOTEST] VIDIOC_G_EXT_CTRLS: {
	"result" : -1
}

Does somebody know why this can happen?

Best regards,

Daniel

[chrispeterson]

@dh84 

Have you tried enabling the isi?

&isi_0 {
	status = "okay";
};

 -Chris

[dh84]

Hi @chrispeterson,

thanks for your reply. Yes I've tried to activate the ISI.
If I try that and capture the image via the ISI, I can the see MIPI Frame counter is running up. So the data is taken from is mipi-csi and no overflows occure. 
That's the reason I assume, that something is wrong with the connection of MIPI-CSI to the ISP.

Did you have to add a linking between the cores manually through media-ctl?

Best regards,

Daniel

[khang_letruong]

Hi @dh84 ,

You might try other values for following properties in dts, for example :

		csis-hs-settle = <13>;  // 4
		csis-clk-settle = <2>;

Best Regards,

K.

[dh84]

Hi @khang_letruong ,

If I change the values, "MIPI Frame Start Errors" occure. So I think my values are ok. My Mipi Clock is 112 MHz with a datarate of 224 MBit/s per lane.

Best regards,

Daniel

[khang_letruong]

Hi @chrispeterson,

I suppose that you are porting the camera driver for being used with the on-chip ISP since I saw that you disabled the ISI interfaces. For visualizing the output on the display, it would be better to run one of the gstreamer commands as  below (as indicated in the end of /opt/imx8-isp/bin/run.sh) : 

# gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=1920,height=1080" ! queue ! imxvideoconvert_g2d ! waylandsink
# gst-launch-1.0 -v v4l2src device=/dev/video0 ! "video/x-raw,format=YUY2,width=3840,height=2160" ! queue ! imxvideoconvert_g2d ! waylandsink
# gst-launch-1.0 -v v4l2src device=/dev/video0 ! waylandsink

You can also stream the video out of the board via an Ethernet connection by following the instruction of following knowledge sharing :https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-8-GStreamer-User-Guide/ta-p/1098942

Or, you can also use the provided utility /opt/imx8-isp/bin/video_test to dump either RAW or processed (YUV) images/frames out.

Hope that helps,

Khang

[chrispeterson]

Hi @khang_letruong 

Thank you for the response. I found that I can get a /dev/video2 device when I run

modprobe vvcam_video

I also get a couple more /dev/v4l-subdev* devices and another /dev/media1. The output of media-ctl is below

root@imx8mpevk:/opt/imx8-isp/bin# media-ctl -d 1 -p
Media controller API version 5.10.52

Media device information
------------------------
driver vvcam-video
model viv_media
serial 
bus info 
hw revision 0x0
driver version 5.10.52

Device topology
- entity 1: viv_v4l20 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "vvcam-isp.0":0 [ENABLED]

- entity 5: vvcam-dwe.0 (2 pads, 0 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev2
pad0: Source
pad1: Sink

- entity 8: vvcam-isp.0 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/v4l-subdev3
pad0: Source
-> "viv_v4l20":0 [ENABLED]

So, at this point it looks like I have the camera connected to the CSI in media 0 and the ISP output connected to dev/video0.   I don't see a way to connect the CSI to the ISP.

I attempted to run those commands you suggested and I get the following result:

root@imx8mpevk:/opt/imx8-isp/bin# ./isp_media_server CAMERA0 &
[1] 1223
root@imx8mpevk:/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video2 ! waylandsink
[ 171.538913] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 217.940400] enter imx378_priv_ioctl (cmd = -2140645888)
[ 217.951996] enter imx378_query_capability
[ 217.967758] enter isp_mi_stop
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR : [MediaPipeline] NativeSensor open error!
ERROR : [V4l2Event] initialize MediaPipeline error!

[ 223.359212] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 223.518907] viv_post_event: unsubscribed event id =14 type=0x08002000
e=0x08002000
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
[ 223.846933] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 224.014916] viv_post_event: unsubscribed event id =7 type=0x08002000
[ 224.178915] viv_post_event: unsubscribed event id =6 type=0x08002000
[ 224.342910] viv_post_event: unsubscribed event id =10 type=0x08002000
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../git/sys/v4l2/gstv4l2src.c(659): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:06.506453500
Setting pipeline to NULL ...
[ 224.538919] viv_post_event: unsubscribed event id =1 type=0x08002000
Freeing pipeline ...

I seems that isp_media_server is expecting a native driver, however the one I have is v4l2 based (which I defaulted to since I didn't see any of the vvcam/native folders that the porting guide mentions).

On another note, I found a different porting guide to what I was using (IMX8MPCSPUG.pdf) and section 6.2.6 mentions that I need to add some defines to the cmake command, however I am using the Yocto build system and I do not see a way to do this.

Further, section 6.2.4 mentions needing a "<sensor>.drv" file and I see n way of creating one.

Thank you,

Chris

[khang_letruong · Accepted solution]

Hi @chrispeterson ,

Firstly, the imx378.drv is important and it is used by the isp_media_server (or communicate with this later). Missing it is critical since there's no way for the isp_media_server to propagate the IOCTL calls to VVCAM kernel driver.

The IMX378.c that you created in isp-imx will serve to generate the imx378.drv. However, it might not be taken into account yet and you will need to add it via some relevant CMakelists.txt files.

Together with IMX378.c, you will ned an IMX378.xml for the calibration parameters,

Aside from the imx378.drv, there's will be libimx378.so, libimx378.so.1, libimx378.so.1.0.0 generated. Let's suppose that you build your isp-imx with ./build-all-isp.sh command, then the generated files would be found in build_output_release_partial/opt/imx8-isp/ and build_output_release_partial/usr/lib/.  You will need to copy them to to the target board :

scp -r build_output_release_partial/opt/imx8-isp target@target-ip:/opt/

scp -r build_output_release_partial/usr/lib/* target@target-ip:/usr/lib/

Also, you will need to adapt the start_isp.sh script and run.sh script to take into account of loading of your newly generated imx378.drv and IMX378.xml as well as the vvcam kernel driver (imx378.ko) described below :

For lower level vvcam kernel module (imx378.ko generated from imx378_mipi.c), you will need to compile it and copy to target board under /lib/module/<kernel-version>/extra/sensor/imx378/ (you might need to create this folder if it does not exist). 

On another note, I found a different porting guide to what I was using (IMX8MPCSPUG.pdf) and section 6.2.6 mentions that I need to add some defines to the cmake command, however I am using the Yocto build system and I do not see a way to do this.

Further, section 6.2.4 mentions needing a "<sensor>.drv" file and I see n way of creating one.

-- > You can modify the isp-imx recipe to add patch for your custom stuff of IMX378 described above. Or you can build it separately then scp-ing to the target board.

You can inspire the example of ov2775 which is a good example.

Regards,

K.

[CanberkOzbey]

I am struggling with similar kind of issue can you explain how can i generate drv and xml files

[khang_letruong]

Hi @CanberkOzbey ,

As a generic advice, you should open a new ticket for easier support.

Regards,

Khang

[chrispeterson]

Hi @khang_letruong 

Thank you very much for the help and suggestions.

I just needed to modify the run.sh for the imx378 and add the step to copy the files over in the recipie.

I did run into an issue where the CamEngineStart() call was returning RET_INVALID_PARM.  As it turns out there was something wrong with the IMX378.xml isp configuration - I copied the basler config and everything appears to work now.

If NXP people see this, it would be very helpful to have a step in the porting guide to modify the run.sh and add a copy of the .drv and .xml files.

Again, Thank you,

-Chris

[chrispeterson]

Hi @khang_letruong 

Thank you very much for the suggestion, I got a lot further.  I can see the camera get configured and looks like it should work, however during starting the  ISP driver returns a parameter error as shown below.

root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx378
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx378"...
Killing preexisting instances of video_test and isp_media_server:
PID TTY STAT TIME COMMAND 1033 pts/0 S+ 0:00 ./video_test -w 4048 -h 3040 -f RAW10 -t file -m 0 -d 2
[ 296.878947] viv_post_event: unsubscribed event id =4 type=0x08002000
Starting isp_media_server with configuration file CAMERA0
[ 297.038965] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 304.427751] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.439580] enter imx378_query_capability
[ 304.483156] enter imx378_priv_ioctl (cmd = -2140645888)
[ 304.488406] enter imx378_query_capability
loading [imx378.drv]...
[ 304.510213] enter imx378_priv_ioctl (cmd = 261)
[ 304.516684] enter imx378_query_supports
WARN : [DEWARP_HW] dw[ 304.521098] enter imx378_priv_ioctl (cmd = -2140645888)
e type /dev/v4l-subdev0 0
[ 304.529218] enter imx378_query_capability
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev1 1
WARN : [DEWARP_HW] dwe type /dev/v4l-subdev2 2
loading [imx378.drv]...[ 304.561912] enter imx378_priv_ioctl (cmd = 261)

[ 304.568089] enter imx378_query_supports
loading [imx378.drv]...[ 304.572873] enter imx378_priv_ioctl (cmd = 261)

[ 304.579125] enter imx378_query_supports
loading [imx378.drv]...[ 304.584259] enter imx378_priv_ioctl (cmd = 261)

[ 304.590577] enter imx378_query_supports
[ 304.606186] enter imx378_priv_ioctl (cmd = 261)
[ 304.610754] enter imx378_query_supports
[ 304.614608] enter imx378_priv_ioctl (cmd = 257)
[ 304.619173] enter imx378_priv_ioctl (cmd = 260)
[ 304.623720] enter imx378_get_clk
[ 304.626963] enter imx378_priv_ioctl (cmd = 259)
[ 304.631530] enter imx378_priv_ioctl (cmd = 256)
[ 304.636073] enter imx378_priv_ioctl (cmd = 262)
[ 304.640612] enter imx378_set_sensor_mode
[ 304.644553] enter imx378_priv_ioctl (cmd = 263)
[ 304.649109] enter imx378_get_sensor_mode
[ 304.653054] enter imx378_priv_ioctl (cmd = 261)
[ 304.657593] enter imx378_query_supports
isp input: 4048x3040 RAW10
isp output: 3840x2160 YUV422I
dewarp input: 3840x2160 YUV422I
dewarp output: 3840x2160 YUV422I
[ 304.751846] enter isp_mi_stop
[ 304.757250] enter imx378_set_fmt
[ 304.760529] enter imx378_write_reg_arry
[ 304.905955] enter imx378_get_format_code
[ 304.909943] enter imx378_priv_ioctl (cmd = 285)
[ 304.914513] enter imx378_set_test_pattern
[ 304.918557] enter imx378_write_reg
[ 304.922576] enter imx378_write_reg
[ 304.926560] enter imx378_write_reg
[ 304.930558] enter imx378_write_reg
[ 304.934583] enter imx378_priv_ioctl (cmd = 279)
[ 304.939124] enter imx378_set_fps
[ 304.942349] enter imx378_write_reg
[ 304.946360] enter imx378_write_reg
[ 304.950353] enter imx378_write_reg
[ 304.954342] enter imx378_write_reg
[ 304.958352] enter imx378_priv_ioctl (cmd = 263)
[ 304.962900] enter imx378_get_sensor_mode
camEngineConfig.type = 1[ 304.967415] enter imx378_priv_ioctl (cmd = 274)

camEngineConfig.pathConfig[0].w[ 304.973526] enter imx378_set_exp
idth = 3840
camEngineConfig.path[ 304.979670] enter imx378_write_reg
Config[0].height = 2160
camEngin[ 304.986491] enter imx378_write_reg
eConfig.pathConfig[0].mode = 4
c[ 304.992758] enter imx378_write_reg
amEngineConfig.pathConfig[0].layo[ 304.999031] enter imx378_write_reg
ut = 3
camEngineConfig.pathConfi[ 305.005318] enter imx378_priv_ioctl (cmd = 277)
g[0].alignMode = 0
camEngineConf[ 305.012126] enter imx378_set_gain
ig.pathConfig[1].width = 0
camEn[ 305.018323] enter imx378_write_reg
gineConfig.pathConfig[1].height =[ 305.025170] enter imx378_write_reg
0
camEngineConfig.pathConfig[1][ 305.031449] enter imx378_write_reg
.mode = 1
camEngineConfig.pathCo[ 305.037712] enter imx378_write_reg
nfig[1].layout = 3
camEngineConfig.pathConfig[1].alignMode = 0
camEngineConfig.pathConfig[2].width = 0
camEngineConfig.pathConfig[2].height = 0
camEngineConfig.pathConfig[2].mode = 1
camEngineConfig.pathConfig[2].layout = 3
camEngineConfig.pathConfig[2].alignMode = 0
camEngineConfig.pathConfig[3].width = 0
camEngineConfig.pathConfig[3].height = 0
camEngineConfig.pathConfig[3].mode = 1
camEngineConfig.pathConfig[3].layout = 3
camEngineConfig.pathConfig[3].alignMode = 65535
camEngineConfig.pathConfig[4].width = 0
camEngineConfig.pathConfig[4].height = 0
camEngineConfig.pathConfig[4].mode = 1
camEngineConfig.pathConfig[4].layout = 3
camEngineConfig.pathConfig[4].alignMode = 65535
camEngineConfig.data.sensor.hSensor = 281473366536816
camEngineConfig.data.sensor.hSubSensor = 0
camEngineConfig.data.sensor.hCamCalibDb = 281473366260096
camEngineConfig.data.sensor.ifSelect = 1
camEngineConfig.data.sensor.sampleEdge = 2
camEngineConfig.data.sensor.hSyncPol = 1
camEngineConfig.data.sensor.vSyncPol = 2
camEngineConfig.data.sensor.bayerPattern = 3
camEngineConfig.data.sensor.subSampling = 3
camEngineConfig.data.sensor.seqCCIR = 1
camEngineConfig.data.sensor.fieldSelection = 1
camEngineConfig.data.sensor.inputSelection = 2
camEngineConfig.data.sensor.dvpPinMapping = 0
camEngineConfig.data.sensor.mode = 4
camEngineConfig.data.sensor.acqWindow.hOffset = 0
camEngineConfig.data.sensor.acqWindow.vOffset = 0
camEngineConfig.data.sensor.acqWindow.width = 4048
camEngineConfig.data.sensor.acqWindow.height = 3040
camEngineConfig.data.sensor.outWindow.hOffset = 0
camEngineConfig.data.sensor.outWindow.vOffset = 0
camEngineConfig.data.sensor.outWindow.width = 4048
camEngineConfig.data.sensor.outWindow.height = 3040
camEngineConfig.data.sensor.isWindow.hOffset = 0
camEngineConfig.data.sensor.isWindow.vOffset = 0
camEngineConfig.data.sensor.isWindow.width = 4048
camEngineConfig.data.sensor.isWindow.height = 3040
camEngineConfig.data.sensor.mipiMode = 43
camEngineConfig.data.sensor.enable3D = 0
camEngineConfig.data.sensor.enableTestpattern = 0
camEngineConfig.data.sensor.flickerPeriod = 1
camEngineConfig.data.sensor.enableAfps = 0
camEngineConfig.data.sensor.szSensorName[256] =
camEngineConfig.data.sensor.szCsiName[256] =
camEngineConfig.data.sensor.csiFormat = 808534599
camEngineConfig.data.sensor.csiPad = 0
camEngineConfig.data.sensor.stitchingMode = 0
camEngineConfig.data.sensor.compress_curve.enable = 0
camEngineConfig.data.sensor.compress_curve.in_bit = 0
camEngineConfig.data.sensor.compress_curve.out_bit = 0
camEngineConfig.data.sensor.expand_curve.enable = 0
camEngineConfig.data.sensor.expand_curve.in_bit = 0
camEngineConfig.data.sensor.expand_curve.out_bit = 0
camEngineConfig.data.sensor.colorspace = 0
camEngineConfig.data.sensor.colorrange = 0
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2340: start() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:427: inputConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1359: cameraConnect() = 13(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2371: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1084: streamingStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1639: previewStart() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_engine_interface.cpp:2382: streamingStop() = 12(RET_WRONG_STATE)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1104: streamingStop() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/un[ 308.127902] enter imx378_priv_ioctl (cmd = 272)
its/cam_device/source/camera/cam_[ 308.133165] enter imx378_s_stream
operations.cpp:1652: previewStop([ 308.139233] enter imx378_write_reg
) = 12(ret)
[ERR] /usr/src/debu[ 308.146157] enter imx378_priv_ioctl (cmd = 260)
g/isp-imx/4.2.2.15.0-r0/isp-imx-4[ 308.152908] enter imx378_get_clk
.2.2.15.0/units/cam_device/source[ 308.158977] enter imx378_priv_ioctl (cmd = 259)
/camera/cam_engine_interface.cpp:[ 308.166368] enter imx378_priv_ioctl (cmd = 257)
757: cprocEnableSet() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:820: inputDisconnect() = 12(ret)
[ERR] /usr/src/debug/isp-imx/4.2.2.15.0-r0/isp-imx-4.2.2.15.0/units/cam_device/source/camera/cam_operations.cpp:1372: cameraDisconnect() = 12(ret)

 I modified the wrapper function to dump the config structure and I do not see anything that stands out that would cause a parameter error.

Any ideas?

Thank you,

Chris

[Amal_Antony3331]

Hi @chrispeterson 

Can you please guide to enable the complete tracing of isp-imx.

I'm getting below logs while running the run.sh file. Seems like IMX219.drv is not get loaded.

It would be good if you can share the run.sh script file and xml file.

root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx219_4K -lm
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx219_4K"...
Killing preexisting instances of video_test and isp_media_server:
PID TTY STAT TIME COMMAND 859 ttymxc1 S 0:00 ./isp_media_server CAMERA0
Removing vvcam-isp...
Removing vvcam-dwe...
Removing vvcam-video...
[  416.253663] enter viv_video_exit_module
Removing imx8-media-dev...
[  416.391695] : Unregistered all entities
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module imx219 ...
imx219
imx219                 32768  0
imx219 already loaded.
Loading module imx8-media-dev ...
imx8_media_dev
[  416.481570] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
[  416.492701] mx8-img-md: Registered sensor subdevice: imx219 6-0010 (1)
[  416.499294]   enter (( imx219_link_setup ))
[  416.503501] mx8-img-md: created link [imx219 6-0010] => [mxc-mipi-csi2.0]
[  416.510316] mxc-md 32c00000.bus:camera: mxc_md_create_links
Loaded /lib/modules/5.10.35-lts-5[  416.520390]    enter imx219_priv_ioctl ((( (cmd = -2140645888))))
.10.y+g1b8cec2a6c43/kernel/driver[  416.529418]  <<<  Command : VIDIOC_QUERYCAP 
s/staging/media/imx/imx8-media-de[  416.536277]   enter (( imx219_ioc_qcap ))
v.ko 
Loading module vvcam-video[  416.543152]  CSI number = 0 
 ...
vvcam_video
[  416.548866] cap->bus_info[VVCAM_CAP_BUS_INFO_I2C_ADAPTER_NR_POS] = 6 
[  416.557118] enter viv_video_init_module
Loaded /lib/modules/5.10.35-lts-5.10.y+g1b8cec2a6c43/extra/video/vvcam-video.ko 
Loading module vvcam-dwe ...
vvcam_dwe
[  416.577394] enter isp_mi_stop
vvcam_dwe              28672  3
vvcam-dwe already loaded.
Loading module vvcam-isp ...
vvcam_isp
[  416.589329] enter isp_mi_stop
vvcam_isp              65536  4
vvcam-isp already loaded.
Starting isp_media_server with configuration file CAMERA0
root@imx8mpevk:/opt/imx8-isp/bin# INFO   : [ISP_MEDIA_SERVER] ******************************************************************
INFO   : [ISP_MEDIA_SERVER] VIV ISP Media Control Framework V4.2.2p13 (Jun  8 2021, 10:39:59)
INFO   : [ISP_MEDIA_SERVER] ******************************************************************

Attaching the run.sh file that I'm currently using.

Also one point to clarify, is IMX219.drv is actually getting loaded while running the run.sh ?

Regards

Charles