Integrating MIPI CSI camera in IMX8MP processor

[kaushlendra]

HI All,

I am trying to integrate mipi camera(IMX219) to our IMX8MP custom board. I have created our our vvcam kernel driver and isi layer driver for imx219 camera using document: https://www.nxp.com/docs/en/user-guide/iMX8MP_CAMERA_DISPLAY_GUIDE.pdf and taking reference of ov2775 driver already present in the sdk. I have changes run.sh file to incorporate our changes and once board is up, we run our run.sh script and the out put is:

root@imx8mpevk:/opt/imx8-isp/bin# ./run.sh -c imx219_4K -lm
RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Trying configuration "imx219_4K"...
Removing vvcam-isp...
[ 138.450707] enter viv_isp_exit_module
[ 138.454595] enter isp_hw_remove
[ 138.457774] vvcam isp driver removed
[ 138.461653] enter isp_hw_remove
[ 138.464861] vvcam isp driver removed
Removing vvcam-dwe...
Removing vvcam-video...
Removing imx8-media-dev...
[ 138.511225] : Unregistered all entities
Removing ov2775...
Removing os08a20...
Removing basler-camera-driver-vvcam...
Loading module imx219 ...
imx219
Loaded /lib/modules/5.10.35-lts-5.10.y+g2ae966500e50/kernel/drivers/media/i2c/imx219.ko
Loading module imx8-media-dev ...
imx8_media_dev
[ 138.688861] imx8_media_dev: module is from the staging directory, the quality is unknown, you have been warned.
Loaded /lib/modules/5.10.35-lts-5.10.y+g2ae966500e50/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Loading module vvcam-video ...
vvcam_video
[ 138.734792] enter viv_video_init_module
Loaded /lib/modules/5.10.35-lts-5.10.y+g2ae966500e50/extra/video/vvcam-video.ko
Loading module vvcam-dwe ...
vvcam_dwe
[ 138.775120] enter viv_dwe_init_module
Loaded /lib/modules/5.10.35-lts-5.10.y+g2ae966500e50/extra/vvcam-dwe.ko
Loading module vvcam-isp ...
vvcam_isp
[ 138.810823] enter viv_isp_init_module
[ 138.814961] enter isp_hw_probe
[ 138.818269] vvcam isp driver registered
[ 138.823319] enter isp_hw_probe
[ 138.826602] vvcam isp driver registered
Loaded /lib/modules/5.10.35-lts-5.10.y+g2ae966500e50/extra/vvcam-isp.ko

after this we ran isp_media_server:

root@imx8mpevk:/opt/imx8-isp/bin# ./isp_media_server CAMERA0 &
[1] 1083
root@imx8mpevk:/opt/imx8-isp/bin#

This is my devices after drivers loaded:

oot@imx8mpevk:/opt/imx8-isp/bin# v4l2-ctl --list-devices
VIV (platform:viv0):
/dev/video2

VIV (platform:viv1):
/dev/video3

vsi_v4l2dec (platform:vsi_v4l2dec):
/dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
/dev/video0

viv_media (platform:vvcam-video):
/dev/media0

root@imx8mpevk:/opt/imx8-isp/bin#

I tried running media-ctl command and this is my output:

root@imx8mpevk:/opt/imx8-isp/bin# media-ctl -p
Media controller API version 5.10.35

Media device information
------------------------
driver vvcam-video
model viv_media
serial
bus info
hw revision 0x0
driver version 5.10.35

Device topology
- entity 1: viv_v4l20 (1 pad, 0 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink

- entity 5: viv_v4l21 (1 pad, 0 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink

- entity 9: vvcam-isp.0 (1 pad, 0 link)
type Node subtype V4L flags 0
pad0: Source

- entity 11: vvcam-isp.1 (1 pad, 0 link)
type Node subtype V4L flags 0
pad0: Source

root@imx8mpevk:/opt/imx8-isp/bin#

Also, I tried launching the camera feed on /dev/video3 interface:

root@imx8mpevk:/opt/imx8-isp/bin# gst-launch-1.0 -v v4l2src device=/dev/video3 ! waylandsink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 1008.960547] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1009.120536] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1009.284542] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1009.448539] viv_post_event: unsubscribed event id =14 type=0x08002000
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
[ 1009.612524] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1009.780564] viv_post_event: unsubscribed event id =7 type=0x08002000
[ 1009.944558] viv_post_event: unsubscribed event id =6 type=0x08002000
[ 1010.108555] viv_post_event: unsubscribed event id =10 type=0x08002000
[ 1010.122522] cma: cma_alloc: alloc failed, req-size: 4050 pages, ret: -12
[ 1010.138444] cma: cma_alloc: alloc failed, req-size: 4050 pages, ret: -12
[ 1010.147789] cma: cma_alloc: alloc failed, req-size: 4050 pages, ret: -12
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../git/sys/v4l2/gstv4l2src.c(659): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:01.513390750
Setting pipeline to NULL ...
[ 1010.316598] viv_post_event: unsubscribed event id =1 type=0x08002000
Freeing pipeline ...
root@imx8mpevk:/opt/imx8-isp/bin#

the devicetree added is pasting here:

&i2c3 {
clock-frequency = <100000>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c3>;
status = "okay";

/delete-node/ov5640_mipi@3c;

mux_0: mux_0@71 {
compatible = "nxp, pca9849";
reg = <0x71>;
#address-cells = <1>;
#size-cells = <0>;
status = "okay";
powerdown-gpios = <&gpio1 10 GPIO_ACTIVE_LOW>;
reset-gpios = <&gpio3 19 GPIO_ACTIVE_HIGH>;

/* IMX219 configuration which is connected to first channel of TMUX646 */
i2c@0 {
#address-cells = <1>;
#size-cells = <0>;
reg = <0>;
status = "okay";

imx219@10 {
compatible = "sony,imx219";
reg = <0x10>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_csi0_pwn>, <&pinctrl_csi0_rst>, <&pinctrl_csi_mclk>;
clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
clock-names = "xclk";
assigned-clocks = <&clk IMX8MP_CLK_IPP_DO_CLKO2>;
assigned-clock-parents = <&clk IMX8MP_CLK_24M>;
assigned-clock-rates = <24000000>;
csi_id = <0>;
mclk = <24000000>;
mclk_source = <0>;
mipi_csi;
status = "okay";

port {
imx219_mipi_0_ep: endpoint {
remote-endpoint = <&mipi_csi0_ep>;
data-lanes = <1 2>;
clock-lanes = <0>;
};
};
};
};
};

};

&cameradev {
status = "okay";
};

&isi_0 {
status = "disabled";
};

&isi_1 {
status = "disabled";
};

&isp_0 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_0 {

#address-cells = <1>;
#size-cells = <0>;
status = "okay";

port@0 {
reg = <0>;
mipi_csi0_ep: endpoint {
remote-endpoint = <&imx219_mipi_0_ep>;
data-lanes = <2>;
csis-hs-settle = <13>;
csis-clk-settle = <2>;
csis-wclk;
};
};

};

Can somebody, please help me on how I can make this work?

[Naz]

Could you share following header files ?

#include "imx219_regs_1080p.h"
#include "imx219_regs_1080p_hdr.h"
#include "imx219_regs_1080p_hdr_low_freq.h"
#include "imx219_regs_1080p_native_hdr.h"

[khang_letruong]

Hi @kaushlendra,

Look like the default frame-rate is too high (120/1) to allocate enough memory buffer :

GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)3840, height=(int)2160, framerate=(fraction)120/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:6:11:1

You can try with appropriate frame-rate based on the calculation of resolution and the bandwidth of the MIPI-CSI interface, or blindly with smaller value to bigger value (5, 10, ...) in the gst-launch command :

gst-launch-1.0 -v v4l2src device=/dev/video3 ! "video/x-raw,format=YUY2,width=3084,height=2016, framerate=5/1" ! queue ! imxvideoconvert_g2d ! waylandsink

Regards,

Khang

[kaushlendra]

HI @khang_letruong ,

As you suggested, I have tried different frame rate starting from 5 and increasing gradually. but everytime I end up getting the same result.

I am not sure how to move forward as I am new to this. Please guide me on how I can resolve this.

[khang_letruong]

Hi @kaushlendra ,

How much RAM memory does your custom board have ? Is the reserve-memory well allocated ? Example :

root@imx-camera:~# dmesg  | grep -i cma
[    0.000000] Reserved memory: created CMA memory pool at 0x00000000c4000000, size 960 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Memory: 4787972K/5988352K available (16764K kernel code, 1214K rwdata, 6432K rodata, 2816K init, 1013K bss, 217340K reserved, 983040K cma-reserved)

Khang

[kaushlendra]

HI @khang_letruong ,

8 GB of RAM memory we are using in our custom board.

While running the command to verify the cma memory allocation:

dmesg | grep -i cma
[ 0.000000] OF: reserved mem: failed to allocate memory for node 'linux,cma'
[ 0.000000] cma: Reserved 32 MiB at 0x00000000fe000000
[ 0.000000] Memory: 7805708K/8388608K available (17536K kernel code, 1470K rwdata, 6784K rodata, 2880K init, 543K bss, 550132K reserved, 32768K cma-reserved)
root@imx8mpevk:~#

Changes we have done to incorporate 8 GB of memory are:

diff --git a/arch/arm/dts/imx8mp-evk.dts b/arch/arm/dts/imx8mp-evk.dts
index f0159d499e..400e3290e7 100644
--- a/arch/arm/dts/imx8mp-evk.dts
+++ b/arch/arm/dts/imx8mp-evk.dts
@@ -31,8 +31,8 @@

memory@40000000 {
device_type = "memory";
- reg = <0x0 0x40000000 0 0xc0000000>,
- <0x1 0x00000000 0 0xc0000000>;
+ reg = <0x0 0x40000000 0x0 0xC0000000>,
+ <0x1 0x00000000 0x1 0x40000000>;
};

reg_can1_stby: regulator-can1-stby {

diff --git a/include/configs/imx8mp_evk.h b/include/configs/imx8mp_evk.h
index 10e87f349d..db5baec070 100644
--- a/include/configs/imx8mp_evk.h
+++ b/include/configs/imx8mp_evk.h
@@ -212,12 +212,12 @@
/* Totally 6GB DDR */
#define CONFIG_SYS_SDRAM_BASE 0x40000000
#define PHYS_SDRAM 0x40000000
-#define PHYS_SDRAM_SIZE 0xC0000000 /* 3 GB */
-#define PHYS_SDRAM_2 0x100000000
+#define PHYS_SDRAM_SIZE 0xC0000000 /* 4 GB */
+#define PHYS_SDRAM_2 0x100000000
#ifdef CONFIG_TARGET_IMX8MP_DDR4_EVK
-#define PHYS_SDRAM_2_SIZE 0x40000000 /* 1 GB */
+#define PHYS_SDRAM_2_SIZE 0x140000000 /* 1 GB */
#else
-#define PHYS_SDRAM_2_SIZE 0xC0000000 /* 3 GB */
+#define PHYS_SDRAM_2_SIZE 0x140000000 /* 4 GB */
#endif

#define CONFIG_MXC_UART_BASE UART2_BASE_ADDR

diff --git a/arch/arm64/boot/dts/freescale/imx8mp.dtsi b/arch/arm64/boot/dts/freescale/imx8mp.dtsi
index 3853084f1d63..4352160b7726 100755
--- a/arch/arm64/boot/dts/freescale/imx8mp.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8mp.dtsi
@@ -186,8 +186,8 @@ ocram: ocram@900000 {
linux,cma {
compatible = "shared-dma-pool";
reusable;
- size = <0 0x3c000000>;
- alloc-ranges = <0 0x40000000 0 0xC0000000>;
+ size = <0 0xc0000000>;
+ alloc-ranges = <0x0 0x40000000 0x0 0xC0000000>;
linux,cma-default;
};

[khang_letruong]

Hi,

The following is possibly the issue as I had similar symptom :

[ 0.000000] OF: reserved mem: failed to allocate memory for node 'linux,cma'
[ 0.000000] cma: Reserved 32 MiB at 0x00000000fe000000

Only 32MiB instead of 960MiB of CMA memory was allocated.

You will need to adjust the layout of the reserved-memory based on your custom RAM.

Regards,

Khang

[kaushlendra]

I am not sure about where we can change the layout for the reserved memory.

[ 0.000000] cma: Reserved 32 MiB at 0x00000000fe000000

 but address 0xfe000000 is associated with OP-TEE which we have disabled as my RAM address size was not getting changed to 8 GB even after above patches.

So, with the above patches, I have disabled OP-TEE to incorporate my 8 GB RAM memory. but still not sure where I need to change to increase the Reserved memory size.

[khang_letruong]

Hi @kaushlendra ,

You will have to manage to avoid the following error :

[ 0.000000] OF: reserved mem: failed to allocate memory for node 'linux,cma'

One suggestion could be removing the patch to keep 6GB of RAM memory as in the case of EVK (and ignore to additional 2GB) to see if the 960MiB of CMA reserved memory could be allocated. Then if yes, you could test your issue of gst-lauch. But I'm quite sure that the failure of CMA reserved memory allocation is the root cause.

Regards,

Khang