topic Re: IMX8MP use both ISI and ISP in i.MX Processors

IMX8MP use both ISI and ISP

malik_cisse — Mon, 28 Feb 2022 10:46:30 GMT

Hi,

Is it actually possible to have both enabled, ISI and ISP?
My use case:

Raw bayer CSI camera ->ISI->ISP-> preview on screen

ISI-> Software ISP-> 12bit TIFF snapshots

This is to enable 12bit high resolution snapshots without stopping live preview.

BR, MC

Re: IMX8MP use both ISI and ISP

joanxie — Tue, 01 Mar 2022 09:10:21 GMT

why do you need connect isi to isp? why don't you connect your camera to ISP directly? imx8mp has dual isp, you can use dual ISP directly

Re: IMX8MP use both ISI and ISP

malik_cisse — Tue, 01 Mar 2022 11:24:24 GMT

Hi Joanxie,

The idea is to use ISI Raw Bayer 12bit output on /dev/video0 while on the same time use ISP processed image on /dev/video1 for example.

This way I have two independant video sources, one for display and one for Raw snapshots.

I know I can set ISP in bypass mode and get RG12 Raw Bayer image but I need to stop the display pipeline first for that.

I don't want to stop display pipeline because this is bad user experience.

BR, MC

Re: IMX8MP use both ISI and ISP

dianapredescu — Tue, 01 Mar 2022 12:11:08 GMT

Hi Malik,

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:

|------------------------> ISI0

CSI2 -------->| (no need any operation of register for this mux)

|-------------------------> ISP0

Regards,

Diana

Re: IMX8MP use both ISI and ISP

malik_cisse — Tue, 01 Mar 2022 14:12:36 GMT

Hi Diana,

Thanks for the hint although I knew that already.

My question is whether I can connect ISI to ISP and use both outputs simultaneously.

Thx, MC

Re: IMX8MP use both ISI and ISP

joanxie — Thu, 03 Mar 2022 03:47:23 GMT

you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

Re: IMX8MP use both ISI and ISP

malik_cisse — Thu, 03 Mar 2022 10:31:18 GMT

Hi Joanxie,

>>you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

Awesome! Well this is exactly what I need. When I tried to enable both ISI and ISP in parallel one year ago it did not work though. no /dev/video0 device node was created. I had to remove ISI from device tree in order to make ISP work.

Can you please point me to any example that demonstrate this?

BR, MC

Re: IMX8MP use both ISI and ISP

joanxie — Fri, 04 Mar 2022 14:23:52 GMT

current bsp already support ISP+ISI in the same time, you can refer to this：

“https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-basler-ov5640.dts?h=lf-5.10.y”

this dts support ov5640 with ISI and basler with ISP in the same time, you can refer to this

Re: IMX8MP use both ISI and ISP

malik_cisse — Fri, 04 Mar 2022 17:06:06 GMT

Unfortunately this link is not working:

“https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-basler-ov5640.dts?h=lf-5.10.y”

Re: IMX8MP use both ISI and ISP

joanxie — Sat, 05 Mar 2022 02:39:55 GMT

you can select version on the top right corner, try again

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-basler-ov5640.dts?h=lf-5.10.y"

Re: IMX8MP use both ISI and ISP

malik_cisse — Mon, 07 Mar 2022 11:22:33 GMT

Hi Joan,

This time the link you sent works.

However when I implement same device tree as you sent I do not get any image on "/dev/video0" anymore.

When launching gstreamer pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=3840,height=2160 ! queue ! videocrop top=0 left=120 right=120 bottom=0 ! imxvideoconvert_g2d rotation=3 ! kmssink can-scale=false sync=false

I get following error:

root@amrum-fumu1-imx8mp-2:~#
root@imx8mp:~# [ 56.241970] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 105.490628] enter isp_mi_stop

Please find my device tree attached.

I also tried variations of ISI settings (isi_0 = okay) with no success.

It seems some sort of software adaptation needs to be done as well. Do you know what needs to be changed to make ISI work?

Thx, MC

Re: IMX8MP use both ISI and ISP

malik_cisse — Mon, 07 Mar 2022 11:23:26 GMT

obviously dt file could not be sent.

Here the important part:

&i2c3 {
cam1: camera1@1a {
compatible = "sony,imx477";
reg = <0x1a>;
#address-cells = <0x1>;
#size-cells = <0x0>;
clocks = <&cam1_clk>;
clock-names = "ext";
pw-enable-gpios = <&gpio1 14 GPIO_ACTIVE_HIGH>;

port {
cam1_mipi_ep: endpoint {
csi-port = <0>;
bus-width = <4>;
remote-endpoint = <&mipi_csi0_ep>;
bus-type = <4>; /* MIPI CSI-2 D-PHY */
data-lanes = <1 2 3 4>;
clock-lanes = <0>;
clock-noncontinuous = <1>;
link-frequencies =
/bits/ 64 <750000000>;
};
};

};
};

&isi_0 {
status = "disabled";
};

&isi_1 {
status = "okay";
};

&isp_0 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_1 {
status = "okay";
};

Re: IMX8MP use both ISI and ISP

khang_letruong — Mon, 07 Mar 2022 14:24:22 GMT

Hi,

I am allowed to share some personal thought on this subject :

you can select version on the top right corner, try again
"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b..."

The solution proposed by @joanxie seems to be with 2 independent pipelines :

1. Basler camera --> MIPI0 --> ISP0
2. OV5640 camera --> MIPI1 --> ISI1

Meanwhile, I would guess that @malik_cisse expects to be able to access simultaneously different /dev/videoX and /dev/videoY for the same input camera :

a. IMX477 --> MIPIx ---> ISPx (bypass the demosiacing for having RAW bayer output on /dev/videoX)
|
+--> ISPy (YUV output on /dev/videoY)

b. IMX477 --> MIPIx ---> ISIx (RAW bayer output on /dev/videoX)
|
+--> ISPx (YUV output on /dev/videoY)

c. IMX477 --> MIPIx --> ISIx --> ISPx (YUV output on /dev/videoX)
|
+--> RAW bayer output on /dev/videoY

d. IMX477 --> MIPIx --> ISIx --> ISPy (YUV output on /dev/videoY)
|
+ --> RAW bayer output on /dev/videoX

However, I think that neither of above a, b, c, d is feasible.

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:
|------------------------> ISI0
CSI2 -------->| (no need any operation of register for this mux)
|-------------------------> ISP0

Moreover, I am not sure if the above theory of @dianapredescu (which is same to above case b) works since I haven't seen similar configuration in any reference dts until now.

And the following application note is only limited at single MIPI to dual ISIs, but no info about ISP : https://www.nxp.com/docs/en/application-note/AN13430.pdf

Best Regards,
K.

Re: IMX8MP use both ISI and ISP

malik_cisse — Mon, 07 Mar 2022 14:39:41 GMT

Thank you Khang for bringing this to the point so necelly.
This is exactly what I am trying to do. Any of the scenarios you have pointed out would unlock my issue if they would work.
I can also imagine that many people will have similar issue.

@dianapredescu: Your statement saying that "CSI2 channel can output to ISI and ISP at the same time" is indeed very interesting but as Khang also pointed out there is no actual proof this is feasible. At least it is not feasible by device tree tweaking only.

Any actual working examples would really be appreciated.

BR, MC

Re: IMX8MP use both ISI and ISP

khang_letruong — Mon, 07 Mar 2022 15:14:09 GMT

Hi,

Another hypothesis which could be more feasible :

And the following application note is only limited at single MIPI to dual ISIs, but no info about ISP : https://www.nxp.com/docs/en/application-note/AN13430.pdf

Combining the scenario in the above application note :

|------------------------> ISI0
CSI2 -------->|
|-------------------------> ISI1

and the theory of @dianapredescu :

|------------------------> ISI0

CSI2 -------->| (no need any operation of register for this mux)

|-------------------------> ISP0

I do hope hope that below configuration would be supported in future :

|------------------------> ISI0
CSI2 -------->|
|-------------------------> ISP1 (with the fact from the mentioned application note that output of MIPI-CSI2 can be hooked to ISI1, why couldn't it be hooked to ISP1 as in current single pipeline ? )

Best Regards,
K.

Re: IMX8MP use both ISI and ISP

dianapredescu — Mon, 07 Mar 2022 15:33:53 GMT

@malik_cisse ,

The only thing I tested myself personally was:

|------------------------> ISI0

CSI2 ---> ISP0-------->|

|-------------------------> ISI1

But that was possible because ISI can have as input CSI2 or AXI Memory, and used the m2m functionality of ISI.

The scenarios @khang_letruong (btw thanks for explaining the use case) mentioned, indeed not sure if are feasible because looking at the 8MP RM it is stated:

There are two instances of ISP on the chip and each is connected to separate instances of
MIPI CSI.

So my understanding would be that I can't have the ISP input coming from anywhere else but camera sensor.

But let me look into the possibility I mentioned few messages above to see if it actually works and if we have some examples.

Regards,

Diana

Re: IMX8MP use both ISI and ISP

khang_letruong — Mon, 07 Mar 2022 15:21:50 GMT

Dear @dianapredescu ,

The only thing I tested myself personally was:
|------------------------> ISI0
CSI2 ---> ISP0-------->|
|-------------------------> ISI1

Are you sure that the above configuration works (i.e hooking the output of ISP to ISI) since it seems to be impossible within the following discussion : https://community.nxp.com/t5/i-MX-Processors/IMX8MP-use-ISI-with-ISP/m-p/1376237

Can you confirm ?

Thanks and regards,
K.

Re: IMX8MP use both ISI and ISP

dianapredescu — Mon, 07 Mar 2022 15:30:54 GMT

Hi @khang_letruong,

I did some experiments few days ago, so here is what I have noticed:

I edited a bit imx8mp.dtsi file to configure isi_1 as m2m device too. I modified imx8mp-evk-basler-ov5640.dtb to enable isi_1, isi_0 and isp_0. After booting the board I am able to find the Gstreamer plugin for m2m:

root@imx8mpevk:~# gst-inspect-1.0 | grep v4l2
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2convert: V4L2 Video Converter
video4linux2: v4l2video3convert: V4L2 Video Converter
video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h265enc: V4L2 H.265 Encoder
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h265dec: V4L2 H265 Decoder
video4linux2: v4l2vp8dec: V4L2 VP8 Decoder
video4linux2: v4l2vp9dec: V4L2 VP9 Decoder

as well as the m2m_devices:

root@imx8mpevk:~# v4l2-ctl --list-devices

mxc-isi-m2m (platform:32e00000.isi:m2m_devic):
/dev/video2

mxc-isi-m2m (platform:32e02000.isi:m2m_devic):
/dev/video3

VIV (platform:viv0):
/dev/video5

So I created a quick pipeline and redirected isp_0 to both isi_0 and isi_1.

gst-launch-1.0 v4l2src device=/dev/video5 num-buffers=20 ! video/x-raw, width=1920, height=1080, framerate=60/1 ! tee name=t ! queue ! v4l2convert ! video/x-raw, width=1920, height=1080, format=RGB16 ! vpuenc_h264 ! queue ! filesink location=out1.h264 t. ! queue ! v4l2video3convert ! video/x-raw, width=1280, height=720, format=BGRA ! vpuenc_h264 ! queue ! filesink location=out2.h264

In my case v4l2convert will use /dev/video2 which is mapped to isi_0 and v4l2video3convert will use /dev/video3 which corresponds to isi_1. Used the tee element which splits the data in 2 identical ways, so at the end I have 1920x1080 and 1280x720 video streams.

Let me know if this is helpful.

Regards,

Diana

Re: IMX8MP use both ISI and ISP

khang_letruong — Mon, 07 Mar 2022 15:44:48 GMT

Dear @dianapredescu

Thanks for the confirmation. But still data behind ISP cannot be read, and this configuration seems "heavier" than one proposed in this application note https://www.nxp.com/docs/en/application-note/AN13430.pdf with intermediate ISP btw MIPI-CSI2 and ISI(s).

Regards,

Khang

Re: IMX8MP use both ISI and ISP

malik_cisse — Mon, 07 Mar 2022 16:18:06 GMT

Hi Diana,

This is getting very interesting.

I learned something in your and Khang's feedback: One can configure ISI to do m2m access as can be seen in https://www.nxp.com/docs/en/application-note/AN13430.pdf page 3

isi interface = <5 0 2>.

However your example is not useful for two reasons:

you are using isi0 and isi1 to convert YUV to RGB just before encoding in H.264. This does not make sense because H.264 encoder uses YUV420 input already.
The imx8mp GPU can do yuv to RGB conversion. No need for isi for this

For me isi only makes sense when connected to CSI directly but unfortunately we cannot seem to connect both isi and isp to CSI at the same time as in your earlier post:

|------------------------> ISI0

CSI2 -------->| (no need any operation of register for this mux)

|-------------------------> ISP0