IMX8MP use both ISI and ISP

[malik_cisse]

Hi,

Is it actually possible to have both enabled, ISI and ISP?
My use case:

Raw bayer CSI camera ->ISI->ISP-> preview on screen

ISI-> Software ISP-> 12bit TIFF snapshots

This is to enable 12bit high resolution snapshots without stopping live preview.

BR, MC

[malik_cisse]

Hi,
I am doing 4K (3840x2160) using ISI.

[billie]

Hi,

Thanks for your answer.

I'd like to use 12MP image sensor through the ISI.

Is there any problem ?

Regards,

Billie.

[dianapredescu]

Hi Malik,

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

Regards,

Diana

[malik_cisse]

Hi Diana,

This is a late feedback on this topic but I wanted to confirm that your scenario indeed works!

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

I could demonstrate this on latest ISP version 20 using Kirkstone BSP. 
The only thing to do to prevent resource mismatches was to implement  the imx8-mipi-csi2-sam.c and vvcam driver as described in https://www.nxp.com/docs/en/application-note/AN13430.pdf.
on page 6.
The device tree patches are not necessary. Only page 6 stuff above.
Thank you again for showing this is actually feasible. Otherwise I would not even have tried it I guess.
Now I have a bullet proof multimedia application based on imx8mp.

[malik_cisse]

Hi Diana,

Thanks for the hint although I knew that already.

My question is whether I can connect ISI to ISP and use both outputs simultaneously.

Thx, MC

[joanxie]

you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

[malik_cisse]

Hi Joanxie,

>>you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

Awesome! Well this is exactly what I need. When I tried to enable both ISI and ISP in parallel one year ago it did not work though. no /dev/video0 device node was created. I had to remove ISI from device tree in order to make ISP work.

Can you please point me to any example that demonstrate this?

BR, MC

[joanxie]

current bsp already support ISP+ISI in the same time, you can refer to this：

“https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b...

this dts support ov5640 with ISI and basler with ISP in the same time, you can refer to this

[malik_cisse]

Unfortunately this link is not working:

“https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b...

[joanxie]

you can select version on the top right corner, try again

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b..."

[malik_cisse]

obviously dt file could not be sent.

Here the important part:

&i2c3 {
cam1: camera1@1a {
compatible = "sony,imx477";
reg = <0x1a>;
#address-cells = <0x1>;
#size-cells = <0x0>;
clocks = <&cam1_clk>;
clock-names = "ext";
pw-enable-gpios = <&gpio1 14 GPIO_ACTIVE_HIGH>;

port {
cam1_mipi_ep: endpoint {
csi-port = <0>;
bus-width = <4>;
remote-endpoint = <&mipi_csi0_ep>;
bus-type = <4>; /* MIPI CSI-2 D-PHY */
data-lanes = <1 2 3 4>;
clock-lanes = <0>;
clock-noncontinuous = <1>;
link-frequencies =
/bits/ 64 <750000000>;
};
};

};
};

&isi_0 {
status = "disabled";
};

&isi_1 {
status = "okay";
};

&isp_0 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_1 {
status = "okay";
};

[malik_cisse]

Hi Joan,

This time the link you sent works.

However when I implement same device tree as you sent I do not get any image on "/dev/video0" anymore.

When launching gstreamer pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=3840,height=2160 ! queue ! videocrop top=0 left=120 right=120 bottom=0 ! imxvideoconvert_g2d rotation=3 ! kmssink can-scale=false sync=false

I get following error:

root@amrum-fumu1-imx8mp-2:~#
root@imx8mp:~# [ 56.241970] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 105.490628] enter isp_mi_stop

Please find my device tree attached.

I also tried variations of ISI settings (isi_0 = okay) with no success.

It seems some sort of software adaptation needs to be done as well. Do you know what needs to be changed to make ISI work?

Thx, MC

[khang_letruong]

Hi,

I am allowed to share some personal thought on this subject :

you can select version on the top right corner, try again

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b..."

The solution proposed by @joanxie seems to be with 2 independent pipelines :

1. Basler camera --> MIPI0  --> ISP0
2. OV5640 camera --> MIPI1 --> ISI1

Meanwhile, I would guess that @malik_cisse expects to be able to access simultaneously different /dev/videoX and /dev/videoY for the same input camera :

a. IMX477 --> MIPIx  ---> ISPx (bypass the demosiacing for having RAW bayer output on /dev/videoX)
                                  |
                                 +--> ISPy (YUV output on /dev/videoY)

b. IMX477 --> MIPIx  ---> ISIx (RAW bayer output on /dev/videoX)
                                  |
                                 +--> ISPx (YUV output on /dev/videoY)

c. IMX477 --> MIPIx --> ISIx  --> ISPx (YUV output on /dev/videoX)
                                        |
                                        +--> RAW bayer output on /dev/videoY

d. IMX477 --> MIPIx --> ISIx --> ISPy (YUV output on /dev/videoY)
                                        |
                                        + --> RAW bayer output on /dev/videoX

However, I think that neither of above a, b, c, d is feasible.

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:

                     |------------------------> ISI0

CSI2 -------->| (no need any operation of register for this mux)

                     |-------------------------> ISP0

Moreover, I am not sure if the above theory of @dianapredescu (which is same to above case b) works since I haven't seen similar configuration in any reference dts until now.

And the following application note is only limited at single MIPI to dual ISIs, but no info about ISP : https://www.nxp.com/docs/en/application-note/AN13430.pdf

Best Regards,
K.

[malik_cisse]

Thank you Khang for bringing this to the point so necelly.
This is exactly what I am trying to do. Any of the scenarios you have pointed out would unlock my issue if they would work.
I can also imagine that many people will have similar issue.

@dianapredescu: Your statement saying that "CSI2 channel can output to ISI and ISP at the same time" is indeed very interesting but as Khang also pointed out there is no actual proof this is feasible. At least it is not feasible by device tree tweaking only. 

Any actual working examples would really be appreciated.

BR, MC

[dianapredescu]

@malik_cisse ,

The only thing I tested myself personally was:

                                   |------------------------>  ISI0

CSI2 ---> ISP0-------->|   

                                   |-------------------------> ISI1

But that was possible because ISI can have as input CSI2 or AXI Memory, and used the m2m functionality of ISI.

The scenarios @khang_letruong (btw thanks for explaining the use case) mentioned, indeed not sure if are feasible because looking at the 8MP RM it is stated: 

There are two instances of ISP on the chip and each is connected to separate instances of
MIPI CSI.

So my understanding would be that I can't have the ISP input coming from anywhere else but camera sensor.

But let me look into the possibility I mentioned few messages above to see if it actually works and if we have some examples.

Regards,

Diana

[khang_letruong]

Dear @dianapredescu ,

The only thing I tested myself personally was:

                                   |------------------------> ISI0

CSI2 ---> ISP0-------->|

                                   |-------------------------> ISI1

Are you sure that the above configuration works (i.e hooking the output of ISP to ISI) since it seems to be impossible within the following discussion :  https://community.nxp.com/t5/i-MX-Processors/IMX8MP-use-ISI-with-ISP/m-p/1376237

Can you confirm ?

Thanks and regards,
K.

[dianapredescu]

Hi @khang_letruong,

I did some experiments few days ago, so here is what I have noticed:

I edited a bit imx8mp.dtsi file to configure isi_1 as m2m device too. I modified imx8mp-evk-basler-ov5640.dtb to enable isi_1, isi_0 and isp_0. After booting the board I am able to find the Gstreamer plugin for m2m:

root@imx8mpevk:~# gst-inspect-1.0 | grep v4l2
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2convert: V4L2 Video Converter
video4linux2: v4l2video3convert: V4L2 Video Converter
video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h265enc: V4L2 H.265 Encoder
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h265dec: V4L2 H265 Decoder
video4linux2: v4l2vp8dec: V4L2 VP8 Decoder
video4linux2: v4l2vp9dec: V4L2 VP9 Decoder

as well as the m2m_devices:

root@imx8mpevk:~# v4l2-ctl --list-devices

mxc-isi-m2m (platform:32e00000.isi:m2m_devic):
/dev/video2

mxc-isi-m2m (platform:32e02000.isi:m2m_devic):
/dev/video3

VIV (platform:viv0):
/dev/video5

So I created a quick pipeline and redirected isp_0 to both isi_0 and isi_1.

gst-launch-1.0 v4l2src device=/dev/video5 num-buffers=20 ! video/x-raw, width=1920, height=1080, framerate=60/1 ! tee name=t ! queue ! v4l2convert ! video/x-raw, width=1920, height=1080, format=RGB16 ! vpuenc_h264 ! queue ! filesink location=out1.h264 t. ! queue ! v4l2video3convert ! video/x-raw, width=1280, height=720, format=BGRA ! vpuenc_h264 ! queue ! filesink location=out2.h264

In my case v4l2convert will use /dev/video2 which is mapped to isi_0 and v4l2video3convert will use /dev/video3 which corresponds to isi_1. Used the tee element which splits the data in 2 identical ways, so at the end I have 1920x1080 and 1280x720 video streams.

Let me know if this is helpful.

Regards,

Diana

[malik_cisse]

Hi Diana,

This is getting very interesting.

I learned something in your and Khang's feedback: One can configure ISI to do m2m access as can be seen in https://www.nxp.com/docs/en/application-note/AN13430.pdf page 3

isi interface = <5 0 2>.

However your example is not useful for two reasons:

1. you are using isi0 and isi1 to convert YUV to RGB just before encoding in H.264. This does not make sense because H.264 encoder uses YUV420 input already.
2. The imx8mp GPU can do yuv to RGB conversion. No need for isi for this

For me isi only makes sense when connected to CSI directly but unfortunately we cannot seem to connect both isi and isp to CSI at the same time as in your earlier post:

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

[dianapredescu · Accepted solution]

@malik_cisse ,

Thanks for the feedback. I didn't say this was an optimal solution and I can understand now it doesn't fit your needs. It was just a particular case asked by a customer (it was used for capturing camera frames with RAW or YUV format through ISP interface , when the end applications only accepted RGB format with the request to avoid transforming the image, from YUV to RGB, via SW or GPU2D) and the only custom ISP/ISI use case I have tested personally until now, apart from what we already have enabled by default. 

I will come back if I'll find anything related to this thread.

Regards,

Diana

[khang_letruong]

Dear @dianapredescu 

Thanks for the confirmation. But still data behind ISP cannot be read, and this configuration seems "heavier" than one proposed in this application note https://www.nxp.com/docs/en/application-note/AN13430.pdf  with intermediate ISP btw MIPI-CSI2 and ISI(s).

Regards,

Khang