IMX8MP use both ISI and ISP

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

IMX8MP use both ISI and ISP

Jump to solution
5,038 Views
malik_cisse
Senior Contributor I

Hi,

Is it actually possible to have both enabled, ISI and ISP?
My use case:

Raw bayer CSI camera ->ISI->ISP-> preview on screen

ISI-> Software ISP-> 12bit TIFF snapshots

This is to enable 12bit high resolution snapshots without stopping live preview.

BR, MC

Labels (1)
0 Kudos
1 Solution
4,832 Views
dianapredescu
NXP Employee
NXP Employee

@malik_cisse ,

Thanks for the feedback. I didn't say this was an optimal solution and I can understand now it doesn't fit your needs. It was just a particular case asked by a customer (it was used for capturing camera frames with RAW or YUV format through ISP interface , when the end applications only accepted RGB format with the request to avoid transforming the image, from YUV to RGB, via SW or GPU2D) and the only custom ISP/ISI use case I have tested personally until now, apart from what we already have enabled by default. 

I will come back if I'll find anything related to this thread.

Regards,

Diana

View solution in original post

24 Replies
3,733 Views
malik_cisse
Senior Contributor I

Hi,
I am doing 4K (3840x2160) using ISI.

0 Kudos
3,724 Views
billie
Contributor I

Hi,

 

Thanks for your answer.

I'd like to use 12MP image sensor through the ISI.

Is there any problem ?

 

Regards,

Billie.

0 Kudos
4,775 Views
dianapredescu
NXP Employee
NXP Employee

Hi Malik,

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

 

Regards,

Diana

0 Kudos
2,956 Views
malik_cisse
Senior Contributor I

Hi Diana,

This is a late feedback on this topic but I wanted to confirm that your scenario indeed works!

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

I could demonstrate this on latest ISP version 20 using Kirkstone BSP. 
The only thing to do to prevent resource mismatches was to implement  the imx8-mipi-csi2-sam.c and vvcam driver as described in https://www.nxp.com/docs/en/application-note/AN13430.pdf.
on page 6.
The device tree patches are not necessary. Only page 6 stuff above.
Thank you again for showing this is actually feasible. Otherwise I would not even have tried it I guess.
Now I have a bullet proof multimedia application based on imx8mp.

4,769 Views
malik_cisse
Senior Contributor I

Hi Diana,

Thanks for the hint although I knew that already.

My question is whether I can connect ISI to ISP and use both outputs simultaneously.

Thx, MC

0 Kudos
4,758 Views
joanxie
NXP TechSupport
NXP TechSupport

you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

0 Kudos
4,751 Views
malik_cisse
Senior Contributor I

Hi Joanxie,

>>you can use ISI and ISP in the same time, don't need connect ISI to ISP, you can use ISI to raw data output and ISP to display

Awesome! Well this is exactly what I need. When I tried to enable both ISI and ISP in parallel one year ago it did not work though. no /dev/video0 device node was created. I had to remove ISI from device tree in order to make ISP work.

Can you please point me to any example that demonstrate this?

BR, MC

0 Kudos
4,725 Views
joanxie
NXP TechSupport
NXP TechSupport

current bsp already support ISP+ISI in the same time, you can refer to this:

https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b...

this dts support ov5640 with ISI and basler with ISP in the same time, you can refer to this

 

0 Kudos
4,714 Views
malik_cisse
Senior Contributor I
0 Kudos
4,703 Views
joanxie
NXP TechSupport
NXP TechSupport
0 Kudos
4,638 Views
malik_cisse
Senior Contributor I

obviously dt file could not be sent.

Here the important part:

&i2c3 {
cam1: camera1@1a {
compatible = "sony,imx477";
reg = <0x1a>;
#address-cells = <0x1>;
#size-cells = <0x0>;
clocks = <&cam1_clk>;
clock-names = "ext";
pw-enable-gpios = <&gpio1 14 GPIO_ACTIVE_HIGH>;

port {
cam1_mipi_ep: endpoint {
csi-port = <0>;
bus-width = <4>;
remote-endpoint = <&mipi_csi0_ep>;
bus-type = <4>; /* MIPI CSI-2 D-PHY */
data-lanes = <1 2 3 4>;
clock-lanes = <0>;
clock-noncontinuous = <1>;
link-frequencies =
/bits/ 64 <750000000>;
};
};

};
};

&isi_0 {
status = "disabled";
};


&isi_1 {
status = "okay";
};

&isp_0 {
status = "okay";
};

&dewarp {
status = "okay";
};

&mipi_csi_1 {
status = "okay";
};

0 Kudos
4,662 Views
malik_cisse
Senior Contributor I

Hi Joan,

This time the link you sent works.

However when I implement same device tree as you sent I do not get any image on "/dev/video0" anymore.

When launching gstreamer pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=3840,height=2160 ! queue ! videocrop top=0 left=120 right=120 bottom=0 ! imxvideoconvert_g2d rotation=3 ! kmssink can-scale=false sync=false

I get following error:

root@amrum-fumu1-imx8mp-2:~#
root@imx8mp:~# [ 56.241970] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 105.490628] enter isp_mi_stop

Please find my device tree attached.

I also tried variations of ISI settings (isi_0 = okay) with no success.

It seems some sort of software adaptation needs to be done as well. Do you know what needs to be changed to make ISI work?

Thx, MC

0 Kudos
4,651 Views
khang_letruong
Senior Contributor III

Hi,

I am allowed to share some personal thought on this subject :

you can select version on the top right corner, try again

"https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8mp-evk-b..."

The solution proposed by @joanxie seems to be with 2 independent pipelines :

1. Basler camera --> MIPI0  --> ISP0
2. OV5640 camera --> MIPI1 --> ISI1

Meanwhile, I would guess that @malik_cisse expects to be able to access simultaneously different /dev/videoX and /dev/videoY for the same input camera :

a. IMX477 --> MIPIx  ---> ISPx (bypass the demosiacing for having RAW bayer output on /dev/videoX)
                                  |
                                 +--> ISPy (YUV output on /dev/videoY)

b. IMX477 --> MIPIx  ---> ISIx (RAW bayer output on /dev/videoX)
                                  |
                                 +--> ISPx (YUV output on /dev/videoY)

c. IMX477 --> MIPIx --> ISIx  --> ISPx (YUV output on /dev/videoX)
                                        |
                                        +--> RAW bayer output on /dev/videoY

d. IMX477 --> MIPIx --> ISIx --> ISPy (YUV output on /dev/videoY)
                                        |
                                        + --> RAW bayer output on /dev/videoX

However, I think that neither of above a, b, c, d is feasible.

not sure if it will answer your question, but in theory one CSI2 channel can output to ISI and ISP at the same time. It should look like this:

                     |------------------------> ISI0

CSI2 -------->| (no need any operation of register for this mux)

                     |-------------------------> ISP0

Moreover, I am not sure if the above theory of @dianapredescu (which is same to above case b) works since I haven't seen similar configuration in any reference dts until now.

And the following application note is only limited at single MIPI to dual ISIs, but no info about ISP : https://www.nxp.com/docs/en/application-note/AN13430.pdf

Best Regards,
K.

0 Kudos
4,640 Views
malik_cisse
Senior Contributor I

Thank you Khang for bringing this to the point so necelly.
This is exactly what I am trying to do. Any of the scenarios you have pointed out would unlock my issue if they would work.
I can also imagine that many people will have similar issue.

@dianapredescu: Your statement saying that "CSI2 channel can output to ISI and ISP at the same time" is indeed very interesting but as Khang also pointed out there is no actual proof this is feasible. At least it is not feasible by device tree tweaking only. 

Any actual working examples would really be appreciated.

BR, MC

0 Kudos
4,622 Views
dianapredescu
NXP Employee
NXP Employee

@malik_cisse ,

The only thing I tested myself personally was:

                                   |------------------------>  ISI0

CSI2 ---> ISP0-------->|   

                                   |-------------------------> ISI1

But that was possible because ISI can have as input CSI2 or AXI Memory, and used the m2m functionality of ISI.

The scenarios @khang_letruong (btw thanks for explaining the use case) mentioned, indeed not sure if are feasible because looking at the 8MP RM it is stated: 

There are two instances of ISP on the chip and each is connected to separate instances of
MIPI CSI.

So my understanding would be that I can't have the ISP input coming from anywhere else but camera sensor.

But let me look into the possibility I mentioned few messages above to see if it actually works and if we have some examples.

Regards,

Diana

0 Kudos
4,615 Views
khang_letruong
Senior Contributor III

Dear @dianapredescu ,

The only thing I tested myself personally was:

                                   |------------------------> ISI0

CSI2 ---> ISP0-------->|

                                   |-------------------------> ISI1

Are you sure that the above configuration works (i.e hooking the output of ISP to ISI) since it seems to be impossible within the following discussion :  https://community.nxp.com/t5/i-MX-Processors/IMX8MP-use-ISI-with-ISP/m-p/1376237

Can you confirm ?

Thanks and regards,
K.

0 Kudos
4,610 Views
dianapredescu
NXP Employee
NXP Employee

Hi @khang_letruong,

I did some experiments few days ago, so here is what I have noticed:

I edited a bit imx8mp.dtsi file to configure isi_1 as m2m device too. I modified imx8mp-evk-basler-ov5640.dtb to enable isi_1, isi_0 and isp_0. After booting the board I am able to find the Gstreamer plugin for m2m:

root@imx8mpevk:~# gst-inspect-1.0 | grep v4l2
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2convert: V4L2 Video Converter
video4linux2: v4l2video3convert: V4L2 Video Converter
video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h265enc: V4L2 H.265 Encoder
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h265dec: V4L2 H265 Decoder
video4linux2: v4l2vp8dec: V4L2 VP8 Decoder
video4linux2: v4l2vp9dec: V4L2 VP9 Decoder

as well as the m2m_devices:

root@imx8mpevk:~# v4l2-ctl --list-devices

mxc-isi-m2m (platform:32e00000.isi:m2m_devic):
/dev/video2

mxc-isi-m2m (platform:32e02000.isi:m2m_devic):
/dev/video3

VIV (platform:viv0):
/dev/video5

So I created a quick pipeline and redirected isp_0 to both isi_0 and isi_1.

gst-launch-1.0 v4l2src device=/dev/video5 num-buffers=20 ! video/x-raw, width=1920, height=1080, framerate=60/1 ! tee name=t ! queue ! v4l2convert ! video/x-raw, width=1920, height=1080, format=RGB16 ! vpuenc_h264 ! queue ! filesink location=out1.h264 t. ! queue ! v4l2video3convert ! video/x-raw, width=1280, height=720, format=BGRA ! vpuenc_h264 ! queue ! filesink location=out2.h264

In my case v4l2convert will use /dev/video2 which is mapped to isi_0 and v4l2video3convert will use /dev/video3 which corresponds to isi_1. Used the tee element which splits the data in 2 identical ways, so at the end I have 1920x1080 and 1280x720 video streams.

Let me know if this is helpful.

Regards,

Diana

0 Kudos
4,594 Views
malik_cisse
Senior Contributor I

Hi Diana,

This is getting very interesting.

I learned something in your and Khang's feedback: One can configure ISI to do m2m access as can be seen in https://www.nxp.com/docs/en/application-note/AN13430.pdf page 3

isi interface = <5 0 2>.

isi_m2m.jpg

However your example is not useful for two reasons:

  1. you are using isi0 and isi1 to convert YUV to RGB just before encoding in H.264. This does not make sense because H.264 encoder uses YUV420 input already.
  2. The imx8mp GPU can do yuv to RGB conversion. No need for isi for this

For me isi only makes sense when connected to CSI directly but unfortunately we cannot seem to connect both isi and isp to CSI at the same time as in your earlier post:

                     |------------------------>  ISI0

CSI2 -------->|   (no need any operation of register for this mux)

                     |-------------------------> ISP0

0 Kudos
4,833 Views
dianapredescu
NXP Employee
NXP Employee

@malik_cisse ,

Thanks for the feedback. I didn't say this was an optimal solution and I can understand now it doesn't fit your needs. It was just a particular case asked by a customer (it was used for capturing camera frames with RAW or YUV format through ISP interface , when the end applications only accepted RGB format with the request to avoid transforming the image, from YUV to RGB, via SW or GPU2D) and the only custom ISP/ISI use case I have tested personally until now, apart from what we already have enabled by default. 

I will come back if I'll find anything related to this thread.

Regards,

Diana

4,602 Views
khang_letruong
Senior Contributor III

Dear @dianapredescu 

Thanks for the confirmation. But still data behind ISP cannot be read, and this configuration seems "heavier" than one proposed in this application note https://www.nxp.com/docs/en/application-note/AN13430.pdf  with intermediate ISP btw MIPI-CSI2 and ISI(s).

Regards,

Khang

 

0 Kudos