ISP 1 only works when ISP 0 is enabled. Why?

[pwhewell]

I have two separate setups

Setup 1:

1. basler -> MIPI0 -> ISP0
2. mt9034 -> MIPI1 -> ISI1

Setup 2:

1. mt9034 -> MIPI0 -> ISI0
2. basler -> MIPI1 -> ISP1

mt9034 is just a stand in for any camera and is not my concern.

v4l2-ctl -d /dev/video3 --stream-mmap=4 --verbose
VIDIOC_QUERYCAP: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 1512.118705 delta: 1512118.705 ms (ts-monotonic, ts-src-eof)

But when i try the same exact setup with Setup 2 i get errors right away

v4l2-ctl -d /dev/video3 --stream-mmap=4 --verbose
VIDIOC_QUERYCAP: ok
                VIDIOC_REQBUFS returned -1 (Resource temporarily unavailable)

media pipelines and v4l2-ctl output for Setup 2:

# media-ctl -p
Media controller API version 5.10.72

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial
bus info
hw revision     0x0
driver version  5.10.72

Device topology
- entity 1: mxc_isi.0 (16 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
        pad0: Sink
                <- "mxc-mipi-csi2.0":4 [ENABLED]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Sink
        pad5: Sink
        pad6: Sink
        pad7: Sink
        pad8: Sink
        pad9: Sink
        pad10: Sink
        pad11: Sink
        pad12: Source
                -> "mxc_isi.0.capture":0 [ENABLED]
        pad13: Source
        pad14: Source
        pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev0
        pad0: Sink
                <- "mt9034 4-0010":0 [ENABLED,IMMUTABLE]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Source
                -> "mxc_isi.0":0 [ENABLED]
        pad5: Source
        pad6: Source
        pad7: Source

- entity 31: mxc-mipi-csi2.1 (8 pads, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                <- "basler-camera-vvcam 5-0036":0 [ENABLED,IMMUTABLE]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Source
        pad5: Source
        pad6: Source
        pad7: Source

- entity 40: basler-camera-vvcam 5-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev2
        pad0: Source
                -> "mxc-mipi-csi2.1":0 [ENABLED,IMMUTABLE]

- entity 42: mt9034 4-0010 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:UYVY8_2X8/640x720 field:none colorspace:srgb
                 crop:(0,0)/640x720]
                -> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]
				
# media-ctl -p -d /dev/media1
Media controller API version 5.10.72

Media device information
------------------------
driver          vvcam-video
model           viv_media
serial
bus info
hw revision     0x0
driver version  5.10.72

Device topology
- entity 1: viv_v4l21 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video3
        pad0: Sink
                <- "vvcam-isp.1":0 [ENABLED]

- entity 5: vvcam-isp.1 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/v4l-subdev4
        pad0: Source
                -> "viv_v4l21":0 [ENABLED]

- entity 7: vvcam-dwe.1 (2 pads, 0 link)
            type Node subtype V4L flags 0
            device node name /dev/v4l-subdev5
        pad0: Source
        pad1: Sink
		

# v4l2-ctl -d /dev/video3 -D
Driver Info:
        Driver name      : viv_v4l2_device
        Card type        : VIV
        Bus info         : platform:viv1
        Driver version   : 5.10.72
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : vvcam-video
        Model            : viv_media
        Serial           :
        Bus info         :
        Media version    : 5.10.72
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.10.72
Interface Info:
        ID               : 0x03000003
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : viv_v4l21
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Sink, Must Connect
          Link 0x0200000a: from remote pad 0x1000009 of entity 'vvcam-isp.1': Data, Enabled
		  
# v4l2-ctl --list-devices
[  349.939100] enter isp_mi_stop
 ():
        /dev/v4l-subdev0

 ():
        /dev/v4l-subdev1
        /dev/v4l-subdev4
        /dev/v4l-subdev5

 (csi1):
        /dev/v4l-subdev2

mxc-isi-cap (platform:32e00000.isi:cap_devic):
        /dev/video2

FSL Capture Media Device (platform:mxc-md):
        /dev/media0

VIV (platform:viv1):
        /dev/video3

vsi_v4l2dec (platform:vsi_v4l2dec):
        /dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
        /dev/video0

viv_media (platform:vvcam-video):
        /dev/media1

I do notice in dmesg a difference between the two setups. When i try to stream during setup 2 using the v4l2 command above i see this output every time. It does not appear in setup 1.

[ 1481.632505] viv_post_event: unsubscribed event id =2 type=0x08002000
[ 1482.584451] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1482.748439] viv_post_event: unsubscribed event id =14 type=0x08002000
[ 1482.924447] viv_post_event: unsubscribed event id =1 type=0x08002000

as far as i can tell the isp is started successfully

Oct 26 18:24:35 ucm-imx8m-plus systemd[1]: Started ISP i.MX 8Mplus daemon.
Oct 26 18:24:35 ucm-imx8m-plus start_isp.sh[1458]: Starting isp_media_server
Oct 26 18:24:35 ucm-imx8m-plus start_isp.sh[1458]: DAA3840_30MC_266 on ISP 1
Oct 26 18:24:35 ucm-imx8m-plus start_isp.sh[1458]: RUN_SCRIPT=/opt/imx8-isp/bin/run.sh
Oct 26 18:24:35 ucm-imx8m-plus start_isp.sh[1458]: RUN_SCRIPT_PATH=/opt/imx8-isp/bin
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Apply configuration  DAA3840_30MC_266...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loading module basler-camera-driver-vvcam ...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: basler_camera_driver_vvcam
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1538]: basler_camera_driver_vvcam    20480  0
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: basler-camera-driver-vvcam already loaded.
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loading module imx8-media-dev ...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: imx8_media_dev
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loaded /lib/modules/5.10.72/kernel/drivers/staging/media/imx/imx8-media-dev.ko
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loading module vvcam-video ...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam_video
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1562]: vvcam_video            36864  0
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam-video already loaded.
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loading module vvcam-dwe ...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam_dwe
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1567]: vvcam_dwe              28672  1
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam-dwe already loaded.
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Loading module vvcam-isp ...
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam_isp
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1572]: vvcam_isp              65536  1
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: vvcam-isp already loaded.
Oct 26 18:24:36 ucm-imx8m-plus start_isp.sh[1458]: Starting isp_media_server with configuration CAMERA1

[chp0423]

@pwhewell Did you finally find the reason of errors in setup2? I am tesing a single sensor setup, but had similar issue with your setup2: VIDIOC_REQBUFS returned -1. Thanks very much!

[pwhewell]

No i did not find a solution. As far as I can tell, ISP1 only works when ISP0 is also enabled.

Others at my company, and also a contractor we had working on another project all couldn't figure out the reasoning behind it. As you can tell from the ticket, NXP hasn't really been super helpful with understanding why either. They think its possible. but they have no example to try and don't recommend it. We went the route of just requiring our customers to configure their setup to be camera 0 when needing a single ISP.

[pwhewell]

Alright,

Further testing shows that for ISP_1 to work I NEED to have ISP_0 enabled. Does anyone at NXP know why this behavior is this way? Is this expected? There is no mention of this in the reference manual so I assume this is not supposed to be this way?

I used a device tree with both ISPs enabled and i was able to get video out of ISP_1. All i did was disable the following:

• isp_0
• mipi_csi_0
• mipi_csi_0 endpoint
• the camera node on the i2c bus that corresponds to mipi_csi_0 (eg. imx219_mipi1)

and then i couldn't get video. After re-enabling them, it worked again.

[pwhewell]

tried using the video test application to help debug possibly.
ISP_LOG_LEVEL=8.
Setup 1 (successful):

> ./video_test -w 1920 -h 1080 -f NV12 -t file -m 1 -d3
INFO   : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO   : [VIDEOTEST]   Driver: viv_v4l2_device
INFO   : [VIDEOTEST] found viv video dev /dev/video3
INFO   : [VIDEOTEST] openDevice Control User Controls
INFO   : [VIDEOTEST] openDevice Control viv_ext_ctrl
INFO   : [VIDEOTEST] openDevice, find viv ctrl id 0x98f901
INFO   : [VIDEOTEST] open camdev ret : 3
INFO   : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO   : [VIDEOTEST]   Driver: viv_v4l2_device
INFO   : [VIDEOTEST]   Card: VIV
INFO   : [VIDEOTEST]   Version: 5.10.72
INFO   : [VIDEOTEST]   All Caps: 84200001
INFO   : [VIDEOTEST]   Dev Caps: 04200001
Video Test:caps supports:{
        count = 5
        {
        index    = 0
        width    = 3840
        height   = 2160
        fps      = 30
        hdr_mode = 0
        }
        {
        index    = 1
        width    = 1920
        height   = 1080
        fps      = 60
        hdr_mode = 0
        }
        {
        index    = 2
        width    = 3840
        height   = 2160
        fps      = 30
        hdr_mode = 0
        }
        {
        index    = 3
        width    = 1920
        height   = 1080
        fps      = 30
        hdr_mode = 0
        }
        {
        index    = 4
        width    = 1920
        height   = 1080
        fps      = 25
        hdr_mode = 0
        }
}
INFO   : [VIDEOTEST] Video Test: Set mode[1]

INFO   : [VIDEOTEST]    0: YUYV 4:2:2 0x56595559 0x0
INFO   : [VIDEOTEST]    1: Y/CbCr 4:2:0 0x3231564E 0x0
INFO   : [VIDEOTEST]    2: Y/CbCr 4:2:2 0x3631564E 0x0
INFO   : [VIDEOTEST]    3: 12-bit Bayer GRGR/BGBG 0x32314142 0x0
INFO   : [VIDEOTEST]
Enum Format[NV12]Framesize:
INFO   : [VIDEOTEST] Enum StepWise FrameSize:min{width:176 height:144}max{width:3840 height:2160}stepsize{width:16 height:8}

Setup 2 (failed):

> ./video_test -w 1920 -h 1080 -f NV12 -t file -m 1 -d3
INFO   : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO   : [VIDEOTEST]   Driver: viv_v4l2_device
INFO   : [VIDEOTEST] found viv video dev /dev/video3
INFO   : [VIDEOTEST] openDevice Control User Controls
INFO   : [VIDEOTEST] openDevice Control viv_ext_ctrl
INFO   : [VIDEOTEST] openDevice, find viv ctrl id 0x98f901
INFO   : [VIDEOTEST] open camdev ret : 3
INFO   : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO   : [VIDEOTEST]   Driver: viv_v4l2_device
INFO   : [VIDEOTEST]   Card: VIV
INFO   : [VIDEOTEST]   Version: 5.10.72
INFO   : [VIDEOTEST]   All Caps: 84200001
INFO   : [VIDEOTEST]   Dev Caps: 04200001
ERROR  : [VIDEOTEST] Video Test:Get Caps Supports Failed[-1]

[pwhewell]

Further help. I compiled video_test and added errno reporting myself. in the case of setup 2:

> ./video_test -w 1920 -h 1080 -f NV12 -t file -m 1 -d3
INFO : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO : [VIDEOTEST] Driver: viv_v4l2_device
INFO : [VIDEOTEST] found viv video dev /dev/video3
INFO : [VIDEOTEST] openDevice Control User Controls
INFO : [VIDEOTEST] openDevice Control viv_ext_ctrl
INFO : [VIDEOTEST] openDevice, find viv ctrl id 0x98f901
INFO : [VIDEOTEST] open camdev ret : 3
INFO : [VIDEOTEST] Open Device: /dev/video3 (fd=3)
INFO : [VIDEOTEST] Driver: viv_v4l2_device
INFO : [VIDEOTEST] Card: VIV
INFO : [VIDEOTEST] Version: 5.10.72
INFO : [VIDEOTEST] All Caps: 84200001
INFO : [VIDEOTEST] Dev Caps: 04200001
ERROR : [VIDEOTEST] Video Test:Get Caps Supports Failed[-1] errno=110 Connection timed out

[joanxie]

you can refer to the link as below, you can try to change the configure file

"https://community.nxp.com/t5/i-MX-Processors/iMX8MP-with-2-cameras-using-ISP/m-p/1393554"

[pwhewell]

Maybe i'm misunderstanding something? What in that link am i supposed to look at?

I've already verified that ISP1 works when i'm using both ISPs. It only starts to fail when i'm no longer using ISP0.

This has now also been verified by someone else at another company. Is there some trick to getting ISP1 working differently from when ISP0 is used vs not being used?

[joanxie]

I checked the ISP1 design, I think you can use this ISP1 alone, but we don't have such dtb for isp1 only, because we don't suggest customer to use this, ISP1 is only 266Mhz, couldn't support 4K, so if customer only use one ISP, we suggest them to use ISP0, you can use ISI1+ISP0, I don't suggest that you spent time on debugging isp1, because ISP1 isn't power than ISP0, suggest that you can use ISP0 for your project

[Egor-G]

Hello! I guess I faced with similar issue. Can you clarify - may I use ISP0 with CSI2 (my camera connected to second CSI interface) or CSI1 - > ISP0 , CSI2 - > ISP1 only?