Trouble with (TC358748) Parallel to CSI2 video bridge

[shaswath]

Hello,

I am working on a project that requires interfacing an FPGA to our custom i.MX8MP board. I am trying to modify the tc358746 (Parallel -> CSI2) driver to support for our yocto lf-5.15.71 version.

I want the video pipeline (FPGA (YUV4:2:2) -> bridge (YUV4:2:2/4-lane) -> mipi_csi0 -> isi0) to output 1920x1080@30Hz with YUV4:2:2 format for now but I am unable to capture any frames using Gstreamer or v4l2-ctl.

The v4l-subdev1 won't show up in the v4l2 device list

Please find the media-ctl output and the dmesg log for the bridge, Any insights on troubleshooting the issue is greatly appreciated.

root@:~$ media-ctl -p
Media controller API version 5.15.71

Media device information
------------------------
driver          mxc-md
model           FSL Capture Media Device
serial          
bus info        
hw revision     0x0
driver version  5.15.71

Device topology
- entity 1: mxc_isi.0 (16 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
        pad0: Sink
                <- "mxc-mipi-csi2.0":4 [ENABLED]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Sink
        pad5: Sink
        pad6: Sink
        pad7: Sink
        pad8: Sink
        pad9: Sink
        pad10: Sink
        pad11: Sink
        pad12: Source
                -> "mxc_isi.0.capture":0 [ENABLED]
        pad13: Source
        pad14: Source
        pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev0
        pad0: Sink
                <- "tc358748 2-000e":0 [ENABLED,IMMUTABLE]
        pad1: Sink
        pad2: Sink
        pad3: Sink
        pad4: Source
                -> "mxc_isi.0":0 [ENABLED]
        pad5: Source
        pad6: Source
        pad7: Source

- entity 31: tc358748 2-000e (1 pad, 1 link)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [fmt:UYVY8_2X8/1920x1080@1/30 field:none colorspace:srgb]
                -> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

root@:~$ v4l2-ctl --list-devices
 ():
        /dev/v4l-subdev0

mxc-isi-cap (platform:32e00000.isi:cap_devic):
        /dev/video2

FSL Capture Media Device (platform:mxc-md):
        /dev/media0

vsi_v4l2dec (platform:vsi_v4l2dec):
        /dev/video1

vsi_v4l2enc (platform:vsi_v4l2enc):
        /dev/video0

root@:~$ v4l2-ctl --device /dev/video2 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture Multiplanar

    [3]: 'YUYV' (YUYV 4:2:2)
         Size: Discrete 1920x1080
         Interval: Discrete 0.033s (30.000 fps)
         Interval: Discrete 0.033s (30.000 fps)

root@:~$ v4l2-ctl -d /dev/video2 --all
Driver Info:
        Driver name      : mxc-isi-cap
        Card type        : mxc-isi-cap
        Bus info         : platform:32e00000.isi:cap_devic
        Driver version   : 5.15.71
        Capabilities     : 0x84201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : mxc-md
        Model            : FSL Capture Media Device
        Serial           : 
        Bus info         : 
        Media version    : 5.15.71
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.15.71
Interface Info:
        ID               : 0x03000014
        Type             : V4L Video
Entity Info:
        ID               : 0x00000012 (18)
        Name             : mxc_isi.0.capture
        Function         : V4L2 I/O
        Pad 0x01000013   : 0: Sink
          Link 0x02000021: from remote pad 0x100000e of entity 'mxc_isi.0' (Video Pixel Formatter): Data, Enabled
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 1920/1080
        Pixel Format      : 'RGBP' (16-bit RGB 5-6-5)
        Field             : None
        Number of planes  : 1
        Flags             : 
        Colorspace        : sRGB
        Transfer Function : Default
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 3840
           Size Image     : 4147200
Crop Capability Video Output:
        Bounds      : Left 0, Top 0, Width 1920, Height 1080
        Default     : Left 0, Top 0, Width 1920, Height 1080
        Pixel Aspect: 1/1
Selection Video Output: compose, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Selection Video Output: compose_default, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Selection Video Output: compose_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags: 
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0
                alpha_component 0x00980929 (int)    : min=0 max=255 step=1 default=0 value=0

[ 15.547696] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 26.966015] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
.
.
.
[ 26.968164] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 26.968181] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 26.968195] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 26.968210] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 26.971188] mxc_isi.0: Not match format, set default
[ 26.971199] mxc_isi.0: Not match format, set default
[ 27.305025] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.305150] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.305352] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.305392] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.305442] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.305507] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.309455] enter tc358748_s_power = ON
[ 27.309471] tc358748 2-000e: tc358748_enable_hw
[ 27.311682] tc358748 2-000e: tc358748_set_pad_format
[ 27.311696] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.311705] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.316716] bypass csc
[ 27.316724] input fmt YUV4
[ 27.316728] output fmt YUYV
[ 27.667770] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.667876] tc358748 2-000e: tc358748_set_pad_format
[ 27.667882] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.667889] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.672900] bypass csc
[ 27.672909] input fmt YUV4
[ 27.672912] output fmt YUYV
[ 27.672922] enter tc358748_s_power = OFF
[ 27.672929] tc358748 2-000e: tc358748_disable_hw
[ 27.701511] enter tc358748_s_power = ON
[ 27.701530] tc358748 2-000e: tc358748_enable_hw
[ 27.703406] tc358748 2-000e: tc358748_set_pad_format
[ 27.703418] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.703427] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 27.708440] bypass csc
[ 27.708451] input fmt YUV4
[ 27.708456] output fmt YUYV
[ 28.035292] tc358748 2-000e: [tc358748_stream,1152]: Stream enable ...
[ 28.035309] tc358748 2-000e: tc358748_set_pll
[ 28.035313] tc358748 2-000e: tc358748_calc_pll
[ 28.035382] tc358748 2-000e: PLL: refclk 10000000, fbd 88, prd 0, frs 1
[ 28.035388] tc358748 2-000e: PLL: pll_clk: 445000000, CSIClk 222500000, CSIByteClk 55625000
[ 28.035393] tc358748 2-000e: PLL: pclk 74166666 (FPGA(Hardcode=>: 74250000MHz))
[ 28.037869] tc358748 2-000e: tc358748_sw_reset
[ 28.038566] tc358748 2-000e: tc358748_apply_misc_config
[ 28.038902] tc358748 2-000e: DATAFMT: 0x60
[ 28.039301] tc358748 2-000e: CONFCTL[PDATAF]: 0x0
[ 28.039642] tc358748 2-000e: FIFOCTL: 0x20
[ 28.039976] tc358748 2-000e: WORDCNT: 0xf00
[ 28.039984] tc358748 2-000e: tc358748_enable_csi_lanes: 1
[ 28.040454] tc358748 2-000e: CLW_CNTRL: 0 0x140
[ 28.040777] tc358748 2-000e: D0W_CNTRL: 0 0x144
[ 28.041158] tc358748 2-000e: D1W_CNTRL: 0 0x148
[ 28.041538] tc358748 2-000e: D2W_CNTRL: 0 0x14c
[ 28.041899] tc358748 2-000e: D3W_CNTRL: 0 0x150
[ 28.041908] tc358748 2-000e: tc358748_apply_dphy_config
[ 28.041912] tc358748 2-000e: dsiclk_nsk: 4494
[ 28.041916] tc358748 2-000e: ui_nsk: 2247
[ 28.041920] tc358748 2-000e: csibclk_nsk: 17977
[ 28.041924] tc358748 2-000e: phy_delay_nsk: 26965
[ 28.042278] tc358748 2-000e: LINEINITCNT: 0x15ba
[ 28.042661] tc358748 2-000e: LPTXTIMECNT: 0x2
[ 28.043043] tc358748 2-000e: TCLK_HEADERCNT: 0xa03
[ 28.043456] tc358748 2-000e: TCLK_TRAILCNT: 0xffffffff
[ 28.043803] tc358748 2-000e: THS_HEADERCNT: 0xffffee03
[ 28.044156] tc358748 2-000e: TWAKEUP: 0x49e0
[ 28.044535] tc358748 2-000e: TCLK_POSTCNT: 0x7
[ 28.044914] tc358748 2-000e: THS_TRAILCNT: 0x1
[ 28.045298] tc358748 2-000e: HSTXVREGEN: 0x1f
[ 28.045307] tc358748 2-000e: tc358748_enable_csi_module: 1
[ 28.045680] tc358748 2-000e: STARTCNTRL: 0x1
[ 28.046025] tc358748 2-000e: CSI_START: 0x1
[ 28.046407] tc358748 2-000e: CSI_CONFW_REG: 2734719110 (0x518)
[ 32.176687] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 32.176772] tc358748 2-000e: tc358748_get_pad_format code=0x2006, w/h=(1920,1080), colorspace=8, field=1
[ 32.178272] tc358748 2-000e: [tc358748_stream,1152]: Stream disable ...
[ 32.240654] enter tc358748_s_power = OFF
[ 32.240673] tc358748 2-000e: tc358748_disable_hw
[ 60.422080] imx-sdma 30bd0000.dma-controller: firmware found.
[ 60.422080] imx-sdma 30e10000.dma-controller: firmware found.
[ 60.422326] imx-sdma 30bd0000.dma-controller: loaded firmware 4.6
[ 66.531400] cfg80211: failed to load regulatory.db

[joanxie]

could you send the logfile and dts file to me

[shaswath]

Please find the attached files.

[joanxie]

from imx8mp side, I don't find any issue,  the connection set up correctly

[ 7.992257] mx8-img-md: Registered sensor subdevice: tc358748 2-000e (1)
[ 8.002921] mx8-img-md: created link [mxc_isi.0] => [mxc_isi.0.capture]
[ 8.018980] mx8-img-md: created link [mxc-mipi-csi2.0] => [mxc_isi.0]
[ 8.025519] mx8-img-md: created link [tc358748 2-000e] => [mxc-mipi-csi2.0]
[ 8.032634] mxc-md 32c00000.bus:camera: mxc_md_create_links

in fact isi0 is for mipi csi0, after you add port@1 in the mipi csi0, you should get some errors in the current bsp, so do you change any mipi driver? but since you set up correctly, this shouldn't affect your capture, you can check the FPGA side, to check if they set the correct clock or send the correct data, to double review the registers of TC358748, also pls dump the mipi csi and ISI registers to check any error bits

[shaswath]

Please find he register dump.

[joanxie]

for now but I am unable to capture any frames using Gstreamer or v4l2-ctl.

> what kind gstreamer command do you use and what message do you get? could you confirm if your signal from TC358748 is total fine? did you check TC358748 side？

[shaswath]

Hello @joanxie,

Yes the input signals to TC358748 is okay and I see signals coming out of TC358748.

To start capture using Gstreamer:

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=1920,height=1080 ! autovideosink

Another issue is when the ISI node is enabled in the dts the system hangs after few minutes.

  

[joanxie]

checked your dts file settings 4 data lane for TC358748, but the logfile shows only 1 lane([ 39.473097] tc358748 2-000e: tc358748_enable_csi_lanes: 1), pls double check it

in your csi&isi dump.txt, [ 58.212425] tc358748 2-000e: PLL: pclk 218750000 (FPGA(Hardcode=>: 74250000MHz))

but in the gst-lauch-1.0.txt, [ 39.413394] tc358748 2-000e: PLL: pclk 218750000 (FPGA(Hardcode=>: 148500000MHz))

so what clock you set indeed? I checked your dts, you set 74.25M pixel clock, I suggest that you can measure the clock and set the hs-settle and clk-settle

[shaswath]

checked your dts file settings 4 data lane for TC358748, but the logfile shows only 1 lane([ 39.473097] tc358748 2-000e: tc358748_enable_csi_lanes: 1), pls double check it

--> That is just to indicate the function call, below that line you can see these line which enables 4 data lanes and 1 clock lane.

tc358748 2-000e: CLW_CNTRL: 0 0x140
tc358748 2-000e: D0W_CNTRL: 0 0x144
tc358748 2-000e: D1W_CNTRL: 0 0x148
tc358748 2-000e: D2W_CNTRL: 0 0x14c
tc358748 2-000e: D3W_CNTRL: 0 0x150

in your csi&isi dump.txt, [ 58.212425] tc358748 2-000e: PLL: pclk 218750000 (FPGA(Hardcode=>: 74250000MHz))

but in the gst-lauch-1.0.txt, [ 39.413394] tc358748 2-000e: PLL: pclk 218750000 (FPGA(Hardcode=>: 148500000MHz))

--> We are using 148.5MHz.

so what clock you set indeed? I checked your dts, you set 74.25M pixel clock, I suggest that you can measure the clock and set the hs-settle and clk-settle

--> hs-settle value was set based on this (pixel rate * bits per pixel) / (number of lanes * 2) which is 148.5 pixclk *16bpp / 2 * 4 lane = 475.2MBits/s so RX_HS_SETTLE is 10.

Also data lanes on our board were interchanged. So after this modification now I am getting a green screen when I launch video capture using Gstreamer or v4l2-ctl

gst-launch-1.0 -e \
v4l2src device=/dev/video3 ! \
'video/x-raw,format=YUY2,width=1920,height=1080,framerate=60/1' ! \
videoconvert ! queue ! waylandsink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:00:09.751003250
Setting pipeline to NULL ...
Total showed frames (544), playing for (0:00:09.750853750), fps (55.790)

[joanxie]

maybe you need consider the format, I checked the CHNL_IMG_CTRL, which is 100000b - YUV422_1P8P with 8-bits per color component; 1-plane, YUV interleaved packed bytes

https://github.com/nxp-imx/linux-imx/blob/lf-5.15.y/drivers/staging/media/imx/imx8-isi-fmt.c

{
.name = "YUYV-16",
.fourcc = V4L2_PIX_FMT_YUYV,
.depth = { 16 },
.color = MXC_ISI_OUT_FMT_YUV422_1P8P,
.memplanes = 1,
.colplanes = 1,
.align = 3,
.mbus_code = MEDIA_BUS_FMT_YUYV8_1X16,
}

you should choose this format from ISI as default, and check your bridge side too

[shaswath]

Hello,

Sorry for the delayed response. I used the format as you suggested and now I see some data but as you can see from the attached sample. It is still not perfect. I tried different MEDIA_BUS_FMT and V4L2_PIX_FMT's and also tried different frequencies (23.98, 50, 60, etc).

MEDIA_BUS_FMT_YUYV8_1X16 / UYVY8_1X16 / 2X8.

V4L2_PIX_FMT_YUYV / UYVY / YUV422P.

But as you can see from the dmesg log there are a lot of CRC error's. Also I notice that from media-ctl -p other than the bridge node none of them have format information.

[joanxie]

you couldn't choose any format you want, depends on what your bridge side, check your TC358748 register, what format did you set there, then choose the correct format of imx8mp side