i.MX8QXP cannot stream video from ADV7280-M on Linux 6.12

[dominic-vt]

Hello,

We have a custom board based on the i.MX8QXP SoC that uses the ADV7280-M decoder for analog camera input. We're trying to upgrade from the Linux 6.6 linux-imx kernel to version 6.12, and on the new kernel streaming from the ADV7280-M does not work at all.

I know that since 6.6 there has been a major reachitecture of the media pipeline for this SoC and now we have to configure the pipeline manually and make sure the formats all match. Through a lot of debug prints in the kernel I've been able to do that and get to a state where streaming seems to start successfully but now frames actually come through, ever.

The media pipeline is as follows:
- adv720m 1-0021 pad 0 fmt:UYVY8_2X8 @ 720x576 25fps field:none
- imx8mq-mipi-csi2 pad 0 fmt:UYVY8_2X8 -> pad 1 fmt:UYVY8_2X8 @ 720x576 field:none
- crossbar pad 2 fmt:UYVY8_2X8 -> pad 6 fmt:UYVY8_2X8 @ 720x576 // Note: this routes pads 6-11 to 'mxc.isi.0-5':0
- mxc_isi.0 pad 0 fmt:UYVY8_2X8 -> pad 1 @ 720x576 field:none
- mxc_isi.0.capture pad 0 (no fmt or field set)

mxc_isi.0.capture is exposed as /dev/video4/.

I've attached the output of "media-ctl -p -d1".

With that the pipeline validation succeeds and all of the .enable_streams, .streamon, .start_streaming, .s_stream functions seem to run and exit with code 0, however no frame actually comes through.

I've tried to stream with the following commands:
- gst-launch-1.0 v4l2src device=/dev/video4 ! video/x-raw,width=720,height=576,framerate=25/1 ! autovideosink
- v4l2-ctl -d /dev/video4 --stream-mmap --stream-count=5

but none of them actually capture anything.

With debug prints on and the kernel log level = 7, I see the following output:

root@imx8qxp-d7:~# v4l2-ctl -d /dev/video4 --stream-mmap --stream-count=5
[ 334.960386] adv7180_set_pad_format:813 enter adv7180 set_fmt field: 0x000000
[ 691.055405] mxc_isi_video_streamon:1302 entering
[ 691.062698] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":6 -> "mxc_isi.0":0
[ 691.077772] mxc-isi 58100000.isi: Link 'crossbar':6 -> 'mxc_isi.0':0 is valid
[ 691.085186] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "imx8mq-mipi-csi2 58227000.csi":1 -> "crossbar":2
[ 691.097362] mxc-isi 58100000.isi: Link 'imx8mq-mipi-csi2 58227000.csi':1 -> 'crossbar':2 is valid
[ 691.106310] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "adv7180 1-0021":0 -> "imx8mq-mipi-csi2 58227000.csi":0
[ 691.119110] mxc-isi 58100000.isi: Link 'adv7180 1-0021':0 -> 'imx8mq-mipi-csi2 58227000.csi':0 is valid
[ 691.128591] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":7 -> "mxc_isi.1":0
[ 691.139021] mxc-isi 58100000.isi: Link 'crossbar':7 -> 'mxc_isi.1':0 is valid
[ 691.146221] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":8 -> "mxc_isi.2":0
[ 691.156649] mxc-isi 58100000.isi: Link 'crossbar':8 -> 'mxc_isi.2':0 is valid
[ 691.163838] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":9 -> "mxc_isi.3":0
[ 691.174269] mxc-isi 58100000.isi: Link 'crossbar':9 -> 'mxc_isi.3':0 is valid
[ 691.181465] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":10 -> "mxc_isi.4":0
[ 691.191975] mxc-isi 58100000.isi: Link 'crossbar':10 -> 'mxc_isi.4':0 is valid
[ 691.199254] mxc-isi 58100000.isi: v4l2_subdev_link_validate_default: validating link "crossbar":11 -> "mxc_isi.5":0
[ 691.209763] mxc-isi 58100000.isi: Link 'crossbar':11 -> 'mxc_isi.5':0 is valid
[ 691.217033] getting state for subdev mxc_isi.0
[ 691.222447] mxc_isi_video_streamon:1348 about to call vb2_streamon
[ 691.228677] mxc_isi_vb2_buffer_queue:1057 buffer 0 queued
[ 691.234117] mxc_isi_vb2_buffer_queue:1057 buffer 1 queued
[ 691.239548] mxc_isi_vb2_buffer_queue:1057 buffer 2 queued
[ 691.245072] mxc_isi_vb2_buffer_queue:1057 buffer 3 queued
[ 691.250505] mxc_isi_vb2_start_streaming:1086 starting stream
[ 691.261239] mxc_isi_pipe_enable:331 entering
[ 691.261250] mxc_isi_pipe_enable:362 sink_info idx=1 src_info idx=3
[ 691.265552] mxc_isi_pipe_enable:363 sink_fmt: width=720 height=576 code=0x002006 encoding=2
[ 691.272020] mxc_isi_pipe_enable:365 src_fmt: width=720 height=576 code=0x002025 encoding=2
[ 691.280498] mxc_isi_channel_config:383 entering
[ 691.293464] mxc_isi_channel_set_csc:239 entering
[ 691.293477] mxc-isi 58100000.isi: CSC: YUV -> YUV
[ 691.302890] mxc_isi_channel_set_csc:276 exiting val & CHNL_IMG_CTRL_CSC_BYPASS = 1
[ 691.302905] mxc_isi_channel_config:408 exiting. csc_bypass = 1 scaler_bypass = 1
[ 691.310561] mxc_isi_crossbar_enable_streams:493 enter
[ 691.323167] mxc_isi_crossbar_enable_streams:503 for device imx8mq-mipi-csi2 58227000.csi
[ 691.331407] mxc_isi_crossbar_enable_streams:532 about to enable stream for imx8mq-mipi-csi2 58227000.csi
[ 691.341396] adv7180_s_stream:939 starting to stream
[ 691.341403] adv7180_s_stream:955 streaming
[ 691.346392] mxc_isi_crossbar_enable_streams:545 exit ret = 0
[ 691.356268] mxc_isi_video_streamon:1356 exit ret = 0
# and here it hangs until I exit with Ctrl-C

I've attached the relevant sections of our device tree as well.

Changes so far

- In the adv7180.c driver, we force the field to V4L2_FIELD_NONE in the probe, so the device should be outputting progressive format.
- In imx8mq-mipi-cs2.c, I've added `MEDIA_BUS_FMT_UYVY8_2X8` to the `imx8mq_mipi_csi_formats` array to allow setting that format.
- In imx8-isi-video.c, in the `mxc_isi_video_validate_format` function I've removed checking for the same media bus format.

I've also tried backporting the imx8mq-mipi-csi2 driver from the mainline kernel as it seems support for the iMX8QXP in that driver was added in a later commit: https://github.com/torvalds/linux/commit/642b70d526ab8daa8f256dfc1eb6bf27c3290cc6.

That had the same result: no frames actually coming through the pipeline.

I would be grateful for any suggestions on what to do to get streaming working!

Thanks in advance!

[dominic-vt · Accepted solution]

Figured out the problem: the ADV chip wasn't actually being powered on since the new MIPI CSI2 driver doesn't call the s_power() callback, unlike the previous driver.

The solution was to implement Runtime PM for the adv7180.c driver. I've attached the patches containing my working implementation.

This would be best practice, as mentioned in the kernel documentation: https://docs.kernel.org/5.17/driver-api/media/camera-sensor.html#power-management

The patches won't apply perfectly on the upstream adv7180.c file because we have a few unrelated modifications in ours, but the changes should be easily copy-pasted if anyone else needs them.

Thank you very much for the suggestions!

[zoey_li]

Hi @dominic-vt ,

The hs_settle timing must be configured according to the sensor data rate. Generally speaking that, you need to configure link_freq parameter in sensor driver and the mipi-csi driver will calucate hs_settle timing according to it. You can find more details in drivers/media/platform/nxp/imx8mq-mipi-csi2.c.

And link_freq = sensor_data_rate_per_lane / 2.

Regards,

Zoey

[dominic-vt]

Hi @zoey_li,

Thanks for the reply! I did set the link frequency in the adv7180.c driver. The data sheet says 432Mbps when the ADV is in progressive mode. 

root@imx8qxp-d7:~# v4l2-ctl -d /dev/v4l-subdev8 --get-ctrl link_frequency
link_frequency: 0 (216000000 0xcdfe600)

Even with this set, there is still no output.

Could it be anything to do with trying to do format conversion in the ISI? 

- entity 14: mxc_isi.0 (2 pads, 2 links, 0 routes)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: SINK
[stream:0 fmt:UYVY8_2X8/720x576 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
compose.bounds:(0,0)/720x576
compose:(0,0)/720x576]
<- "crossbar":6 [ENABLED,IMMUTABLE]
pad1: SOURCE
[stream:0 fmt:YUV8_1X24/720x576 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(0,0)/720x576
crop:(0,0)/720x576]
-> "mxc_isi.0.capture":0 [ENABLED,IMMUTABLE]

(note pad 0 is UYVY8_2X8 and pad 1 is YUV8_1X24)

Regards,

Dominic

[zoey_li]

Hi @dominic-vt ,

For mxc_isi.x subdevice,

- pad 0 indicates the input format, should be the same format as sensor/mipi-csi/crossbar

- pad 1 indicates the color space of output format.

And the final output format of the entire ISI media pipeline is controlled by user space application which call set_fmt ioctl for the video device (mxc_isi.x.capture).

I don't see any obvious error in your printed media topology, and ISI reformat or CSC should not lead to capture hang too. Please try to check the MIPI-CSI status registers (several times during capturing) and see if there is any error reported. 

If still no finding, to narrow down the issue, recommend you to revert all changes you have done for 8QXP mipi-csi and isi drivers, and use media-ctl to configure the media pipeline manually.

By the way, you configured data lane number to 1 in dts, please make sure it match with your sensor setting.

Regards,

Zoey

[dominic-vt]

Hi @zoey_li,

Thank you very much for the suggestion to check the registers! That led me to find that the ADV chip wasn't actually powered on for streaming.

I've posted a couple of patches containing the fix and marked it as the solution in case anyone else has this issue.

Regards,
Dominic

[zoey_li]

Glad to hear that, and thank you for sharing the patches for others' reference.

[joanxie]

it seems that you missed “mxc_isi.0':1 -> 'mxc_isi.0.capture':0", try to use media-ctl to set it and check again

[dominic-vt]

I think I might have missed it in the pipeline summary in the post, sorry! But the link is there in the media-ctl output:

...
- entity 14: mxc_isi.0 (2 pads, 2 links, 0 routes)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
...
pad1: SOURCE
[stream:0 fmt:YUV8_1X24/720x576 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(0,0)/720x576
crop:(0,0)/720x576]
-> "mxc_isi.0.capture":0 [ENABLED,IMMUTABLE]

- entity 17: mxc_isi.0.capture (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video4
pad0: SINK
<- "mxc_isi.0":1 [ENABLED,IMMUTABLE]

[joanxie]

yes, I found it, but refer to your debug print log, I didn't find this link, you can double check it, I tested on the imx8qm before, I remembered that I must add this link via media-ctl -p command, maybe you can try it

[dominic-vt]

I've added a media_entity_operations with empty implementations that return 0 for .link_validate and .link_setup and then ran `media-ctl -d /dev/media1 -l '"mxc_isi.0":1 -> "mxc_isi.0.capture":0 [1]'`.

Still the same result, no frames actually come through.

I even tried setting pad 1 of mxc_isi.0 to RGB888_1X24 to enable the CSC and still nothing.

Edit: the media_entity_operations was added to the mxc_isi.0.capture device in mxc_isi_video_register() to vdev->entity.

[joanxie]

could you offer the full log from your board boot up, and do you have any patch for your modification?

[dominic-vt]

Here are the patches for what I've done so far. They don't include the device tree.

The patches are quite large, but they're mostly debug prints.

I've attached the full dmesg output from boot up to trying to stream. The commands I've run after enabling debug prints for the imx8-isi drivers are, in this order:

- media-ctl -d /dev/media1 -V '"imx8mq-mipi-csi2 58227000.csi":0 [fmt:UYVY8_2X8/720x576]'

- media-ctl -d /dev/media1 -V '"mxc_isi.0":1 [fmt:RGB888_1X24/720x576 field:none]'

- gst-launch-1.0 v4l2src device=/dev/video4 ! video/x-raw,width=720,height=576,framerate=25/1 ! autovideosink

and then stopped streaming after approx 10s.

Thank you!