Trouble with deinterlacing NTCS/PAL video with an IMX8MP

[amberworth]

Deinterlacing does not work on Yocto Linux, based on kernel 5.10.35.

We have adv748x connected to i.MX8MP using MIPI-CSI interface.

adv748x receive NTSC signal and transmit it over CSI interface to i.MX8MP.

here is output of the media-ctl -p command

Media controller API version 5.10.35

Media device information

------------------------

driver          mxc-md

model           FSL Capture Media Device

serial          

bus info        

hw revision     0x0

driver version  5.10.35

Device topology

- entity 1: mxc_isi.0 (16 pads, 2 links)

            type V4L2 subdev subtype Unknown flags 0

pad0: Sink

<- "mxc-mipi-csi2.0":4 [ENABLED]

pad1: Sink

pad2: Sink

pad3: Sink

pad4: Sink

pad5: Sink

pad6: Sink

pad7: Sink

pad8: Sink

pad9: Sink

pad10: Sink

pad11: Sink

pad12: Source

-> "mxc_isi.0.capture":0 [ENABLED]

pad13: Source

pad14: Source

pad15: Sink

- entity 18: mxc_isi.0.capture (1 pad, 1 link)

             type Node subtype V4L flags 0

             device node name /dev/video3

pad0: Sink

<- "mxc_isi.0":12 [ENABLED]

- entity 22: mxc-mipi-csi2.0 (8 pads, 2 links)

             type Node subtype V4L flags 0

             device node name /dev/v4l-subdev0

pad0: Sink

<- "adv748x 2-0070 txa":1 [ENABLED,IMMUTABLE]

pad1: Sink

pad2: Sink

pad3: Sink

pad4: Source

-> "mxc_isi.0":0 [ENABLED]

pad5: Source

pad6: Source

pad7: Source

- entity 31: adv748x 2-0070 txa (2 pads, 3 links)

             type V4L2 subdev subtype Unknown flags 0

             device node name /dev/v4l-subdev3

pad0: Sink

[fmt:UYVY8_2X8/720x240 field:alternate colorspace:smpte170m]

<- "adv748x 2-0070 afe":8 [ENABLED]

<- "adv748x 2-0070 hdmi":1 []

pad1: Source

[fmt:UYVY8_2X8/720x240 field:alternate colorspace:smpte170m]

-> "mxc-mipi-csi2.0":0 [ENABLED,IMMUTABLE]

- entity 34: adv748x 2-0070 afe (9 pads, 1 link)

             type V4L2 subdev subtype Decoder flags 0

             device node name /dev/v4l-subdev1

pad0: Sink

pad1: Sink

pad2: Sink

pad3: Sink

pad4: Sink

pad5: Sink

pad6: Sink

pad7: Sink

pad8: Source

[fmt:UYVY8_2X8/720x240 field:alternate colorspace:smpte170m]

-> "adv748x 2-0070 txa":0 [ENABLED]

I'm able to capture with command
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=720,height=240 ! queue ! kmssink

but this command capture only half of image - frame with only odd or only even lines

(720x240 instead 720x480).
When i try to enable deinterlacing it fails:

# gst-launch-1.0 --gst-debug=v4l2src:5 v4l2src device=/dev/video3 ! video/x-raw,width=720,height=240,interlace-mode=interlaced ! queue ! kmssink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

0:00:00.745451625   776 0xaaab147314c0 DEBUG                v4l2src gstv4l2src.c:515:gst_v4l2src_negotiate:<v4l2src0> caps of src: video/x-raw(format:Interlaced), format=(string)YUY2, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)YUY2, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)YUY2, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)BGRA, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)BGRx, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)BGR, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)BGR, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)BGR, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)RGB, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)RGB, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)RGB, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)NV12, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)NV12, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)NV12, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw(format:Interlaced), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate; video/x-raw, format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string){ progressive, interleaved }; video/x-raw(format:Interlaced), format=(string)RGB16, framerate=(fraction)[ 0/1, 2147483647/1 ], width=(int)[ 1, 4096 ], height=(int)[ 1, 8192 ], interlace-mode=(string)alternate

New clock: GstSystemClock

0:00:00.745651375   776 0xaaab147314c0 DEBUG                v4l2src gstv4l2src.c:523:gst_v4l2src_negotiate:<v4l2src0> caps of peer: video/x-raw, width=(int)720, height=(int)240, interlace-mode=(string)interlaced, format=(string){ BGRx, BGRA, RGBx, RGBA, NV12_10LE }, framerate=(fraction)[ 0/1, 2147483647/1 ]

0:00:01.078356375   776 0xaaab147314c0 DEBUG                v4l2src gstv4l2src.c:529:gst_v4l2src_negotiate:<v4l2src0> intersect: EMPTY

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.

Additional debug info:

../git/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:

streaming stopped, reason not-negotiated (-4)

Execution ended after 0:00:00.333395500

Setting pipeline to NULL ...

Freeing pipeline ...

I investigated the imx8-isi-hw.c and didn't find deinterlacing implementation.
There are 2 functions mxc_isi_channel_deinterlace_init and mxc_isi_channel_set_deinterlace but first of them is empty and second is never used.

Maybe I use wrong pipeline?

Is there other way to get interlaced image from the Capture Media Device?
Is there implementation of the ISI deinterlacing?

[petero5]

Did anyone ever manage to get odd/even (aka top/bottom) indications from v4l2_buffer via i.MX 8 MIPI CSI-2 (imx8-isi, mxc_isi)?

I'm managing to dequeue interlaced frames (50 fps, 720 x 288) from ADV7280A-M.

But the members of the v4l2_buffer are not changing to indicate odd/even.

https://community.nxp.com/t5/i-MX-Processors/Does-i-MX-8X-support-V4L2-FIELD-ALTERNATE-with-MIPI-CSI...

[ToarteFretter]

Hi @petero5,

Quite some time ago that we worked on this issue...

All I remember is that we worked closely with NXP on this case and the conclusion at the end was that actually the i.MX8 (QM in our case) is not able to de-interlace properly any kind of video. Basically, it boils down to the fact that the odd/even indication is lost from the MIPI-CSI towards the ISI.

Don't know the details anymore, but there were some ideas with interrupts and so on, trying to detect if the odd/even was improperly used to de-interlaced and then try to kind of give the de-interlacing a kick to get it right. But also there, as far as I remember, this was a hopeless thing. Oh wait, there was maybe a way if you could get the front-end (the one that generates the MIPI-CSI stream) to produce a different amount of lines in an odd versus an even field. And then use this interrupt that you can make occur when going over a certain amount of lines to detect the odd/even cadence. But then there was also this issue where you can impossible detect inside the registers which field is currently captured?

So we ended up not using the weave de-interlacing method or even the high quality blending de-interlacing method (which looked nice in 50% of the cases), but we did stick with bob de-interlacing.

Sorry for the bad news.

Best regards.

[marioschuknecht]

Hi amberworth,

Were you able to get deinterlacing NTCS/PAL to work with the mx8mplus?

Regards,

Mario

[igorpadykov]

Hi Amber

one can try to port below patch for i.MX8MP:

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/ISL79987-and-adv7180-de-interlace-driver...

Camera porting documents can be found on

https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces...

Best regards
igor

[marioschuknecht]

Hi igor

So far I have no success to deinterlace frames of the adv7280a-m with the i.MX8MP. Even with the patch applied to kernel 5.4

Basically frames can be received via mipi csi-2 on the i.MX8MP. Only as soon as i.MX8MP deinterlacing is activated, nothing happens.

Here the register dump for PAL (frame 720x576, field 720x288), Deinterlace method is weaving, no blending, no ntsc_top.

[  130.526648] ISI CHNLC register dump, isi0
[  130.530928]            CHNL_CTRL[0x00]: e0000000
[  130.535661]        CHNL_IMG_CTRL[0x04]: 20002001
[  130.540383]    CHNL_OUT_BUF_CTRL[0x08]: 7c707
[  130.544850]         CHNL_IMG_CFG[0x0c]: 12002d0
[  130.549488]             CHNL_IER[0x10]: 3cfc0000
[  130.554231]             CHNL_STS[0x14]: 100
[  130.558520]    CHNL_SCALE_FACTOR[0x18]: 10001000
[  130.563254]    CHNL_SCALE_OFFSET[0x1c]: 00
[  130.567467]        CHNL_CROP_ULC[0x20]: 00
[  130.571681]        CHNL_CROP_LRC[0x24]: 00
[  130.575884]      CHNL_CSC_COEFF0[0x28]: 00
[  130.580090]      CHNL_CSC_COEFF1[0x2c]: 00
[  130.584291]      CHNL_CSC_COEFF2[0x30]: 00
[  130.588502]      CHNL_CSC_COEFF3[0x34]: 00
[  130.592709]      CHNL_CSC_COEFF4[0x38]: 00
[  130.596920]      CHNL_CSC_COEFF5[0x3c]: 00
[  130.601128]     CHNL_ROI_0_ALPHA[0x40]: 00
[  130.605342]       CHNL_ROI_0_ULC[0x44]: 00
[  130.609555]       CHNL_ROI_0_LRC[0x48]: 00
[  130.613762]     CHNL_ROI_1_ALPHA[0x4c]: 00
[  130.617969]       CHNL_ROI_1_ULC[0x50]: 00
[  130.622180]       CHNL_ROI_1_LRC[0x54]: 00
[  130.626385]     CHNL_ROI_2_ALPHA[0x58]: 00
[  130.630591]       CHNL_ROI_2_ULC[0x5c]: 00
[  130.634800]       CHNL_ROI_2_LRC[0x60]: 00
[  130.639002]     CHNL_ROI_3_ALPHA[0x64]: 00
[  130.643220]       CHNL_ROI_3_ULC[0x68]: 00
[  130.647431]       CHNL_ROI_3_LRC[0x6c]: 00
[  130.651637] CHNL_OUT_BUF1_ADDR_Y[0x70]: 6a400000
[  130.656361] CHNL_OUT_BUF1_ADDR_U[0x74]: 00
[  130.660570] CHNL_OUT_BUF1_ADDR_V[0x78]: 00
[  130.664774]   CHNL_OUT_BUF_PITCH[0x7c]: 5a0
[  130.669069]     CHNL_IN_BUF_ADDR[0x80]: 00
[  130.673277]    CHNL_IN_BUF_PITCH[0x84]: 00
[  130.677480]     CHNL_MEM_RD_CTRL[0x88]: 00
[  130.681688] CHNL_OUT_BUF2_ADDR_Y[0x8c]: 6a100000
[  130.686414] CHNL_OUT_BUF2_ADDR_U[0x90]: 00
[  130.690623] CHNL_OUT_BUF2_ADDR_V[0x94]: 00
[  130.694831]     CHNL_SCL_IMG_CFG[0x98]: 12002d0
[  130.699469]       CHNL_FLOW_CTRL[0x9c]: 00

 Are the registers configured correctly?

Any idea what the problem could be?

Thanks in advance.

Mario

[akochubey]

Just in case anyone would be interested:

I had exactly the same problem (IMX8QM): MIPI CSI2 block receives frames OK, but as soon as deinterlace is turned on in ISI, no frames are getting through ISI.

This could be "fixed" by setting VC_INTERLACED (0x30) MIPI CSI CSR register to appropriate value (to me, VC0 is interlaced). I had to modify drivers/staging/media/imx/imx8-mipi-csi2.c to do this.

Frames are getting through after that change.

However, the result was ... a bit disappointing. "Need more work".

1. There is 50% chance that ISI weaving deinterlacing will result in a corrupted video.

It seems that odd/even flags are lost somewhere between CSI and ISI and ISI uses wrong field order. As a result, lines order is wrong and instead of getting smooth picture, you get it very "minecraft"-styled.

In CSI protocol, there is "F-bit", which indicates even/odd field, but it is unclear to me how it passes CSI -> Pixel Link -> ISI chain. Ref.Manual says "VC Interlaced - to drive the interlace port of the Pixel Link Master, via the HSYNC/VSYNC generator."

If that is simply counts up VSYNCs, it will never work, of course (right as observed, 50/50). If so, the only working deinterlace method would be blending (?).

2. Color space conversion stopped to work.

I'm unable to tell whether this is intended behaviour, because to my understanding, only "blending" requires CSC resources and weaving only requires different RAM layout at the output block.

Again, documentation is unclear here. See CHNL_BYPASS of ISI CHNL_CTRL CSR in Ref.Manual.

[ToarteFretter]

Hi,

It seems we have the same problem over here on the i.MX8QM…

Were you able to get the odd/even issue solved?

We’re also wondering how the odd/even is getting transferred over the pixel link between the MIPI-CSI and the ISI? The MIPI signals described in the reference manual do not contain any F-signal too…

We have also 50% chance that the odd/even fields are ordered/deinterlaced in the right way. This for both NTSC and PAL signals arriving (decoded) digitally on the MIPI input.

And if you would be curious, the blending mode has exactly the same problem. The problem is already visible in the intermediate YUYV image generated by ISI channel 1, which then enters ISI channel 0.

Any help or feedback would be nice.

Thanks!

Regards.

[marioschuknecht]

I was able to find out the following:

The interrupt callback mipi_csis_irq_handler of drivers/staging/media/imx/imx8-mipi-csi2-sam.c is executed 50 times per second. And the error bit status is always set to 00001000. The bit 12 means "Indication of lost of Frame Start packet, CH0."

But if deinterlacing is disabled, frames are received and error bit 12 is also set.

Can "lost of Frame Start packet" have influence on deinterlacing?

And without de-interlacing the frames are passed through?