IMX8MP De-interleave CSI2 PAL 576i video

[TerryBarnaby1]

We have a custom board that uses a TP2855 analogue video to CSI2 front end chip with an IMX8MP.

As well as FHD 1920x1080p25 video inputs we need to support SD 720x576i25 PAL video streams.

Unfortunately, as we have found out (NXP Datasheets and reference manuals "REALLY" need to be clear on both capabilities and lack of common capabilities (and correct) !) the IMX8MP CSI video input and image processing system is pretty crippled. The IMX8X is better (why doesn't a family of devices have common capabilities) and the old IMX6 was pretty good and in quite a few ways better.

Anyway, we now need to look at how to de-interlace the PAL video stream somehow. I am hoping a software driven hardware DMA or 2D graphics process bit blit can be used.

1. Has anyone done this or have any pointers on how to do this ?

2. Is there any information available from IMX8MP hardware on if the CSI field being received is an odd or even field to synchronise the de-interlace ?

[dominiquevovard]

Dear @TerryBarnaby1,

thank you for your feedback.
We have the same situation but with isl7998x and we have tried to use deinterlace GStreamer plugin or videoscale with different settings of CSI-2 interlaced fields/frames with the waylandsink on iMX8MP, but nothing seems to work...

Do you have developed your own GStreamer plugin to deinterlace the video stream or just use an existing GStreamer solution ?

Just in case, could you share your plugin sources if it is not that's not too much to ask, and could you report the CPU load that you can observe ?

Kind regards,
Dom

[TerryBarnaby1]

I couldn't get the standard gstreamer de-interlace plugin to work either. There is a lack of documentation generally on gstreamer and the NXP side making this difficult as I am unclear as to exactly what the gstreamer de-interlacer does and what it needs in the input stream etc. My feeling is that it probably needs 50Hz fields with an odd/even identifier somehow and that does not look possible with the NXP IMX8MP video hardware.

My simple, quick and dirty plugin (attached) simply takes (in the case of PAL) 25Hz 720x576 frames in that consists of 288 lines of the first field followed by 288 lines of the second field and outputs 25Hz 720x576 frames with the lines interleaved correctly. It relies on the ability of the TP2855 to effectively output the two fields in one CSI2 video frame. This is pretty basic and no good if motion is involved. I will be creating a better one later in the project now that we know we can support PAL cameras. It would be possible to use the IMX8MP 2D engine to do this and do more sophisticated de-interlacing, but I haven't had time to look at that route yet.

On our IMX8MP system (4 core, 1600 Mhz) it takes about 8% of one CPU core and obviously the RAM in/out bandwidth as well.

[dominiquevovard]

Hi @TerryBarnaby1 

Many thanks for sharing with us your code. I appreciate!!!
We have the same fields/frame configuration and CPU load seems to be acceptable but if we have some works with an acceleration stack that can give some results, we will share them too.

kind regards
Dom

[TerryBarnaby1]

Note the Makefile in there is dependent on our local build system, so would need changing.

[TerryBarnaby1]

Just for others information, I have managed to work around this lack of interlaced to progressive support in the IMX8MP by using a mode in the TP2855 frontend chip I'm using that can output the two interlaced fields in one CSI2 frame. A simple software de-interfacer implemented as a gstreamer module converts the two fields to a progressive field. Not ideal at all, but works. Looking at the IMX8 documentation it "looks" like the IMX8MP ISI block can do de-interlacing, but the CSI2 hardware block does not determine/pass the relevant info from the CSI2 stream for this or the virtual channels. Oh why, oh why NXP did you scupper this video hardware so much and also can you please update your manuals (won't take much time, is there a procedure for fixing manual errors ?) so others don't fall into this trap ?

[joanxie]

yes, software is a solution, but maybe performance isn't good enough, thanks for you suggestion, I am agree with you, this document isn't update yet

[joanxie]

unfortunately i.mx8mp doesn’t support Virtual channel mode and i.mx8mp doesn’t support interlaced mode;

[dominiquevovard]

Hi @joanxie

I'm using isl799x on our design based on i.MX8MP...

Do you confirm that iMX8MP ISI doesn't support interlaced mode and chapter 13.4.3.4.1 in i.MX 8M Plus Applications Processor Reference Manual, Rev. 1, 06/2021 is fuly wrong ?

Regards,
Dom

[TerryBarnaby1]

1. Actually is there some definitive information on if the IMX8MP's hardware can do de-interlacing of a camera's interlaced CSI2 video stream. I have seen conflicting information on this and also what exactly, if it is available, can the de-interlacing hardware do ?

2. From the IMX8MP reference manual, it states it has both ISI and ISP hardware blocks. In the ISI information it is stated it supports: "Interlaced to progressive conversions". Is this correct ?

3. If so I note the Linux kernels drivers/staging/media/imx/imx8-isi-* code, which is built for the 5.10.52-lts kernel I am using, has functions such as mxc_isi_channel_set_deinterlace() and also uses registers like: CHNL_OUT_BUF_PITCH which might be useful to me. Are these supported on the IMX8MP (it is difficult to determine what is supported or not in the different IMX8 MPU's) ?

4. Is there any information on the ISI's and ISP 's registers so I can see if it is possible to use these somehow ? The register documentation appears to be missing for this.

If the IMX8MP's CSI/ISI/ISP blocks cannot handle de-interlacing a PAL 720x576i25 video stream (ie. it can create 720x576i25 interleaved frames using a simple weave algorithm from a set of 720x288 50Hz fields, in the correct order) then I will have to look at doing this some other way.

On my item 2, working out which interleaved field is which, if there isn't the information available from the CSI2 hardware block or ISI/ISP blocks it looks like I can use a TP2855 feature that will provide CSI frames with the two fields inside stored in the top half and bottom half of the full frame.

So now I just need to be able to take these two fields and interlace them line by line to get a complete interleaved frame (we have pretty static images so a simple weave interlaced full frame will likely be fine for now. From what I understand the ISI/ISP blocks are responsible for DMA'ing the frames to RAM. If the IMX8MP's ISI/ISP blocks support the CHNL_OUT_BUF_PITCH register when DMA'ing the CSI video stream to RAM, I might be able to use this somehow.

5. Otherwise any pointers/recommendations on using either the CPU's SDMA engine to do the 2D (bit-blit with line stride) DMA or 2D graphics engine, ideally within the kernel so the /dev/video* video4linux stream produces interleaved interlaced frames ?

Otherwise I guess I can look at adding a gstreamer processing stage to do this somehow.

[joanxie]

for reference manual, it seems that this is copy from imx8qm, imx8qm/8qxp which has virtual channels can support de-interlacing, but for imx8mp ISI, which doesn't have virtual channel, so couldn't support de-interlace mode

"https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/ISL79987-and-adv7180-de-interlace-driver..."