Questions about GStreamer and IPU Video Capture

[steveanderson]

I am pretty sure this is most appropriately an iMX6 question than a GStreamer question, so I chose this forum...

I am porting software to Yocto FIDO to get the most recent bugfixes (and bugs) from all the related software.

My board is a custom System On Module using an iMX6q.

I am using the parallel capture interface CSI0 with a GS2960, which produces YUYV video (AKA YUY2, V422, YUNV).

-- FROM:  IMX6DQRM 37.4.3.2.1 Parallel Interface

CSI can work with several data formats according to SENS_DATA_FORMAT configuration.

In case the data format is YUV, the output of the CSI is always YUV444 (even if the data arrives in YUV422 format).

This seems to imply that only the electronic interface will ever see the YUYV signal, downstream will start with YUV444.

My driver goes about the details of setting up the CSI0 interface, but what format should I be telling GStreamer to deal with?

Obviously I am in the YUV colorspace, but what pixel format will GStreamer need (from the driver, from the command line)?

What format should I be reporting from the driver?

If memory is stuffed with YUV444, is that what I should report?

If so, then which fourcc format code is that? (Y444? IYU2?)

What video format should be referenced in the gstreamer command line, then?

Some guidance here would be very helpful...  Thanks...

[Yuri]

  You may run a demo image (X11, QT5) in order to see what data

formats are used by Gstreamer elements, using gst-inspect command 

In particular we can see the following SINK and SRC capabilities

For imxv4l2src: v4l2 based camera src

  SRC template: 'src'

    Capabilities:

      ANY

For vpuenc

($ gst-inspect vpuenc )

  SINK template: 'sink'

    Capabilities:

      video/x-raw-yuv

                 format: TNVP

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

  SRC template: 'src'

    Capabilities:

      video/mpeg

            mpegversion: 4

           systemstream: false

      video/x-h263

      video/x-h264

      image/jpeg

For vpudec
($ gst-inspect vpudec)

SINK template: 'sink'

    Capabilities:

      video/mpeg

            mpegversion: 4

      video/x-h264

      video/x-h263

      video/mpeg

           systemstream: false

            mpegversion: { 1, 2 }

      video/x-wmv

             wmvversion: 3

                 format: WVC1

      video/x-wmv

             wmvversion: 3

      video/x-xvid

      video/x-vp8

      image/jpeg

  SRC template: 'src'

    Capabilities:

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

      video/x-raw-yuv

                 format: YV12

      video/x-raw-yuv

                 format: TNVP

      video/x-raw-yuv

                 format: TNVF

For mfw_isink: IPU-based video sink

SINK template: 'sink'

    Capabilities:

      video/x-raw-yuv

                 format: NV12

      video/x-raw-yuv

                 format: I420

      video/x-raw-yuv

                 format: YUY2

      video/x-raw-yuv

                 format: Y444

      video/x-raw-yuv

                 format: Y42B

      video/x-raw-yuv

                 format: 422P

      video/x-raw-yuv

                 format: UYVY

      video/x-raw-yuv

                 format: YUYV

      video/x-raw-yuv

                 format: Y800

      video/x-raw-gray

                    bpp: 8

      video/x-raw-rgb

                    bpp: 16

      video/x-raw-rgb

                    bpp: 24

      video/x-raw-rgb

                    bpp: 32

For mfw_ipucsc: IPU-based video converter

  SRC template: 'src'

    Capabilities:

      video/x-raw-yuv

                 format: NV12

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-yuv

                 format: I420

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-yuv

                 format: UYVY

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-rgb

                    bpp: 16

                  depth: 16

             endianness: 1234

               red_mask: 63488

             green_mask: 2016

              blue_mask: 31

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

  SINK template: 'sink'

    Capabilities:

      video/x-raw-yuv

                 format: NV12

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-yuv

                 format: I420

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-yuv

                 format: UYVY

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

      video/x-raw-rgb

                    bpp: 16

                  depth: 16

             endianness: 1234

               red_mask: 63488

             green_mask: 2016

              blue_mask: 31

                  width: [ 1, 2147483647 ]

                 height: [ 1, 2147483647 ]

              framerate: [ 0/1, 2147483647/1 ]

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[steveanderson]

I am still looking for the answer to this question.

I will re-re-state the most succinct form I can think of.

My confusion is based on a sentence in the IMX6DQRM section 37.4.3.2.1:

"In case the data format is YUV, the output of the CSI is always YUV444 (even if the data arrives in YUV422 format)."

I am writing a driver for a chip which outputs YUV422 data on the parallel port to the CSI.

This paragraph suggests that while I clearly need to configure the CSI driver to correctly capture YUV422 data, it seems I will need to report (eg for VIDIOC_TRY_FMT) YUV444 since this paragraph seems to say that is what will be buffered in memory when a frame is captured.

I just want to confirm this since I am not finding any example code working with mxc_v4l2_capture using the same kind of video stream data.  This also seems it should be the case for YUV420, for that matter...

I suppose that once the other hardware issues are out of my way and I actually capture a frame, I'll know pretty quickly if the chroma data is sensible or not...  But it would be quite nice to hear from someone who actually knows how this works.  I don't see any other mention of this in related manuals.

[Yuri · Accepted solution]

  The phrase “the output of the CSI is always YUV444” relate to

internal data from Data Packing Unit to IDMAC and to IC are of
YUV444. Please look at Figure 37-15 (CSI Block Diagram) and
Figure 37-1 (IPU Block Diagram) of the RM.

   The IDMAC in its turn can convert data, as described in section

37.4.2.3 (FCW & FCR - Format converter write and read).

As for mxc_v4l2_capture, You may look at GST plugin sources, in particular
at the following from the recent BSP (3.14.28) :

gst1.0-fsl-plugins-4.0.2.tar.gz

gst-fsl-plugins-4.0.2.tar.gz

Regards,

Yuri.

[steveanderson]

Thankyou, Yuri, that was the connection I had missed.  This has at least pushed me on to the next hurdle, which I believe requires changes mxc_v4l2_capture and probably extending v4l2-int-device - which it looks like someone moved to the mxc capture folder from the v4l2 core folder - at least in FIDO...

This still leaves issues with the imx plugins for gstreamer, some of which I have already figured out.

However, my comment about the gst-inspect output still holds.  Take the output for gst-inspect-1.0 imxv4l2src, for which I copy/paste the relevant portion here:

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw

So what does this mean?  It is totally useless.  Analyzing the code, it only works with I420, which is not quite totally useless, but is only a very short stroll away from there.  I am planning rework on that.  This will be a whole lot more complex than adding YUYV (YUY2) to imxipuvideosink and imxipuvideotransform, but that help output is just text from the author, and it is not necessarily accurate.

[steveanderson]

No, not an answer to my question, let me try to re-iterate this because, Yuri, you are probably one of the few who might actually be able to answer the actual question, so please read the below carefully as I have tried to be very clear about the exact problem I am perceiving.  I think the last sentence, below, more clearly re-states the root question - the rest presents the context of my understanding of the software I am asking about - which connects to gstreamer (and other) software but does not seem to be doing so correctly.

The problem with your answer is that gs-inspect only regurgitates what a plugin tells about itself, its capabilities, which are not even necessarily its actual capabilities,  and I don't want to get into a discussion of how useless imxv4l2src is in comparison to the capabilities of the csi/ipuv3...  so back to my question...

The problem with 'gs-launch -v' is that it describes (at best) what the gstreamer software has been told is being captured or (at worst, like imxv4l2src) it reports the only video format the software will actually work with just before it crashes out because you are producing a different format the plugin should be designed to handle.

No, my question is not really a gstreamer question at all, it is an CSI/IPU/mxc_v4l2_capture question - a driver question...  yet it is related to trying to make correct use of gstreamer and its vast capabilities by having the driver properly report what is actually captured.

Let me repeat the confusing/offending paragraph from the CSI documentation:

-- FROM:  IMX6DQRM 37.4.3.2.1 Parallel Interface

CSI can work with several data formats according to SENS_DATA_FORMAT configuration.

In case the data format is YUV, the output of the CSI is always YUV444 (even if the data arrives in YUV422 format).

So, as I stated in the first question, I have a video converter chip which is providing the CSI with a YUYV video stream at 1280x720P60.  Obviously to capture that I will need to configure the CSI for that particular input stream - one it is clearly capable of managing.

That paragraph in the reference manual, however, tells me that once captured, the data in memory has been converted from YUV422 to YUV444.  The problem is that the mxc_v4l2_capture software does not seem to have the capacity to report this change in video data format downstream.  Therefore, in fact, it seems that mxc_v4l2_capture is not correctly reporting the kind of data which is captured into memory, nor is it capable of doing so as it currently seems to be written.

Instead it will report the format which is being received (eg with v4l2src which actually (mostly) negotiates the input), but the manual says that the data in memory has been modified to a different format.  In other words, my driver configures the CSI to capture YUYV, which is automatically converted by the hardware to YUV444 which is actually stored in memory, but when gstreamer queries the msc_v4l2_capture driver about the video format, it merely reports the source YUYV, instead of what is actually being stored into memory YUV444.

Shouldn't the mxc_v4l2_capture actually be reporting the pixel format which has been captured into memory, especially if it is actually different from the data being presented to the CSI, so the downstream software (ie gstreamer) is actually being properly informed about the data it is processing?

[steveanderson]

And, just to extend or punctuate the question.  Consider imxv4l2src, which only seems to work with I420 - a terribly crippling limitation, but after that video stream has been stored to memory, it has presumably gone through the CSI as well as the de-interlacer, and so it is no longer 'I' nor is it '420' but is actually stored in memory as de-interlaced 'P' and '444' because it has been converted from YUV420 to YUV444 as well as being de-interlaced?