How to convert simultaneously two video streams (from YUY2 to NV12) using HW CSC？

[RobbieJiang]

HW: IMX8QM
BSP: YOCTO 4.19.35

We need to encode 2 camera video streams simultanesouly.

Camera sensor outputs YUY2 raw data, but 'v4l2h264enc' can only accept NV12 data format.
So we have to convert YUY2 to NV12.

There are 3 possible video converters gstreamer elements:
imxvideoconvert_g2d, v4l2convert, videoconvert.

As we prefer to hardware conversion, so 'videoconvert' element will not be considered.
And 'imxvideoconvert_g2d' can NOT convert YUY2 format to NV12 format:

gst-inspect-1.0 imxvideoconvert_g2d
...

SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)RGB16, (string)RGBx, (string)RGBA, (string)BGRA, (string)BGRx, (string)BGR16, (string)ARGB, (string)ABGR, (string)xRG
B, (string)xBGR, (string)UYVY, (string)YUY2 }
video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition)
format: { (string)RGB16, (string)RGBx, (string)RGBA, (string)BGRA, (string)BGRx, (string)BGR16, (string)ARGB, (string)ABGR, (string)xRG
B, (string)xBGR, (string)UYVY, (string)YUY2 }

So the only choice is 'v4l2convert'.

The following pipeline can run without any problem:

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)YUY2,width=1280,height=960,framerate=30/1" ! v4l2convert ! 'video/x-raw,format=(string)NV12,width=1280,height=960,framerate=(fraction)30/1' ! v4l2h264enc ! fakesink sync=false async=false

However, seems that only 'one' instatnce of 'v4l2convert' can run.
If we tried to run another pipeline on another sensor:

gst-launch-1.0 v4l2src device=/dev/video2 ! "video/x-raw,format=(string)YUY2,width=1280,height=960,framerate=30/1" ! v4l2convert ! 'video/x-raw,format=(string)NV12,width=1280,height=960,framerate=(fraction)30/1' ! v4l2h264enc ! fakesink sync=false async=false

It reports: "WARNING: erroneous pipeline: no element "v4l2convert".
and ‘dmesg’ cmd shows
[ 1381.072904] (null): mxc_isi_capture_open: ISI channel[0] is busy
[ 1381.093449] mxc-isi 58100000.isi: mxc_isi_m2m_open: ISI channel[0] is busy

From 'gst-inspect-1.0 v4l2convert':

device : Device location
flags: readable
String. Default: "/dev/video4"

device-fd : File descriptor of the device
flags: readable
Integer. Range: -1 - 2147483647 Default: -1

Both 'device' and 'device-fd' properties are only readable, not writable.
Seems changing device is not supported.

How can we simultaneously convert two camera video streams (from YUY2 to NV12 raw format) by using hardware color space conversion?

[jiada]

@RobbieJiang 

I came into same situation, wanted to have HW video converter to do converting simultaneously from multiple camera video streams,

did you find out if it's possible to have "v4l2convert" achieve this task on iMX8 platform?

thanks,

Jiada 

[joanxie]

pls check if your two camera link to the same ISI? it seems the first camera use ISI0 and the second camera link to ISI0 too, you need set the second camera to the different ISI, did you use the nxp imx8qm mek board? and connect one camer via mipi_csi0 and another one via mipi_csi1??

[RobbieJiang]

Thanks for reply.

Do you mean different ISI channel?
Actually, there is only one ISI in MX8QM SoC,
but this ISI has 8 processing pipelines, which can implement the HW based CSC.

However,  seems there is no way to configure different channel using gstreamer element 'v4l2convert':

gst-inspect-1.0 v4l2convert:

...

device : Device location
flags: readable
String. Default: "/dev/video4"

device-fd : File descriptor of the device
flags: readable
Integer. Range: -1 - 2147483647 Default: -1

...

Note both  "device" and "device-fd" properties are ONLY READABLE, 

means no way to change it.

So the potential solution is to set the related configurations in kernel, not in the application.

[joanxie]

what's the result of v4l2-ctl --list-device?

[Anton_K]

Hi, I have some similar problem, actually I got single camera, but also only readable property on "device" option for default device on v4l2convert.

root@imx8qxp-var-som:~# v4l2-ctl --list-device
vpu B0 (platform:):
	/dev/video12
	/dev/video13

mxc-isi-cap (platform:58100000.isi:cap_devic):
	/dev/video1

mxc-isi-m2m (platform:58100000.isi:m2m_devic):
	/dev/video0

mxc-isi-cap (platform:58140000.isi:cap_devic):
	/dev/video2

mxc-jpeg codec (platform:58400000.jpegdec):
	/dev/video3

mxc-jpeg codec (platform:58450000.jpegenc):
	/dev/video4

FSL Capture Media Device (platform:mxc-md):
	/dev/media0

root@imx8qxp-var-som:~# gst-inspect-1.0 v4l2convert
...
  device              : Device location
                        flags: readable
                        String. Default: "/dev/video0"
...

root@imx8qxp-var-som:~# v4l2-ctl -L -d /dev/video0

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0
  min_number_of_capture_buffers 0x00980927 (int)    : min=3 max=16 step=1 default=3 value=3 flags=read-only
   min_number_of_output_buffers 0x00980928 (int)    : min=1 max=16 step=1 default=1 value=1 flags=read-only
                alpha_component 0x00980929 (int)    : min=0 max=255 step=1 default=0 value=0

I have to set /dev/video1 for v4l2src to be workable (here /dev/video0 by default).

My camera OV5640. 
Also my problem is using v4l2convert, I can't use it by pipeline:

GST_DEBUG=3 XDG_RUNTIME_DIR=/run/user/0 gst-launch-1.0 -vvv v4l2src device=/dev/video1 io-mode=dmabuf ! video/x-raw,format=YUY2,framerate=30/1,width=1280,height=720 ! v4l2convert ! video/x-raw,format=NV12,framerate=30/1,width=1280,height=720 ! v4l2h264enc output-io-mode=dmabuf-import ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! waylandsink