IMX6DQ and gstremer-1.0 video pipeline issue: imxv4l2src Capabilities are Empty

[adamschafer]

Greetings,

We have a custom board with an adv7282 video decoder feeding into IPU1/CSI0 and an out through an adv7391 encoder connected to IPU1/DISP2.  We've managed to bring up these chips/interfaces and have successfully tested them with the utilities in the /unit_tests directory -- the following script will record ~1 second of video from the decoder, save it to a file, and then push it out through the encoder:

WIDTH=720
HEIGHT=480

echo "Capturing video from input\n"
/unit_tests/mxc_v4l2_capture.out -iw $WIDTH -ih $HEIGHT -ow $WIDTH -oh $HEIGHT -d /dev/video0 -m tmp

echo "Pushing video to output"
/unit_tests/mxc_v4l2_output.out -iw $WIDTH -ih $HEIGHT tmp

However, we're running into an issue when trying to pipe the decoded input video to the output encoder using gstreamer:

root@imx6dqscm-1gb-hub-11x:~# gst-launch-1.0 imxv4l2src device=/dev/video0 ! imxv4l2sink
====== IMXV4L2SRC: 4.0.9 build on May 16 2017 20:39:59. ======
====== IMXV4L2SINK: 4.0.9 build on May 16 2017 20:39:59. ======
WARNING: erroneous pipeline: could not link imxv4l2src0 to imxv4l2sink0

Being new to gstreamer I've been pretty lost trying to find a solution to this issue.  After digging through some debug steps, I think that the issue is related to the src and sink capabilities, as shown in the bolded sections below:

root@imx6dqscm-1gb-hub-11x:~# gst-launch-1.0 imxv4l2src device=/dev/video0 ! imxv4l2sink --gst-debug=GST_CAPS:6
====== IMXV4L2SRC: 4.0.9 build on May 16 2017 20:39:59. ======
====== IMXV4L2SINK: 4.0.9 build on May 16 2017 20:39:59. ======
0:00:01.043703334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2src0:src> get pad caps with filter (NULL)
0:00:01.044001334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2src0:src> query returned EMPTY
0:00:01.044082000 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2sink0:sink> get pad caps with filter (NULL)
0:00:01.044308667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2sink0:sink> query returned video/x-raw, format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:01.044481000 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2src0:src> get pad caps with filter (NULL)
0:00:01.044571667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2src0:src> query returned EMPTY
0:00:01.044724334 647 0x34b1a0 LOG GST_CAPS gstutils.c:809:gst_element_get_compatible_pad_template: checking pad template sink
0:00:01.044794667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:814:gst_element_get_compatible_pad_template: compatible direction: found sink pad template "sink"
0:00:01.044860334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:817:gst_element_get_compatible_pad_template: intersecting EMPTY
0:00:01.044918667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:819:gst_element_get_compatible_pad_template: ..and video/x-raw, format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition), format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:01.045093667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:825:gst_element_get_compatible_pad_template: caps are not compatible
0:00:01.045246667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2sink0:sink> get pad caps with filter (NULL)
0:00:01.045431334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2sink0:sink> query returned video/x-raw, format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:01.045568334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2src0:src> get pad caps with filter (NULL)
0:00:01.045653667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2src0:src> query returned EMPTY
0:00:01.045744334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2838:gst_pad_query_caps:<imxv4l2sink0:sink> get pad caps with filter (NULL)
0:00:01.045912667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:2845:gst_pad_query_caps:<imxv4l2sink0:sink> query returned video/x-raw, format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:01.378598000 647 0x34b1a0 LOG GST_CAPS gstutils.c:809:gst_element_get_compatible_pad_template: checking pad template src
0:00:01.378672334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:814:gst_element_get_compatible_pad_template: compatible direction: found src pad template "src"
0:00:01.378739667 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:817:gst_element_get_compatible_pad_template: intersecting video/x-raw, format=(string){ RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:01.378869334 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:819:gst_element_get_compatible_pad_template: ..and EMPTY
0:00:01.378929000 647 0x34b1a0 DEBUG GST_CAPS gstutils.c:825:gst_element_get_compatible_pad_template: caps are not compatible
WARNING: erroneous pipeline: could not link imxv4l2src0 to imxv4l2sink0

Does anyone have an idea why imxv4l2src would return an empty set of capabilities?  Is there a way I can define the capabilities of imxv4l2src via the gstreamer commandline?

Thank you for your time.  Please let me know if I can provide any more helpful information.

Regards,

-Adam

[igorpadykov]

Hi Adam

what bsp used in the case, please try nxp bsps described on

http://www.nxp.com/products/software-and-tools/software-development-tools/i.mx-software-and-tools/i.... 

use fsl-image-gui image described in attached Yocto Guide Table 1. Freescale Yocto project images

and pipelines given in sect.7.3 Multimedia use cases Linux Guide .

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[adamschafer]

Hi Igor,

We used the Linux 3.14.52 BSP as a baseline, then created a custom yocto layer that included gstreamer along with our patches.

I should point out that our imxv4l2sink is reporting the correct caps:

root@imx6dqscm-1gb-hub-11x:/unit_tests# gst-inspect-1.0 imxv4l2sink
====== IMXV4L2SINK: 4.0.9 build on May 16 2017 20:39:59. ======
Factory Details:
Rank primary + 1 (257)
Long-name IMX Video (video4linux2) Sink
Klass Sink/Video
Description Displays frames on IMX SoC video4linux2 device
Author Multimedia Team <shmmmw@freescale.com>

Plugin Details:
Name imxv4l2.imx
Description IMX SoC v4l2-based video source/sink
Filename /usr/lib/gstreamer-1.0/libgstimxv4l2plugins.so
Version 4.0.9
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com

GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstImxV4l2Sink

Implemented Interfaces:
GstVideoOverlay

Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition)
format: { RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]

When I enable debug statements, I've noticed that mxc_vout.c uses vidioc_enum_fmt_vid_out to inform the v4l2 layer of the supported pixel formats.  mxc_v4l2_capture.c similarly supports the VIDIOC_ENUM_FMT command, but it is never called when gstreamer attempts to inspect the node:

[ 31.060573]
In MVC: mxc_v4l_open
[ 31.060586] device name is Mxc Camera
[ 31.060596] In ioctl_g_fmt_cap
[ 31.062189] End of mxc_v4l_open: v2f pix widthxheight 288 x 352
[ 31.062384] End of mxc_v4l_open: crop_bounds widthxheight 720 x 480
[ 31.062390] End of mxc_v4l_open: crop_defrect widthxheight 720 x 480
[ 31.062396] End of mxc_v4l_open: crop_current widthxheight 720 x 480
[ 31.062401] On Open: Input to ipu size is 720 x 480
[ 31.062419] adv7282:ioctl_dev_init
[ 32.043158] ADV7282(ioctl_dev_init): mipi dphy error1 [0x1001010]
[ 32.063141] ADV7282(ioctl_dev_init): Exiting the adv7282 ioctl probe successfully
[ 32.069344] In MVC:mxc_v4l_ioctl: cmd: 80685600
[ 32.069352] 0In MVC: mxc_v4l_do_ioctl 80685600
[ 32.069357] case VIDIOC_QUERYCAP
[ 32.069396] In MVC:mxc_v4l_ioctl: cmd: 80685600
[ 32.069402] 0In MVC: mxc_v4l_do_ioctl 80685600
[ 32.069406] case VIDIOC_QUERYCAP
[ 32.069414] In MVC:mxc_v4l_ioctl: cmd: c02c5651
[ 32.069420] 0In MVC: mxc_v4l_do_ioctl c02c5651
[ 32.069425] case VIDIOC_DBG_G_CHIP_IDENT
[ 32.069447] In MVC:mxc_v4l_ioctl: cmd: 80085617
[ 32.069454] 0In MVC: mxc_v4l_do_ioctl 80085617
[ 32.069459] case VIDIOC_G_STD
[ 32.069464] In mxc_v4l2_g_std
[ 32.069469] In ioctl_g_fmt_cap
[ 32.071380] MVC: mxc_v4l2_g_std: Changing standard
[ 32.071575] In mxc_v4l2_s_std b000
[ 32.071581] Setting standard to NTSC b000
[ 32.071590] In MVC:mxc_v4l_ioctl: cmd: 40085618
[ 32.071596] 0In MVC: mxc_v4l_do_ioctl 40085618
[ 32.071600] case VIDIOC_S_STD
[ 32.071604] In mxc_v4l2_s_std b000
[ 32.071608] Setting standard to NTSC b000
[ 32.071641] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071648] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071654] case VIDIOC_ENUM_FRAMESIZES
[ 32.071661] In MVC:mxc_v4l_ioctl: cmd: c034564b
[ 32.071667] 0In MVC: mxc_v4l_do_ioctl c034564b
[ 32.071670] case VIDIOC_ENUM_FRAMEINTERVALS
[ 32.071678] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071682] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071686] case VIDIOC_ENUM_FRAMESIZES
[ 32.071692] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071696] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071700] case VIDIOC_ENUM_FRAMESIZES
[ 32.071705] In MVC:mxc_v4l_ioctl: cmd: c034564b
[ 32.071710] 0In MVC: mxc_v4l_do_ioctl c034564b
[ 32.071714] case VIDIOC_ENUM_FRAMEINTERVALS
[ 32.071719] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071724] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071728] case VIDIOC_ENUM_FRAMESIZES
[ 32.071733] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071737] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071741] case VIDIOC_ENUM_FRAMESIZES
[ 32.071747] In MVC:mxc_v4l_ioctl: cmd: c034564b
[ 32.071751] 0In MVC: mxc_v4l_do_ioctl c034564b
[ 32.071755] case VIDIOC_ENUM_FRAMEINTERVALS
[ 32.071760] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071765] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071768] case VIDIOC_ENUM_FRAMESIZES
[ 32.071774] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071778] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071782] case VIDIOC_ENUM_FRAMESIZES
[ 32.071787] In MVC:mxc_v4l_ioctl: cmd: c034564b
[ 32.071792] 0In MVC: mxc_v4l_do_ioctl c034564b
[ 32.071796] case VIDIOC_ENUM_FRAMEINTERVALS
[ 32.071801] In MVC:mxc_v4l_ioctl: cmd: c02c564a
[ 32.071805] 0In MVC: mxc_v4l_do_ioctl c02c564a
[ 32.071809] case VIDIOC_ENUM_FRAMESIZES
[ 32.072082] In MVC:mxc_v4l_close
[ 32.072092] mxc_v4l_close: release resource
[ 32.072099] MVC: In mxc_free_frame_buf
[ 32.072105] In MVC:mxc_free_frames

Does it make sense that inspecting imxv4l2src never calls the VIDIOC_ENUM_FMT command?

Thanks for your assistance.

-Adam

[westbrook]

Hi Adam,

Did you find a reason or a solution for the Empty capabilities for imxv4l2src?

I have met the same issue, I used gst-inspect-1.0 to check the imxv4l2src, and I found that the capabilities are EMPTY, which makes camera preview failed:

# gst-launch-1.0 imxv4l2src device=/dev/video0 ! imxv4l2sink

Thank you in advance.

BR,

Xiao

[aschafer]

Hi Xaio,

To get around this problem, I needed to add support for the "VIDIOC_ENUM_FRAMEINTERVALS" ioctl to my driver.  Unfortunately, I don't remember the exact details of why this solution worked, but I'm more than happy to share the code I modified:

I created this function:

/*!
* ioctl_enum_frameintervals - V4L2 sensor interface handler for
* VIDIOC_ENUM_FRAMEINTERVALS ioctl
* @s: pointer to standard V4L2 device structure
* @fival: standard V4L2 VIDIOC_ENUM_FRAMEINTERVALS ioctl structure
*
* Return 0 if successful, otherwise -EINVAL.
*/
static int ioctl_enum_frameintervals(struct v4l2_int_device *s,
struct v4l2_frmivalenum *fival)
{

if (fival->pixel_format == adv7282_device->sen.pix.pixelformat
&& fival->width == adv7282_device->sen.pix.width
&& fival->height == adv7282_device->sen.pix.height
&& fival->index < 1)
{
fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
fival->discrete.numerator = adv7282_device->sen.streamcap.timeperframe.numerator;
fival->discrete.denominator = adv7282_device->sen.streamcap.timeperframe.denominator;
return 0;
}

return -EINVAL;
}

Then I added this entry to the v4l2_int_ioctl_desc structure array for the driver:

{vidioc_int_enum_frameintervals_num, (v4l2_int_ioctl_func *) ioctl_enum_frameintervals}

Sorry I don't have a more precise answer for you!

[westbrook]

Hi Adam,

Good news, I tried to add "VIDIOC_ENUM_FRAMEINTERVALS" ioctl to my driver, and now the gstreamer works:

# gst-launch-1.0 imxv4l2src device=/dev/video0 ! imxv4l2sink

Thank you so much.

BR,

Xiao

[westbrook]

Hi Adam,

Thanks so much for your reply.

I will try on my kernel.

Best Regards,

Xiao