iMX6 Quad plus / gstreamer-1.0 / RAW mode limitation

[gabrielgrange]

We made video test with a specific board with iMX6 Quad Plus and Ethernet Gigabit link. It works under 3.14.52 kernel from yocto Jethro Linux distrib.

We tested our board with a PC that generate video flow thanks to a Windows gstreamer version (from https://gstreamer.freedesktop.org/data/pkg/windows/1.10.2 ).

The configuration is as follow:

In PC side, we use gstreamer Windows version to generate video flow (from https://gstreamer.freedesktop.org/data/pkg/windows/1.10.2 )

We start this command from C:\gstreamer\1.0\x86_64\bin directory:

gst-launch-1.0.exe videotestsrc ! videoconvert ! videoscale ! timeoverlay ! video/x-raw,sampling=YCbCr-4:2:0,width=720, height=576,framerate=60/1 ! queue ! rtpvrawpay -v ! udpsink host=192.10.2.83 port=1234

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-
raw, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I
420, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-m
ode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-
raw, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I
420, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-m
ode=(string)progressive
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:src: caps = video/x-raw,
width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420,
sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode=
(string)progressive
/GstPipeline:pipeline0/GstTimeOverlay:timeoverlay0.GstPad:src: caps = video/x-ra
w, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I42
0, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mod
e=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw,
width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420,
sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode=
(string)progressive
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, width=(i
nt)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420, sampling
=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)p
rogressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw
, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420
, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode
=(string)progressive
/GstPipeline:pipeline0/GstTimeOverlay:timeoverlay0.GstPad:video_sink: caps = vid
eo/x-raw, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(str
ing)I420, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interl
ace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:sink: caps = video/x-raw
, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420
, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode
=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x
-raw, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)
I420, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-
mode=(string)progressive
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw, width=(in
t)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420, sampling=
(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)pr
ogressive
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0.GstPad:src: caps = application/
x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sa
mpling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)720, height=(string)5
76, colorimetry=(string)BT601-5, payload=(int)96, ssrc=(uint)2884138433, timesta
mp-offset=(uint)3456393280, seqnum-offset=(uint)24060, a-framerate=(string)60
/GstPipeline:pipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp
, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, samplin
g=(string)YCbCr-4:2:0, depth=(string)8, width=(string)720, height=(string)576, c
olorimetry=(string)BT601-5, payload=(int)96, ssrc=(uint)2884138433, timestamp-of
fset=(uint)3456393280, seqnum-offset=(uint)24060, a-framerate=(string)60
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0.GstPad:sink: caps = video/x-raw
, width=(int)720, height=(int)576, framerate=(fraction)60/1, format=(string)I420
, sampling=(string)YCbCr-4:2:0, pixel-aspect-ratio=(fraction)1/1, interlace-mode
=(string)progressive
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0: timestamp = 3456393280
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0: seqnum = 24060
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

In iMX6 side, we use these shell commands :

ifconfig enp1s0 192.10.2.83 netmask 255.255.252.0

then

gst-launch-1.0 udpsrc uri=udp://192.10.2.83:1234 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW,sampling=YCbCr-4:2:0, depth=(string)8, width=(string)720, height=(string)576, payload=(int)96" ! queue ! rtpvrawdepay !  autovideosink -v

Setting pipeline to PAUSED ...

display(/dev/fb0) resolution is (1024x768).

====== OVERLAYSINK: 4.0.8 build on Jul 13 2016 15:19:55. ======

display(/dev/fb0) resolution is (1024x768).

display(/dev/fb0) resolution is (1024x768).

Pipeline is live and does not need PREROLL ...

/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = "application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)RAW\,\ sampling\=\(string\)YCbCr-4:2:0\,\ depth\=\(string\)8\,\ width\=\(string\)720\,\ height\=\(string\)576\,\ payload\=\(int\)96"

/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = "application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)RAW\,\ sampling\=\(string\)YCbCr-4:2:0\,\ depth\=\(string\)8\,\ width\=\(string\)720\,\ height\=\(string\)576\,\ payload\=\(int\)96"

/GstPipeline:pipeline0/GstRtpVRawDepay:rtpvrawdepay0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)0/1"

Setting pipeline to PLAYING ...

/GstPipeline:pipeline0/GstRtpVRawDepay:rtpvrawdepay0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)0/1"

New clock: GstSystemClock

/GstPipeline:pipeline0/GstImxIpuVideoTransform:imxipuvideotransform0.GstPad:src: caps = "video/x-raw\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)0/1\,\ format\=\(string\)I420"

/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = "video/x-raw\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)0/1\,\ format\=\(string\)I420"

/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstOverlaySink:autovideosink0-actual-sink-overlay.GstPad:sink: caps = "video/x-raw\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)0/1\,\ format\=\(string\)I420"

/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink: caps = "video/x-raw\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)0/1\,\ format\=\(string\)I420"

/GstPipeline:pipeline0/GstImxIpuVideoTransform:imxipuvideotransform0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)720\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)0/1"

/GstPipeline:pipeline0/GstRtpVRawDepay:rtpvrawdepay0.GstPad:sink: caps = "application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)RAW\,\ sampling\=\(string\)YCbCr-4:2:0\,\ depth\=\(string\)8\,\ width\=\(string\)720\,\ height\=\(string\)576\,\ payload\=\(int\)96"

• As a result we obtain a wrong iMX6 video display with many abnormal black lines despite CPU usage never exceed 100% and Ethernet bandwith is under 39Mb/sec.
• Is something is wrong in my iMX6 RAW decoding or is there are gstreamer limit with RAW format?

Regards,

Gabriel GRANGE

[gabrielgrange]

We complete our tests with iMX6 quad plus without udp transfert. We operate with CPU frequency at 792MHz.

If we generate 720x576x60fps YUV4.2.0 raw video flow (35Mo/s bandwith is need in this mode) :

gst-launch-1.0 -v videotestsrc pattern=ball ! video/x-raw,sampling=YCbCr-4:2:0,width=720, height=576,framerate=60/1 ! queue ! rtpvrawpay ! udpsink host=127.0.0.1 port=1234  &

Then if we read and display in local the video flow :

gst-launch-1.0 udpsrc uri=udp://127.0.0.1:1234 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW,sampling=YCbCr-4:2:0, depth=(string)8, width=(string)720, height=(string)576, colorimetry=(string)BT601-5, payload=(int)96" ! queue ! rtpvrawdepay ! imxipuvideosink

We have a display problem that don't seem link with CPU charge.

The result is better if we add a large buffer to receive data flow, it takes several ten seconds to generate display problem :

gst-launch-1.0 udpsrc buffer-size=900000000 uri=udp://127.0.0.1:1234 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW,sampling=YCbCr-4:2:0, depth=(string)8, width=(string)720, height=(string)576, colorimetry=(string)BT601-5, payload=(int)96" ! queue2 ! rtpvrawdepay ! decodebin ! imxipuvideosink

I don't understand why we have a problem with iMX6 performance or software gstreamer-1.0 management. How can i improve my research ?

Regards,

Gabriel Grange

[Bio_TICFSL]

Hi Gabriel,

For gstreamer mechanism and some useful samples for streaming raw videos and images, please check:

Yocto/gstreamer/streaming – Gateworks 

Hope this helps

[gabrielgrange]

Thanks for your answer.
I read these samples but i didn't found specific point to manage RAW format decoding with iMX6 or a way to improve iMX6 display process.

Regards,

Gabriel