Audio / Video trough Camera stream and gstreamer on i.MX6DQS and i.MX8

[#1]

This note show how to use the open source gstreamer1.0-rtsp-server package on i.MX6QDS and i.MX8x to stream video files and camera using RTP protocol.  The i.MX 6ULL and i.MX 7 doesn't have Video Processing Unit (VPU).

Real Time protocol is a very common network protocol for delivering media over IP networks. On the board, you will need a GStreamer pipeline that encodes the raw video, adds the RTP payload, and sends over a network sink. A generic pipeline would look as follows:

video source ! video encoder ! RTP payload ! network sink

• Video source: often it is a camera, but it can be a video from a file or a test pattern, for example.
• Video encoder: a video encoder as H.264, H.265, VP8, JPEG and others.
• RTP payload: an RTP payload that matches the video encoder.
• Network sink: a video sync that streams over the network, often via UDP.
•  
• Prerequisites:
• MX6x o MX8x board with the L5.10.35 BSP installed.
• A host PC with either Gstreamer or VLC player installed.

Receiving h.264/h.265 Encoded RTP Video Stream on a Host Machine

Using GStreamer

GStreamer is a low-latency method for receiving RTP video. On your host machine, install Gstreamer and send the following command:

$ gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink sync=false

 

Using Host PC: VLC Player

Optionally, you can use VLC player to receive RTP video on a PC. First, in your PC, create a sdp file with the following content:

 stream.sdpv=0m=video 5000 RTP/AVP 96c=IN IP4 127.0.0.1a=rtpmap:96 H264/90000

After this, with the GStreamer pipepline on the device running, open this .sdp file with VLC Player on the host PC.

Sending h.264 and h.265 Encoded RTP Video Stream

GStreamer provides an h.264 encoding element by software named x264enc. Use this plugin if your board does not support h.264 encoding by hardware or if you want to use the same pipeline on different modules. Note that the video performance will be lower compared with the plugins with encoding accelerated by hardware.

# gst-launch-1.0 videotestsrc ! videoconvert ! x264enc ! rtph264pay config-interval=1 pt=96 ! udpsink host=<host-machine-ip> port=5000

Note: Replace <host-machine-ip> by the IP of the host machine. In all examples you can replace videotestsrc by v4l2src element to collect a stream from a camera

i.MX8X

# gst-launch-1.0 videotestsrc ! videoconvert ! v4l2h264enc ! rtph264pay config-interval=1 pt=96 ! udpsink host=<host-machine-ip> port=5000

i.MX 8M Mini Quad/ 8M Plus

# gst-launch-1.0 videotestsrc ! videoconvert ! vpuenc_h264 ! rtph264pay config-interval=1 pt=96 ! udpsink host=<host-machine-ip> port=5000 i.MX6X

The i.MX6QDS does not support h.265 so the h.264 can work:

# gst-launch-1.0 videotestsrc ! videoconvert ! vpuenc_h264 ! rtph264pay config-interval=1 pt=96 ! udpsink host=<host-machine-ip> port=5000

Using Other Video Encoders

While examples of streaming video with other encoders are not provided, you may try it yourself. Use the gst-inspect tool to find available encoders and RTP payloaders on the board:

# gst-inspect-1.0 | grep -e "encoder"# gst-inspect-1.0 | grep -e "rtp" -e " payloader"

Then browse the results and replace the elements in the original pipelines.

On the receiving end, you will have to use a corresponding payload. Inspect the payloader element to find the corresponding values. For example:

# gst-inspect-1.0 rtph264pay

Install rtp in your yocto different form L5.10.35 BSP, to install gstreamer1.0-rtsp-server in any Yocto Project image, please follow the steps below:

1. Enable meta-multimedia layer:
   Add the following on your build/conf/bblayers.conf:

2. BBLAYERS += "$"${BSPDIR}/sources/meta-openembedded/meta-multimedia"

1. Include gstreamer1.0-rtsp-server into the image:
   Add the following on your build/conf/local.conf:

4. IMAGE_INSTALL_append += "gstreamer1.0-rtsp-server"

1. Run bitbake and mount your sdcard.
2. Copy the binaries:
   Access the gstreamer1.0-rtsp-server examples folder:

7. $ cd /build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer1.0-rtsp-server/$version/build/examples/.libs

Copy the test-uri and test-launch to the rootfs /usr/bin folder.

$ sudo cp test-uri test-launch /media/USER/ROOTFS_PATH/usr/bin

Be sure that the IPs are correctly set:

• SERVER:

• => ifconfig eth0 $SERVERIP

• CLIENT:

• => ifconfig eth0 $CLIENTIP

Video file example

• SERVER:

• => test-uri file:///home/root/video_file.mp4

• CLIENT:

• => gst-launch-1.0 playbin uri=rtsp://$SERVERIP:8554/test

You can try to improve the framerate performance using manual pipelines in the CLIENT with the rtspsrc plugin instead of playbin. Follow an example:

=> gst-launch-1.0 rtspsrc location=rtsp://$SERVERIP:8554/test caps = 'application/x-rtp'  ! queue max-size-buffers=0 ! rtpjitterbuffer latency=100 ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! decodebin ! queue max-size-buffers=0 ! imxv4l2sink sync=false

Camera example

• SERVER:

• => test-launch "( imxv4l2src device=/dev/video0 ! capsfilter caps='video/x-raw, width=1280, height=720, framerate=30/1, mapping=/test' ! vpuenc_h264 ! rtph264pay name=pay0 pt=96 )"

• CLIENT:

• => gst-launch-1.0 rtspsrc location=rtsp://$SERVERIP:8554/test ! decodebin ! autovideosink sync=false

The rtspsrc has two properties very useful for RTSP streaming:

• Latency: Useful for low-latency RTSP stream playback (default 200 ms);
• Buffer-mode: Used to control buffer mode. The slave mode is recommended for low-latency communications.

Using these properties, the example below gets 29 FPS without a sync=false property in the sink plugin. The key achievement here is the fact that there is no dropped frame:

=> gst-launch-1.0 rtspsrc location=rtsp://$SERVERIP:8554/test latency=100 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! vpudec ! imxv4l2sink