AVB/TSN demo on i.MX8MP

[#1]

• Overview
• AVB/TSN
• Demo introduction
• Traffic control
• Credit Base Shaper (CBS)
• Enhancements to Scheduled Traffic (EST)
• Build demo
• Build L5.4.24_2.1.0
• Rebuild kernel
• Install the toolchain
• Create a install folder
• Build libavtp
• Build ALSA aaf plugin
• Build Gstreamer AVTP plugins
• Build Gstreamer 1.17.x (commit e4f7cdb0df7b)
• Build gst-plugins-base (commit 22827e8f)
• Build gst-plugins-bad (commit ed14e0d5a)
• Build H.264 SW plugins
• Build x264
• Build gst-plugins-ugly (commit 995a135d)
• Build libav
• Install binaries
• System Setup
• VLAN
• Qdiscs
• MQPRIO qdisc
• TAPRIO qdisc
• CBS qdisc
• ETF qdisc
• TimeSync
• Run demo
• ALSA AAF audio
• Gstreamer AAF audio
• Gstreamer CVF video
• Deep dive
• Packet sniffer
• Precise latency measurement
• References

Overview

AVB/TSN

Wikipedia: Audio Video Bridging (AVB) is a common name for the set of technical standards which provide improved synchronization, low-latency, and reliability for switched Ethernet networks. AVB was initially developed by the Institute of Electrical and Electronics Engineers (IEEE) Audio Video Bridging task group of the IEEE 802.1 standards committee. In November 2012, Audio Video Bridging task group was renamed to Time-Sensitive Networking (TSN) task group to reflect the expanded scope of its work, which is to "provide the specifications that will allow time-synchronized low latency streaming services through IEEE 802 networks". Further standardization efforts are ongoing in IEEE 802.1 TSN task group.

AVB and TSN technologies and standards:

Standard	Area of Definition	Title of Standard	AVB/TSN
IEEE 802.1AS	Timing and synchronization	Timing and Synchronization for Time-Sensitive Applications (gPTP)	AVB TSN
IEEE 802.1Qav	Forwarding and queuing	Forwarding and Queuing for Time-Sensitive Streams (FQTSS)	AVB
IEEE 802.1Qat	Path control and reservation	Stream Reservation Protocol (SRP)	AVB
IEEE 802.1BA	Bridging	Audio Video Bridging (AVB) Systems	AVB
IEEE 1722 AVTP	AV transport	Layer 2 Transport Protocol for Time Sensitive Applications	AVB
IEEE 1722 AVDECC	Device manage and control	Device Discovery, Enumeration, Connection Management and Control Protocol	AVB
IEEE 802.1Qbu and IEEE 802.3br	Forwarding and queuing	Frame preemption	TSN
IEEE 802.1Qbv	Forwarding and queuing	Enhancements for scheduled traffic	TSN
IEEE 802.1Qca	Path control and reservation	Path control and reservation	TSN
IEEE 802.1Qcc	Central configuration method	Enhancements and performance improvements	TSN
IEEE 802.1Qci	Time-based ingress policing	Per-stream filtering and policing	TSN
IEEE 802.1CB	Seamless redundancy	Frame replication and elimination for reliability	TSN

Since many of the standards are only for TSN switch/bridges and i.MX8MP is design to be a TSN/AVB endpoint, this demo does not implement a full stack or full standards. It only demonstrates the basic end-to-end point (talker to listener) A/V streaming without bridge or switch.

Below table shows what this demo supports:

Standard	Software	Hardware
IEEE 802.1AS	Linuxptp	ENET_QoS IP
IEEE 802.1Qav/Qbu/Qbv	TC qdisc (taprio, mqprio, etf, cbs)	ENET_QoS IP (multi queue + EST, FPE, CBS)
IEEE 1722 AVTP	Libavtp	N/A

Demo introduction

Two i.MX8MP EVK boards are used for this demo, one act as a AVB talker to send A/V streams, the other one act as a AVB listener to receive A/V streams who can be playback to audio codec and sink video to screen. The two boards are connected by a RJ45 ethernet cable on each ENET2 port (only ENET2 port has TSN features).

Three streams’ type and SR (Stream Reservation) class are defined as below to grantee time sensitive (sub-microsecond synchronization), low latency and bandwidth on the ethernet:

• Stream A: SR class A, AVTP Compressed Video Format, H.264 profile High, 1920x1080, 30 fps.
• Stream B: SR class B, AVTP Audio Format, PCM 16-bit sample, 48 kHz, stereo, 12 frames per AVTPDU.
• Other stream: Best-effort streams

These three TSN streams would be allocated into different traffic control (TC) class for egress in Linux. Different TC class would be mapped to different hardware queues with dedicated DMA channel, thanks to the multi-queue support by ENET_QoS IP. Then the traffic control apply different scheduling policy on each queues for different types of streams egress, to make sure highest priority or critical packet can be sent in the correct time slot.

The TSN streams are transmitted over Virtual LANs (VLANs). Bridges use the VLAN priority information (PCP) to identify Stream Reservation (SR) traffic classes which are handled according to the Forward and Queuing Enhancements for Time-Sensitive Streams (FQTSS) mechanisms described in Chapter 34 of the IEEE 802.1Q standard.

The demo is built up by following blocks:

• Linux TC (traffic control): streams egress control to meet AVB/TSN requirements, which take advantage of the i.MX8MP TSN ENET IP.
• Linux PTP: clock sync in network, which take advantage of the i.MX8MP TSN ENET IP.
• Libavtp: Time Sensitive Applications AV Transport protocol.
• ALSA: AVTP audio format plugin uses the libavtp to transmit and receive AVTP audio PCM streams.
• Gstreamer: avtp plugin uses the libavtp to transmit and receive AVTP audio/video streams (video should be encoded as H264, audio PCM). x264enc and libav is a software H.264 video encoder/decoder, which alternative to the on chip VPU acceleration plugin, due to the VPU plugin is not supported in the latest Gstreamer version (1.17.x).

 

Traffic control

This demo can use two different traffic control qdisc settings:

• mqprio + cbs: use CBS features
• taprio + pfifo: use EST and FPE features (802.1Qbu/bv).

The pfifo is the default traffic control class, which use FIFO schedule policy for egress packets. The CBS class is actually handled by hardware IP to select which queue for transmit in a certain time slot.

Credit Base Shaper (CBS)

CBS parameters come straight from the IEEE 802.1Q-2018 specification. They are the following:

• idleSlope: rate credits are accumulated when queue isn’t transmitting;
• sendSlope: rate credits are spent when queue is transmitting;
• hiCredit: maximum amount of credits the queue is allowed to have;
• loCredit: minimum amount of credits the queue is allowed to have;

 

Enhancements to Scheduled Traffic (EST)

The IEEE 802.1Qbv defines the schedule for each of the queues on every egress port which makes the implementation aware of traffic arrival schedule. This information can be used to block the lower priority traffic from transmission in this time window/slot. This ensures that scheduled traffic is forwarded from sender to receiver through all the network nodes with a deterministic delay. The i.MX8MP uses the gate control list with configurable time slot and frame preempt (IEEE 802.1Qbu) features to support EST.

Other than the CBS, the gate control list control the egress transmission by a fixed open interval for that queue.

Build demo

Build L5.4.24_2.1.0

$MACHINE=imx8mpevk DISTRO=fsl-imx-xwayland source imx-setup-release.sh -b build-8mp

$bitbake  fsl-image-validation-imx

Prepare a SD card and burn it with the built out images.

To add tc/tcpdump command in iproute2 package, please add package into IMAGE_INSTALL_append variable into conf/local.conf:

IMAGE_INSTALL_append = " iproute2 iproute2-tc tcpdump

Rebuild kernel

Rebuild the kernel after applying the 0001-qenet-add-queue-avoid-panic.patch (attached), and overwrite the Image and imx8mp-evk.dtb on the boot partition of the SD card.

Install the toolchain

$bitbake -f fsl-image-validation-imx -c populate_sdk

$sh tmp/deploy/sdk/fsl-imx-xwayland-glibc-x86_64-fsl-image-validation-imx-aarch64-imx8mpevk-toolchain-5.4-zeus.sh

The toolchain would be installed into /opt/fsl-imx-xwayland/5.4-zeus/

Create a install folder

$mkdir <your install folder>

Create a folder to install all of the shared libraries, binaries and configure files which built out manually in this doc. After built done, you should copy all of the contents in this folder to target board root.

Build libavtp

$source /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux

$git clone https://github.com/Avnu/libavtp.git

$cd libavtp

$meson build

$ninja -C build

Copy the built out .so and .pc into the toolchain rootfs:

$sudo cp build/libavtp.so* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/lib/

$sudo cp build/meson-private/*.pc /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/lib/pkgconfig/

To make sure you have avtp package installed correctly:

$pkg-config --list-all | grep avtp

Copy the .so into the install folder:

$cp build/libavtp.so* <install folder>/usr/lib/

Copy header files:

$cp libavtp/include/* files <toolchain_path>/sysroots/aarch64-poky-linux/usr/lib

Build ALSA aaf plugin

The AAF (ALSA AVTP Audio Format) plugin is a PCM plugin that uses Audio Video Transport Protocol (AVTP) to transmit/receive audio samples through a Time-Sensitive Network (TSN) capable network. The plugin enables media applications to easily implement AVTP Talker and Listener functionalities.

$cd <yocto build>/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/alsa-plugins-1.1.9

$./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=<your install folder>/usr --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot  --disable-static  --enable-aaf --disable-jack --disable-libav --disable-maemo-plugin --disable-maemo-resource-manager --enable-pulseaudio --enable-samplerate --with-speex=lib

$make

$make install

 

Build Gstreamer AVTP plugins

Build Gstreamer 1.17.x (commit e4f7cdb0df7b)

$git clone https://gitlab.freedesktop.org/gstreamer/gstreamer.git

$cd gstreamer

$patch -p1 < gstreamer-1.0-pass-build.patch

$meson build --prefix=<your install folder>/usr

$ninja -C build

$ninja -C build install

After Gstreamer is installed into <your install folder>, please fix the “prefix” path in the .pc files by, and copy to the toolchain folders:

$cd <your install folder>

$grep -lR <your install folder> ./lib/pkgconfig/ | xargs sed -i 's/<your install folder>/\/usr/g'

NOTE: Make sure you use the ‘\’ for ‘/’ convert in your path.

$cp -rf ./usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/

 

Build gst-plugins-base (commit 22827e8f)

$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-base.git

$cd gst-plugins-base

$patch -p1 < gst-plugins-base-pass-build.patch

$meson build --prefix=<your install folder>/usr

$ninja -C build

$ninja -C build install

 

Build gst-plugins-bad (commit ed14e0d5a)

$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad.git

$cd gst-plugins-bad

$meson build --prefix=<your install folder>/usr

$ninja -C build

$ninja -C build install

• If you met gdbus-codegen issue, please remove the “--c-generate-autocleanup” and “--output-directory” parameters in the build/build.ninja
• If you met issue of include file like: “…/gst/gl/gstglapi.h:24:10: fatal error: gst/gl/gstglconfig.h: No such file or directory.” Please modify the build/build.ninja to correct the -I header file parameter of the build.

After gst-plugins-base and gst-plugins-bad installed into <your install folder>, please fix the “prefix” path in the .pc files and copy them into the toolchain folders:

$cd <your install folder>

$grep -lR <your install folder> ./lib/pkgconfig/ | xargs sed -i 's/<your install folder>/\/usr/g'

NOTE: Make sure you use the ‘\’ for ‘/’ convert in your path.

$cp -rf ./usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/

Build H.264 SW plugins

AVTPDU uses H.264 as payload for streaming, this requires H.264 encoder/decoder plugins, either software or hardware accelerations. Since we upgrade the Gstreamer and its plugins to a new version, the VPU plugins cannot be used anymore. So software H.264 plugins are required: x264 for encoder, libav for decoder.

Build x264

As the yocto actually has the x264 recipes, but not included in our bblayers, we need to copy the x264 source into our bblayers path under <yocto>/source to build:

$cp -rf ./poky/meta/recipes-multimedia/x264 ./meta-openembedded/meta-multimedia/recipes-multimedia/

$vi ./meta-openembedded/meta-multimedia/recipes-multimedia/x264_git.bb

Remove the LICENSE_FLAGS line

$bitbake -f x264 -c do_install

$sudo cp -rf tmp/work/aarch64-poky-linux/x264/r2917+gitAUTOINC+72db437770-r0/image/usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/

Build gst-plugins-ugly (commit 995a135d)

$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-ugly.git

$cd gst-plugins-ugly$meson build --prefix=<your install folder>/usr

$ninja -C build

$ninja -C build install

If you met build issue, please remove "if have_cxx" from meson.build

Build libav

As the yocto actually has the libav recipes, but not included in our bblayers, we need to copy the its source into our bblayers path under <yocto>/source:

$cp -rf ./poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-libav ./meta-openembedded/meta-multimedia/recipes-multimedia/gstreamer

$vi ./meta-openembedded/meta-multimedia/recipes-multimedia/gstreamer/gstreamer1.0-libav_1.16.1.bb

Remove the LICENSE_FLAGS line

$bitbake -f gstreamer1.0-libav -c do_install

$cp tmp/work/aarch64-mx8mp-poky-linux/gstreamer1.0-libav/1.14.0-r0/image/usr/lib/gstreamer-1.0/libgstlibav.so <your install folder>/usr/lib/gstreamer-1.0

Now you have all of the Gstreamer plugins which required for AVB/TSN audio/video demo:

• avtpsrc, avtpsink, avtpaafpay, avtpaafdepay, avtpcvfpay, avtpcvfdepay
• x264_enc (encoder), avdec_h264 (decoder)

Install binaries

Final step is to copy all of your built out files from <your install folder> into your board / root, and boot up the board.

Verify the Gstreamer plugins install correctly or not:

$export GST_PLUGIN_PATH= /usr/lib/gstreamer-1.0/

$gst-inspect-1.0

Check if the above Gstreamer plugins we built out can be found by gst-instpect.

System Setup

VLAN

The ENTE_QoS is assigned to eth1 instance. So create eth1.5 for vlan id 5:

$ip link add link eth1 name eth1.5 type vlan id 5 \ 

       egress-qos-map 2:2 3:3

$ip link set eth1.5 up

Qdiscs

The TSN control plane is implemented through the Linux Traffic Control (TC) System. The transmission algorithms specified in the Forwarding and Queuing for Time-Sensitive Streams (FQTSS) chapter of IEEE 802.1Q-2018 are supported via TC Queuing Disciplines (qdiscs). Linux currently provides the following qdiscs relating to TSN:

• Multiply queue qdiscs (These two qdiscs are mutually exclusive, select one for your demo):
  • MQPRIO: Mapping the TC class into different hardware queue in IP.
  • TAPRIO: Implements the Enhancements for Scheduled Traffic introduced by IEEE 1Qbv/Qbu. This is supported by i.MX8MP ENET_QoS IP through EST features:
    • Qbv – Time aware shaper
    • Qbu - frame preemption.
• CBS qdisc: Implements the Credit-Based Shaper introduced by the IEEE 802.1Qav This is supported by i.MX8MP ENET_QoS IP through EST (Enhancements to Scheduled Traffic) features. It works with mqprio qdisc together.
• ETF qdisc: While not an FQTSS feature, Linux also provides the Earliest TxTime First (ETF) qdisc which enables the LaunchTime Since the i.MX8MP ENET_QoS IP does not support LaunchTime feature, this qdisc configurations would be ignored.

 

MQPRIO qdisc

$tc qdisc add dev eth1 parent root handle 100 mqprio \

        num_tc 3 \

        map 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 \

        queues 1@0 1@1 1@2 \

        hw 1

NOTE, since ENET_QoS Q0 does not support hardware CBS, we have to avoid using Q0 for AVB streaming. Here’s the mapping:

• socket SO_PRIORITY 2 -> TC CLASS 2 -> Q2
• socket SO_PRIORITY 3 -> TC CLASS 1 -> Q1
• Other socket priority -> TC CLASS 0 -> Q0

 

TAPRIO qdisc

Create a root qdisc handle to map the different CLASS streams to hardware queues w/ GCL (gate control list). This root handle maps the CLASS A stream to queue Q1, CLASS B stream to Q2, and others to Q0 by the “map” and “queues” parameters, same as mqprio above. The Q1/Q2(CLASS1/2) gates are open in the first 300us interval, then only Q1(CLASS1) gate is open in the next 300us with all other CLASS gated. The last sched-entry defines in after Q1 is open after 300us, all of the CLASS gates are open for next 200us. Then loopback to the first sched-entry for gate control.

Please note in this demo, we only enable features to support the 802.1Qbv, the Qbu (Frame preempt) requires patches for iproute2.

$tc qdisc add dev eth1 parent root handle 100 taprio \

        num_tc 3 \

        map 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 \

        queues 1@0 1@1 1@2 \

        base-time 1000000000 \

        sched-entry S 0x6 300000 \

        sched-entry S 0x2 300000 \

        sched-entry S 0x7 200000 \

        flags 0x2

 

CBS qdisc

Q1 CBS for video, Q2 CBS for audio:

$tc qdisc replace dev eth1 parent 100:2 handle 777 cbs \

        idleslope 98688 sendslope -901312 hicredit 153 locredit -1389 \

        offload 1

$tc qdisc replace dev eth1 parent 100:3 handle 888 cbs \

        idleslope 3648 sendslope -996352 hicredit 12 locredit -113 \

       offload 1

NOTE: By default, the Q0 will be allocated for pfifo qdisc class if we do not define them.

 

ETF qdisc

As ENET_QOS does not support hardware launch time in IP, the ETF qdisc would not be used here.

 

TimeSync

Run the ptp4l and phc2sys in background, and use check_clocks to check the ptp sync works.

$ptp4l -i eth1 -f ./gPTP.cfg --step_threshold=1 &

$pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 \

        clockAccuracy 0xfe offsetScaledLogVariance 0xffff \

        currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 \

        ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 \

        timeSource 0xa0"

$phc2sys -s eth1 -c CLOCK_REALTIME --step_threshold=1 \

        --transportSpecific=1 -w & 

$./check_clocks -d eth1

The PTP Hardware Clock (PHC)  synced between PTP master/slave means the RMS offset between PHC and GM (master clock) is < 100ns. PHC and system clock (CLOCK_REALTIME) synced means the clock offset < 100ns

NOTE:

• The check_clocks source code can be downloaded from here.
• cfg is described below, if you want to identify which is master and which is slave, use “masterOnly 1” or “slaveOnly 1” in this configuration file.

#                                                                

# 802.1AS example configuration containing those attributes which

# differ from the defaults.  See the file, default.cfg, for the  

# complete list of available options.                             

#                                                                

[global]                                                         

gmCapable               1                                        

priority1               248                                       

priority2               248                                      

logAnnounceInterval     0                                        

logSyncInterval         -3                                       

syncReceiptTimeout      3                                         

neighborPropDelayThresh 800                                      

min_neighbor_prop_delay -20000000                                

assume_two_step         1                                        

path_trace_enabled      1                                        

follow_up_info          1                                        

transportSpecific       0x1                                       

ptp_dst_mac             01:80:C2:00:00:0E                        

network_transport       L2                                       

delay_mechanism         P2P                              

Run demo

ALSA AAF audio

To run the alsa AAF demo, please add aaf0 and converter0 plugin device into /etc/asound.conf:

pcm.aaf0 {

        type aaf

        ifname eth1.5

        addr 01:AA:AA:AA:AA:AA

        prio 2

        streamid AA:BB:CC:DD:EE:FF:000B

        mtt 50000

        time_uncertainty 1000

        frames_per_pdu 12

        ptime_tolerance 100

} 

pcm.converter0 {

        type linear

        slave {

                pcm "hw:2,0"

                format S16_LE

        }

}

The “aaf0” plugin device defines the ethernet interface which AAF runs on, the socket priority which mapping to Traffic Class in kernel TC, the stream-id for the aaf streaming.

The “converter0” plugin device is used for convert the S16_BE format to S16_LE for the wm8960 PCM audio.

Select one device as AVB talker, and run:

$speaker-test -p 25000 -F S16_BE -c 2 -r 48000 -D aaf0

Select one device as AVB talker, and run:

$arecord -F 25000 -t raw -f S16_BE -c 2 -r 48000 -D aaf0 | aplay -F 25000 -t raw -f S16_BE -c 2 -r 48000 -D converter0

You can hear the sound on the listener device. You can also check which qdisc queue is used for AAF by:

$tc -s qdisc

Gstreamer AAF audio

Select one device as AVB talker, and run:

$gst-launch-1.0 clockselect. \( clock-id=realtime \

    audiotestsrc samplesperbuffer=12 is-live=true ! \

    audio/x-raw,format=S16BE,channels=2,rate=48000 ! \

    avtpaafpay mtt=50000000 tu=1000000 streamid=0xAABBCCDDEEFF000B processing-deadline=0 ! \

    avtpsink ifname=eth1.5 address=01:AA:AA:AA:AA:AA priority=2 processing-deadline=0 \)

Select one device as AVB listener, and run:

$gst-launch-1.0 clockselect. \( clock-id=realtime \

    avtpsrc ifname=eth1.5 ! avtpaafdepay streamid=0xAABBCCDDEEFF000B ! \

    queue max-size-bytes=0 max-size-buffers=0 max-size-time=0 ! \

    audioconvert ! audioresample !  alsasink device="hw:2,0" \)

 

Gstreamer CVF video

Select one device as AVB talker, and run:

$gst-launch-1.0 clockselect. \( clock-id=realtime \

    videotestsrc is-live=true ! video/x-raw,width=480,height=320,framerate=15/1 ! \

    clockoverlay ! x264enc bframes=0  key-int-max=1 speed-preset=1 tune=4 ! h264parse config-interval=-1 ! \

    avtpcvfpay processing-deadline=20000000 mtu=1400 mtt=2000000 tu=125000 streamid=0xAABBCCDDEEFF000A ! \

    avtpsink ifname=eth1.5 priority=3 processing-deadline=20000000 \)

NOTE:

• To eliminate the h.264 software encoding/decoding overhead with acceptable latency for this demo, we use several parameters for x264enc element:
  • bframes: x264enc and avdec_h264 together was found to have issues, remove bframes in the stream would help.
  • key-int-max: decoder can only decode the frame when a keyframe is present on stream to make sure decoder can work as faster as it can, the distance between two keyframes must be set to closest.
  • speed-preset: to low down the CPU loading for encoding, we use the option to make encoding as faster as we can. (=1 means ultrafast)
  • tune: 4 means zero latency
• ‘mtu=1400’ parameters for avtpcvfpay element is very important, if using the default mtu=1500, the listener cannot get the AVTPDUs package correctly from VLAN. The reason is unknown yet.

Select one device as AVB listener, and run:

$gst-launch-1.0 clockselect. \( clock-id=realtime \

    avtpsrc ifname=eth1.5 ! avtpcvfdepay streamid=0xAABBCCDDEEFF000A ! \

    queue max-size-bytes=0 max-size-buffers=0 max-size-time=0 ! \

    avdec_h264 ! videoconvert ! clockoverlay halignment=right ! waylandsink \)

The demo screenshot below: there are two clocks show on the videotestsrc stream: left one is the timestamp recorded before x264enc encoding on the AVB talker side, right one is the timestamp recorded after avdec_h265 decoding and do video convert to YUV frames on AVB listener side. You can see the timestamp is sync in seconds.

 

Deep dive

Packet sniffer

Use tcpdump on board to dump the L2 ethernet packet:

./tcpdump -i eth1 ether proto 0x22f0 -w dump.pcap

The AVTP ether protocol code is 0x22f0 embedded inside the ether frame, or you can use "vlan 5" VLAN id for tcpdump parameters to dump.

Then open this dump.pcap in the windows/Linux PC by the wireshark tool, it will automatically show the protocol inside the package, it can also parser the IEEE1722 (AVTP) CVF/AFF package header as below:

Precise latency measurement

• The clockoverlay plugin used in the above talker/listener is actually seconds level precision, which can not reflect the latency from talker videotestsrc -> listener avdec_h264 decoding finish. Here need a little hack to the clockoverlay element in the gst-plugins-base to get the millisecond precision. The patch is attached (gst-plugins-base-clockoverlay-us.patch), please apply and rebuild the gst-plugins-base, then replace the libgstpango.so on the board /usr/lib/gstreamer-1.0/. When doing CVF demo, you can take a picture of the screen, and check the two clock's diff. During my test, the latency is about 50ms, which include all the cost of encoding, AVTP packaging, streaming, ethernet transmit, ethernet receive, AVTP unpack and frame decoding.

• To measure the package latency from transmit port (talker) to receive port (listener), you can use the tcpdump on both end-points. And compare the Epoch Time the packet dumped: "Epoch Time: 1596252905.688243000 seconds". The delta of the epoch time of the same packet is around 100us~500us. This latency actually includes the AF_PACKET clone cost in kernel netfilter, also the tcpdump application schedule latency.

References

• Getting started with AVB on Linux: https://tsn.readthedocs.io/avb.html
• TSN vs AVB: https://dornerworks.com/blog/avb-vs-tsn-choose-the-best-deterministic-ethernet-solution-for-your-net...
• ALSA aaf: https://fossies.org/linux/alsa-plugins/doc/aaf.txt
• Gstreamer avtp: https://gstreamer.freedesktop.org/documentation/avtp/index.html
• AVTP: https://avnu.org/wp-content/uploads/2014/05/AVnu-AAA2C_Audio-Video-Transport-Protocol-AVTP_Dave-Olse...
• MX8MP RM

[MengJu_Lin]

@xinyu_chen ,

When build ALSA aaf plugin.

"$./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=<your install folder>/usr --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot  --disable-static  --enable-aaf --disable-jack --disable-libav --disable-maemo-plugin --disable-maemo-resource-manager --enable-pulseaudio --enable-samplerate --with-speex=lib

"

When define --with-libtool-sysroot, 

--with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot

Can I interpret it as "<yocto build>/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot "?

[OliverSJ]

@xinyu_chen 

How can i test with local mp4 video file?

I ran below cmd:

gst-launch-1.0 clockselect. \( clock-id=realtime \
    filesrc location=sintel_trailer-480p.mp4 ! qtdemux ! h264parse config-interval=-1 ! \
    avtpcvfpay processing-deadline=20000000 mtu=1400 mtt=2000000 tu=125000 streamid=0xAABBCCDDEEFF000A ! \
    avtpsink ifname=eth1.5 priority=3 processing-deadline=20000000 \)

But, i got error:

Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: DebugGstSystemClock
Got EOS from element "clockselect0".
Execution ended after 0:00:00.544463000
Setting pipeline to NULL ...
Freeing pipeline ...

with 

basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc0> pad not activated yet

[brady3]

@xinyu_chen 

It seems that files changed in gstreamer-1.0-pass-build.patch are not the same as them under gstreamer path with cloning https://gitlab.freedesktop.org/gstreamer/gstreamer.git

For example, data/meson.build doesn't exist under gstreamer path.

diff --git a/data/meson.build b/data/meson.build
index c9130e0..3477dfe 100644
--- a/data/meson.build
+++ b/data/meson.build
@@ -1,5 +1,5 @@
if (bashcomp_found)
- subdir('bash-completion/helpers')
- install_data('bash-completion/completions/gst-launch-1.0', install_dir : bash_completions_dir)
- install_data('bash-completion/completions/gst-inspect-1.0', install_dir : bash_completions_dir)
+# subdir('bash-completion/helpers')
+# install_data('bash-completion/completions/gst-launch-1.0', install_dir : bash_completions_dir)
+# install_data('bash-completion/completions/gst-inspect-1.0', install_dir : bash_completions_dir)
endif

[xinyu_chen]

Please use the right version by this commit:

Build Gstreamer 1.17.x (commit e4f7cdb0df7b)

[brady3]

@xinyu_chen 

Can you please help to check whether my gstreamer's version is not correct(1.16.3)?

Gstreamer source code is as below:

I used the below command to get current version:

git checkout -b dev e4f7cdb0df7b 

[brady3]

Another question @xinyu_chen 

There is no 'lib' path under my install folder(xxx/packages):

Does it mean "my install folder" ? 

[xinyu_chen]

correct, the /lib path is in your device.

[brady3]

Thanks for your reply @xinyu_chen 

But I still didn't fully get what you said. Actually,  my problem is that Gstreamer 1.17 can not be installed successfully on my machine. And there is error occured when I run below command:

if my folder where Gstreamer was installed into is "~/work/avb/packages". Can you please share me the correct command to fix the "prefix" path in the .pc files? Thanks a lot!

[xinyu_chen]

Oh, this command means, change the "~/work/avb/packages" to "/usr" in all of the .pc file inside ./lib/pkgconfig

[brady3]

@xinyu_chen Where is the path " ./lib/pkgconfig" ?  It doesn't exist under my installer folder. 

"./lib/pkgconfig" should mean "lib/pkgconfig" under my installer folder. right?

but there is no "lib/pkgconfig" under my installer folder

[brady3]

@xinyu_chen   I used another command to perform fixing prefix from <my installer flder> to '/usr':

grep -lR '/home/yzg/work/avb/packages' --include='*.pc' .| xargs sed -i 'sI/home/yzg/work/avb/packages|/usr|g'

It worked fine but I am not sure whether this command is correct.

Now, I met another problem about installing gst-plugins-base

It seemed that Gstreamer was not installed successfully. Can you please help to clarify this issue? Thanks a lot!

[brady3]

@xinyu_chen Can you please give comments on my previous question? Thanks very much!

[EvanWu123]

Hi @xinyu_chen ,

I build all the library step by step. But after I put "<install folder>/usr" into "root/usr" and input command "gst-inspect-1.0", I got some errors.

The following is my log:

root@imx8mpevk:~# export GST_PLUGIN_PATH=/usr/lib/gstreamer-1.0/
root@imx8mpevk:~# gst-inspect-1.0

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:36.879: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:37.029: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstbeepdec.so': /usr/lib/libgstfsl-1.0.so.0: undefined symbol: gst_allocator_phymem_get_type

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:37.115: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstimxcompositor.so': /usr/lib/libgstfsl-1.0.so.0: undefined symbol: gst_allocator_phymem_get_type

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:37.122: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstx264.so': libx264.so.157: cannot open shared object file: No such file or directory

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:37.217: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstvideo4linux2.so': /usr/lib/gstreamer-1.0/libgstvideo4linux2.so: undefined symbol: gst_buffer_add_dmabuf_meta

(gst-inspect-1.0:795): GLib-GObject-WARNING **: 11:28:38.196: cannot register existing type 'GstDashDemux'

(gst-inspect-1.0:795): GLib-CRITICAL **: 11:28:38.196: g_once_init_leave: assertion 'result != 0' failed

(gst-inspect-1.0:795): GStreamer-CRITICAL **: 11:28:38.196: gst_element_register: assertion 'g_type_is_a (type, GST_TYPE_ELEMENT)' failed

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:38.405: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstimxvideoconvert.so': /usr/lib/libgstfsl-1.0.so.0: undefined symbol: gst_allocator_phymem_get_type

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:38.510: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstvpu.so': /usr/lib/libgstfsl-1.0.so.0: undefined symbol: gst_allocator_phymem_get_type

(gst-inspect-1.0:795): GStreamer-WARNING **: 11:28:38.795: Failed to load plugin '/usr/lib/gstreamer-1.0/libgstaiurdemux.so': /usr/lib/libgstfsl-1.0.so.0: undefined symbol: gst_allocator_phymem_get_type
mpegpsmux: mpegpsmux: MPEG Program Stream Muxer
spectrum: spectrum: Spectrum analyzer
shapewipe: shapewipe: Shape Wipe transition filter
videoparsersbad: h263parse: H.263 parser
videoparsersbad: h264parse: H.264 parser
videoparsersbad: diracparse: Dirac parser
videoparsersbad: mpegvideoparse: MPEG video elementary stream parser
videoparsersbad: mpeg4videoparse: MPEG 4 video elementary stream parser
videoparsersbad: pngparse: PNG parser
videoparsersbad: jpeg2000parse: JPEG 2000 parser
videoparsersbad: h265parse: H.265 parser
videoparsersbad: vc1parse: VC1 parser
dtmf: dtmfsrc: DTMF tone generator
dtmf: rtpdtmfsrc: RTP DTMF packet generator
dtmf: rtpdtmfdepay: RTP DTMF packet depayloader
dvdsub: dvdsubdec: DVD subtitle decoder
dvdsub: dvdsubparse: DVD subtitle parser
fieldanalysis: fieldanalysis: Video field analysis
jpeg: jpegenc: JPEG image encoder
jpeg: jpegdec: JPEG image decoder
videofilter: gamma: Video gamma correction
videofilter: videobalance: Video balance
videofilter: videoflip: Video flipper
videofilter: videomedian: Median effect
y4menc: y4menc: YUV4MPEG video encoder
dvbsubenc: dvbsubenc: DVB subtitle encoder
aiff: aiffparse: AIFF audio demuxer
aiff: aiffmux: AIFF audio muxer
subenc: srtenc: Srt encoder
subenc: webvttenc: WebVTT encoder
soup: souphttpsrc: HTTP client source
soup: souphttpclientsink: HTTP client sink
dvdlpcmdec: dvdlpcmdec: DVD LPCM Audio decoder
theora: theoradec: Theora video decoder
theora: theoraenc: Theora video encoder
theora: theoraparse: Theora video parser
compositor: compositor: Compositor
audiomixmatrix: audiomixmatrix: Matrix audio mix
sbc: sbcdec: Bluetooth SBC audio decoder
sbc: sbcenc: Bluetooth SBC audio encoder
ivfparse: ivfparse: IVF parser
legacyrawparse: videoparse: Video Parse
legacyrawparse: audioparse: Audio Parse
interleave: interleave: Audio interleaver
interleave: deinterleave: Audio deinterleaver
transcode: transcodebin: Transcode Bin
transcode: uritranscodebin: URITranscode Bin
videotestsrc: videotestsrc: Video test source
audiorate: audiorate: Audio rate adjuster
adpcmdec: adpcmdec: ADPCM decoder
ogg: oggdemux: Ogg demuxer
ogg: oggmux: Ogg muxer
ogg: ogmaudioparse: OGM audio stream parser
ogg: ogmvideoparse: OGM video stream parser
ogg: ogmtextparse: OGM text stream parser
ogg: oggparse: Ogg parser
ogg: oggaviparse: Ogg AVI parser
matroska: matroskademux: Matroska demuxer