Wikipedia: Audio Video Bridging (AVB) is a common name for the set of technical standards which provide improved synchronization, low-latency, and reliability for switched Ethernet networks. AVB was initially developed by the Institute of Electrical and Electronics Engineers (IEEE) Audio Video Bridging task group of the IEEE 802.1 standards committee. In November 2012, Audio Video Bridging task group was renamed to Time-Sensitive Networking (TSN) task group to reflect the expanded scope of its work, which is to "provide the specifications that will allow time-synchronized low latency streaming services through IEEE 802 networks". Further standardization efforts are ongoing in IEEE 802.1 TSN task group.
AVB and TSN technologies and standards:
Standard |
Area of Definition |
Title of Standard |
AVB/TSN |
IEEE 802.1AS |
Timing and synchronization |
Timing and Synchronization for Time-Sensitive Applications (gPTP) |
AVB TSN |
IEEE 802.1Qav |
Forwarding and queuing |
Forwarding and Queuing for Time-Sensitive Streams (FQTSS) |
AVB |
IEEE 802.1Qat |
Path control and reservation |
Stream Reservation Protocol (SRP) |
AVB |
IEEE 802.1BA |
Bridging |
Audio Video Bridging (AVB) Systems |
AVB |
IEEE 1722 AVTP |
AV transport |
Layer 2 Transport Protocol for Time Sensitive Applications |
AVB |
IEEE 1722 AVDECC |
Device manage and control |
Device Discovery, Enumeration, Connection Management and Control Protocol |
AVB |
IEEE 802.1Qbu and IEEE 802.3br |
Forwarding and queuing |
Frame preemption |
TSN |
IEEE 802.1Qbv |
Forwarding and queuing |
Enhancements for scheduled traffic |
TSN |
IEEE 802.1Qca |
Path control and reservation |
Path control and reservation |
TSN |
IEEE 802.1Qcc |
Central configuration method |
Enhancements and performance improvements |
TSN |
IEEE 802.1Qci |
Time-based ingress policing |
Per-stream filtering and policing |
TSN |
IEEE 802.1CB |
Seamless redundancy |
Frame replication and elimination for reliability |
TSN |
Since many of the standards are only for TSN switch/bridges and i.MX8MP is design to be a TSN/AVB endpoint, this demo does not implement a full stack or full standards. It only demonstrates the basic end-to-end point (talker to listener) A/V streaming without bridge or switch.
Below table shows what this demo supports:
Standard |
Software |
Hardware |
IEEE 802.1AS |
Linuxptp |
ENET_QoS IP |
IEEE 802.1Qav/Qbu/Qbv |
TC qdisc (taprio, mqprio, etf, cbs) |
ENET_QoS IP (multi queue + EST, FPE, CBS) |
IEEE 1722 AVTP |
Libavtp |
N/A |
Two i.MX8MP EVK boards are used for this demo, one act as a AVB talker to send A/V streams, the other one act as a AVB listener to receive A/V streams who can be playback to audio codec and sink video to screen. The two boards are connected by a RJ45 ethernet cable on each ENET2 port (only ENET2 port has TSN features).
Three streams’ type and SR (Stream Reservation) class are defined as below to grantee time sensitive (sub-microsecond synchronization), low latency and bandwidth on the ethernet:
These three TSN streams would be allocated into different traffic control (TC) class for egress in Linux. Different TC class would be mapped to different hardware queues with dedicated DMA channel, thanks to the multi-queue support by ENET_QoS IP. Then the traffic control apply different scheduling policy on each queues for different types of streams egress, to make sure highest priority or critical packet can be sent in the correct time slot.
The TSN streams are transmitted over Virtual LANs (VLANs). Bridges use the VLAN priority information (PCP) to identify Stream Reservation (SR) traffic classes which are handled according to the Forward and Queuing Enhancements for Time-Sensitive Streams (FQTSS) mechanisms described in Chapter 34 of the IEEE 802.1Q standard.
The demo is built up by following blocks:
This demo can use two different traffic control qdisc settings:
The pfifo is the default traffic control class, which use FIFO schedule policy for egress packets. The CBS class is actually handled by hardware IP to select which queue for transmit in a certain time slot.
CBS parameters come straight from the IEEE 802.1Q-2018 specification. They are the following:
The IEEE 802.1Qbv defines the schedule for each of the queues on every egress port which makes the implementation aware of traffic arrival schedule. This information can be used to block the lower priority traffic from transmission in this time window/slot. This ensures that scheduled traffic is forwarded from sender to receiver through all the network nodes with a deterministic delay. The i.MX8MP uses the gate control list with configurable time slot and frame preempt (IEEE 802.1Qbu) features to support EST.
Other than the CBS, the gate control list control the egress transmission by a fixed open interval for that queue.
$MACHINE=imx8mpevk DISTRO=fsl-imx-xwayland source imx-setup-release.sh -b build-8mp
$bitbake fsl-image-validation-imx
Prepare a SD card and burn it with the built out images.
To add tc/tcpdump command in iproute2 package, please add package into IMAGE_INSTALL_append variable into conf/local.conf:
IMAGE_INSTALL_append = " iproute2 iproute2-tc tcpdump
Rebuild the kernel after applying the 0001-qenet-add-queue-avoid-panic.patch (attached), and overwrite the Image and imx8mp-evk.dtb on the boot partition of the SD card.
$bitbake -f fsl-image-validation-imx -c populate_sdk
$sh tmp/deploy/sdk/fsl-imx-xwayland-glibc-x86_64-fsl-image-validation-imx-aarch64-imx8mpevk-toolchain-5.4-zeus.sh
The toolchain would be installed into /opt/fsl-imx-xwayland/5.4-zeus/
$mkdir <your install folder>
Create a folder to install all of the shared libraries, binaries and configure files which built out manually in this doc. After built done, you should copy all of the contents in this folder to target board root.
$source /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux
$git clone https://github.com/Avnu/libavtp.git
$cd libavtp
$meson build
$ninja -C build
Copy the built out .so and .pc into the toolchain rootfs:
$sudo cp build/libavtp.so* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/lib/
$sudo cp build/meson-private/*.pc /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/lib/pkgconfig/
To make sure you have avtp package installed correctly:
$pkg-config --list-all | grep avtp
Copy the .so into the install folder:
$cp build/libavtp.so* <install folder>/usr/lib/
Copy header files:
$cp libavtp/include/* files <toolchain_path>/sysroots/aarch64-poky-linux/usr/lib
The AAF (ALSA AVTP Audio Format) plugin is a PCM plugin that uses Audio Video Transport Protocol (AVTP) to transmit/receive audio samples through a Time-Sensitive Network (TSN) capable network. The plugin enables media applications to easily implement AVTP Talker and Listener functionalities.
$cd <yocto build>/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/alsa-plugins-1.1.9
$./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=<your install folder>/usr --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot --disable-static --enable-aaf --disable-jack --disable-libav --disable-maemo-plugin --disable-maemo-resource-manager --enable-pulseaudio --enable-samplerate --with-speex=lib
$make
$make install
$git clone https://gitlab.freedesktop.org/gstreamer/gstreamer.git
$cd gstreamer
$patch -p1 < gstreamer-1.0-pass-build.patch
$meson build --prefix=<your install folder>/usr
$ninja -C build
$ninja -C build install
After Gstreamer is installed into <your install folder>, please fix the “prefix” path in the .pc files by, and copy to the toolchain folders:
$cd <your install folder>
$grep -lR <your install folder> ./lib/pkgconfig/ | xargs sed -i 's/<your install folder>/\/usr/g'
NOTE: Make sure you use the ‘\’ for ‘/’ convert in your path.
$cp -rf ./usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/
$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-base.git
$cd gst-plugins-base
$patch -p1 < gst-plugins-base-pass-build.patch
$meson build --prefix=<your install folder>/usr
$ninja -C build
$ninja -C build install
$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad.git
$cd gst-plugins-bad
$meson build --prefix=<your install folder>/usr
$ninja -C build
$ninja -C build install
After gst-plugins-base and gst-plugins-bad installed into <your install folder>, please fix the “prefix” path in the .pc files and copy them into the toolchain folders:
$cd <your install folder>
$grep -lR <your install folder> ./lib/pkgconfig/ | xargs sed -i 's/<your install folder>/\/usr/g'
NOTE: Make sure you use the ‘\’ for ‘/’ convert in your path.
$cp -rf ./usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/
AVTPDU uses H.264 as payload for streaming, this requires H.264 encoder/decoder plugins, either software or hardware accelerations. Since we upgrade the Gstreamer and its plugins to a new version, the VPU plugins cannot be used anymore. So software H.264 plugins are required: x264 for encoder, libav for decoder.
As the yocto actually has the x264 recipes, but not included in our bblayers, we need to copy the x264 source into our bblayers path under <yocto>/source to build:
$cp -rf ./poky/meta/recipes-multimedia/x264 ./meta-openembedded/meta-multimedia/recipes-multimedia/
$vi ./meta-openembedded/meta-multimedia/recipes-multimedia/x264_git.bb
Remove the LICENSE_FLAGS line
$bitbake -f x264 -c do_install
$sudo cp -rf tmp/work/aarch64-poky-linux/x264/r2917+gitAUTOINC+72db437770-r0/image/usr/* /opt/fsl-imx-xwayland/5.4-zeus/sysroots/aarch64-poky-linux/usr/
$git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-ugly.git
$cd gst-plugins-ugly$meson build --prefix=<your install folder>/usr
$ninja -C build
$ninja -C build install
If you met build issue, please remove "if have_cxx" from meson.build
As the yocto actually has the libav recipes, but not included in our bblayers, we need to copy the its source into our bblayers path under <yocto>/source:
$cp -rf ./poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-libav ./meta-openembedded/meta-multimedia/recipes-multimedia/gstreamer
$vi ./meta-openembedded/meta-multimedia/recipes-multimedia/gstreamer/gstreamer1.0-libav_1.16.1.bb
Remove the LICENSE_FLAGS line
$bitbake -f gstreamer1.0-libav -c do_install
$cp tmp/work/aarch64-mx8mp-poky-linux/gstreamer1.0-libav/1.14.0-r0/image/usr/lib/gstreamer-1.0/libgstlibav.so <your install folder>/usr/lib/gstreamer-1.0
Now you have all of the Gstreamer plugins which required for AVB/TSN audio/video demo:
Final step is to copy all of your built out files from <your install folder> into your board / root, and boot up the board.
Verify the Gstreamer plugins install correctly or not:
$export GST_PLUGIN_PATH= /usr/lib/gstreamer-1.0/
$gst-inspect-1.0
Check if the above Gstreamer plugins we built out can be found by gst-instpect.
The ENTE_QoS is assigned to eth1 instance. So create eth1.5 for vlan id 5:
$ip link add link eth1 name eth1.5 type vlan id 5 \
egress-qos-map 2:2 3:3
$ip link set eth1.5 up
The TSN control plane is implemented through the Linux Traffic Control (TC) System. The transmission algorithms specified in the Forwarding and Queuing for Time-Sensitive Streams (FQTSS) chapter of IEEE 802.1Q-2018 are supported via TC Queuing Disciplines (qdiscs). Linux currently provides the following qdiscs relating to TSN:
$tc qdisc add dev eth1 parent root handle 100 mqprio \
num_tc 3 \
map 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 \
hw 1
NOTE, since ENET_QoS Q0 does not support hardware CBS, we have to avoid using Q0 for AVB streaming. Here’s the mapping:
Create a root qdisc handle to map the different CLASS streams to hardware queues w/ GCL (gate control list). This root handle maps the CLASS A stream to queue Q1, CLASS B stream to Q2, and others to Q0 by the “map” and “queues” parameters, same as mqprio above. The Q1/Q2(CLASS1/2) gates are open in the first 300us interval, then only Q1(CLASS1) gate is open in the next 300us with all other CLASS gated. The last sched-entry defines in after Q1 is open after 300us, all of the CLASS gates are open for next 200us. Then loopback to the first sched-entry for gate control.
Please note in this demo, we only enable features to support the 802.1Qbv, the Qbu (Frame preempt) requires patches for iproute2.
$tc qdisc add dev eth1 parent root handle 100 taprio \
num_tc 3 \
map 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 \
base-time 1000000000 \
sched-entry S 0x6 300000 \
sched-entry S 0x2 300000 \
sched-entry S 0x7 200000 \
flags 0x2
Q1 CBS for video, Q2 CBS for audio:
$tc qdisc replace dev eth1 parent 100:2 handle 777 cbs \
idleslope 98688 sendslope -901312 hicredit 153 locredit -1389 \
offload 1
$tc qdisc replace dev eth1 parent 100:3 handle 888 cbs \
idleslope 3648 sendslope -996352 hicredit 12 locredit -113 \
offload 1
NOTE: By default, the Q0 will be allocated for pfifo qdisc class if we do not define them.
As ENET_QOS does not support hardware launch time in IP, the ETF qdisc would not be used here.
Run the ptp4l and phc2sys in background, and use check_clocks to check the ptp sync works.
$ptp4l -i eth1 -f ./gPTP.cfg --step_threshold=1 &
$pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 \
clockAccuracy 0xfe offsetScaledLogVariance 0xffff \
currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 \
ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 \
timeSource 0xa0"
$phc2sys -s eth1 -c CLOCK_REALTIME --step_threshold=1 \
--transportSpecific=1 -w &
$./check_clocks -d eth1
The PTP Hardware Clock (PHC) synced between PTP master/slave means the RMS offset between PHC and GM (master clock) is < 100ns. PHC and system clock (CLOCK_REALTIME) synced means the clock offset < 100ns
NOTE:
#
# 802.1AS example configuration containing those attributes which
# differ from the defaults. See the file, default.cfg, for the
# complete list of available options.
#
[global]
gmCapable 1
priority1 248
priority2 248
logAnnounceInterval 0
logSyncInterval -3
syncReceiptTimeout 3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step 1
path_trace_enabled 1
follow_up_info 1
transportSpecific 0x1
ptp_dst_mac 01:80:C2:00:00:0E
network_transport L2
delay_mechanism P2P
To run the alsa AAF demo, please add aaf0 and converter0 plugin device into /etc/asound.conf:
pcm.aaf0 {
type aaf
ifname eth1.5
addr 01:AA:AA:AA:AA:AA
prio 2
streamid AA:BB:CC:DD:EE:FF:000B
mtt 50000
time_uncertainty 1000
frames_per_pdu 12
ptime_tolerance 100
}
pcm.converter0 {
type linear
slave {
pcm "hw:2,0"
format S16_LE
}
}
The “aaf0” plugin device defines the ethernet interface which AAF runs on, the socket priority which mapping to Traffic Class in kernel TC, the stream-id for the aaf streaming.
The “converter0” plugin device is used for convert the S16_BE format to S16_LE for the wm8960 PCM audio.
Select one device as AVB talker, and run:
$speaker-test -p 25000 -F S16_BE -c 2 -r 48000 -D aaf0
Select one device as AVB talker, and run:
$arecord -F 25000 -t raw -f S16_BE -c 2 -r 48000 -D aaf0 | aplay -F 25000 -t raw -f S16_BE -c 2 -r 48000 -D converter0
You can hear the sound on the listener device. You can also check which qdisc queue is used for AAF by:
$tc -s qdisc
Select one device as AVB talker, and run:
$gst-launch-1.0 clockselect. \( clock-id=realtime \
audiotestsrc samplesperbuffer=12 is-live=true ! \
audio/x-raw,format=S16BE,channels=2,rate=48000 ! \
avtpaafpay mtt=50000000 tu=1000000 streamid=0xAABBCCDDEEFF000B processing-deadline=0 ! \
avtpsink ifname=eth1.5 address=01:AA:AA:AA:AA:AA priority=2 processing-deadline=0 \)
Select one device as AVB listener, and run:
$gst-launch-1.0 clockselect. \( clock-id=realtime \
avtpsrc ifname=eth1.5 ! avtpaafdepay streamid=0xAABBCCDDEEFF000B ! \
queue max-size-bytes=0 max-size-buffers=0 max-size-time=0 ! \
audioconvert ! audioresample ! alsasink device="hw:2,0" \)
Select one device as AVB talker, and run:
$gst-launch-1.0 clockselect. \( clock-id=realtime \
videotestsrc is-live=true ! video/x-raw,width=480,height=320,framerate=15/1 ! \
clockoverlay ! x264enc bframes=0 key-int-max=1 speed-preset=1 tune=4 ! h264parse config-interval=-1 ! \
avtpcvfpay processing-deadline=20000000 mtu=1400 mtt=2000000 tu=125000 streamid=0xAABBCCDDEEFF000A ! \
avtpsink ifname=eth1.5 priority=3 processing-deadline=20000000 \)
NOTE:
Select one device as AVB listener, and run:
$gst-launch-1.0 clockselect. \( clock-id=realtime \
avtpsrc ifname=eth1.5 ! avtpcvfdepay streamid=0xAABBCCDDEEFF000A ! \
queue max-size-bytes=0 max-size-buffers=0 max-size-time=0 ! \
avdec_h264 ! videoconvert ! clockoverlay halignment=right ! waylandsink \)
The demo screenshot below: there are two clocks show on the videotestsrc stream: left one is the timestamp recorded before x264enc encoding on the AVB talker side, right one is the timestamp recorded after avdec_h265 decoding and do video convert to YUV frames on AVB listener side. You can see the timestamp is sync in seconds.
Use tcpdump on board to dump the L2 ethernet packet:
./tcpdump -i eth1 ether proto 0x22f0 -w dump.pcap
The AVTP ether protocol code is 0x22f0 embedded inside the ether frame, or you can use "vlan 5" VLAN id for tcpdump parameters to dump.
Then open this dump.pcap in the windows/Linux PC by the wireshark tool, it will automatically show the protocol inside the package, it can also parser the IEEE1722 (AVTP) CVF/AFF package header as below:
When build ALSA aaf plugin.
"$./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=<your install folder>/usr --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot --disable-static --enable-aaf --disable-jack --disable-libav --disable-maemo-plugin --disable-maemo-resource-manager --enable-pulseaudio --enable-samplerate --with-speex=lib
"
When define --with-libtool-sysroot,
--with-libtool-sysroot=/opt/samba/nxa23059/jailhouse/yocto-5.x/build-8mp/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot
Can I interpret it as "<yocto build>/tmp/work/aarch64-mx8mp-poky-linux/alsa-plugins/1.1.9-r0/recipe-sysroot "?
How can i test with local mp4 video file?
I ran below cmd:
gst-launch-1.0 clockselect. \( clock-id=realtime \ filesrc location=sintel_trailer-480p.mp4 ! qtdemux ! h264parse config-interval=-1 ! \ avtpcvfpay processing-deadline=20000000 mtu=1400 mtt=2000000 tu=125000 streamid=0xAABBCCDDEEFF000A ! \
avtpsink ifname=eth1.5 priority=3 processing-deadline=20000000 \)
But, i got error:
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: DebugGstSystemClock
Got EOS from element "clockselect0".
Execution ended after 0:00:00.544463000
Setting pipeline to NULL ...
Freeing pipeline ...
with
basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc0> pad not activated yet