The VPU hardware Encoder in i.MX 8M Plus doesn’t support 4K resolution. The objective of this article is to achieve 4K (3840X2160) Encode by performing 2 '1920X2160' Encodes per 4K frame on i.MX 8M Plus SOC. On the decoding side, 2 '1920X2160' Decodes shall be performed simulatenuously and combine them together to recreate the 4K Decoded frame.
For this POC, i.MX 8M Plus is used as the Encoder and i.MX 8QM/i.MX 95 is used as the Decoder. This POC uses H.264 codec, but this approach can be used for H.265 codec too.
This article as has various experiments performed on i.MX 8M Plus for powe optimizations, for use-cases that require 4K Encoding at lower power consumption. The power numbers documented this page are the GROUP_SOC_FULL data captured by the BCU tool.
Implementation using GStreamer
The i.MX 8M Plus captures 4K@30fps stream on MIPI CSI interface and splits frame into 2 halves (1920X2160) and encodes them separately and transmit them over the network (as RTP packets on two separate ports).
On the receiving side, i.MX 8QM/i.MX 95 SOC receives the frames over network (from two different ports), decode and combine them two form a 4K frame.
Note: This POC doesn't document the mechanism to synchronize the two encoded/decoded frames over network. It is possible that one of the two halves gets dropped in the network transmission or in the pipeline and may result in incorrect 4K frame.
A conceptual block diagram is shown below.
Sender's pipeline
- We use GStreamer plug-ins to videocrop each 4K frame into two 1920X2160 frames (two halves).
- Each half is encoded using VPU encoder in H.264 format and transmitted over network.
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=2160,width=3840,framerate=15/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1920 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5004 \
t. ! videocrop top=0 left=1920 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5005
Receiver's pipeline
- Both halves are decoded separately using the VPU decoder.
- Decoded halves are then stitched together to reconstruct 4KP30 frame of 3840X2160 resolution.
- The resulted video feed is then presented on 4K display.
- The two halves must be synchronized both at sender side and receiver side.
gst-launch-1.0 \
imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=2160 \
sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=2160 ! waylandsink sync=false \
udpsrc port=5004 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_0 \
udpsrc port=5005 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_1
The power measurement obtained was ~1.9W. The following snapshot depicts the different power numbers for each rail, we focused on reducing GROUP_SOC_FULL using below experiments.
Table of Contents
- Software
- Hardware
- Setup
- Results
- Release notes
1 Software
The software used during the experiments is part of Linux BSP available at Embedded Linux for i.MX Applications Processors, the test was evaluated on Linux 6.6.3_1.0.0 release.
2 Hardware
Hardware components required are i.MX 8M Plus or i.MX 8QM/i.MX 95 EVKs.
| Component | i.MX 8M Plus | i.MX 8QM/i.MX 95 |
|---|---|---|
| Power Supply | Yes |
Yes |
| HDMI Display | Yes | |
| USB micro-B cable (Type-A male to Micro-B male) | Yes |
Yes |
| HDMI cable | Yes | |
| IMX-MIPI-HDMI (MIPI-DSI to HDMI adapter) | Yes | |
| MIPI-CSI camera module | Yes |
Yes |
3 Setup
The picture depicts board setup demonstrated for the POC. i.MX 8M Plus EVK is used for camera input, a 4K MIPI CSI camera. i.MX 8Quad Max is used on the receiving side to reconstruct 4K frame and display on 4K monitor.
4 Results
This section has details on various experiments performed to optimize power on i.MX 8M Plus SOC.
Part 1: DDR data rate at 2400MTS.
Default DDR data rate on i.MX 8M Plus is 4000MTS. We reduced it to 2400MTS. Change following lines in u-boot Makefile (..../board/freescale/imx8mp_evk/Makefile). Generate a new flash.bin file using appropriate toolchain and flash the image.
obj-$(CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS) += lpddr4_timing.o
Add following lines in uboot defconfig (configs/imx8mp_evk_defconfig)
CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS=y
This reduces the power measurement to ~1.64W. See the snapshot below.
Part 2: DDR data rate at 2400MTS with nominal drive mode (i.e. VDD_SOC at 0.85V).
In overdrive mode, VDD_SOC is 0.95V. We switch to nominal drive mode where in VDD_SOC is 0.85V. Change following lines in uboot Makefile (.../board/freescale/imx8mp_evk/Makefile)
obj-$(CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS) += lpddr4_timing.o
Add following lines in uboot defconfig (configs/imx8mp_evk_defconfig)
CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS=y
CONFIG_IMX8M_VDD_SOC_850MV=y
Generate a new flash.bin file using appropriate toolchain and flash the image.
With new flash.bin, it reduced the power measurement is ~1.35W. See snapshot below.
Part 3: Disable Ethernet
While keeping the above optimizations (from Part 1 and Part 2), disable the ethernet in dts and use below GST pipeline.
Sender's pipeline:
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=2160,width=3840,framerate=15/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1920 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink \
t. ! videocrop top=0 left=1920 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink
This reduces the power measurement to ~1.28W. See snapshot below.
Part 4: Reduce the input resolution from the camera.
For use-cases that required QHD resolution (2560X1440), this experiment shows the GST pipeline and power numbers for 2560X1440 resolution.
Also, include all the changes from Part 2.
Sender's pipeline:
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=1440,width=2560,framerate=30/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1280 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5004 \
t. ! videocrop top=0 left=1280 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5005
Receiver's pipeline:
gst-launch-1.0 \
imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=1440 \
sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=1440 ! waylandsink sync=false \
udpsrc port=5004 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_0 \
udpsrc port=5005 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_1
This reduces the power measurement to ~1.068W. See below snapshot.
Part 5: Reduce the input resolution with Ethernet disabled.
Reduced the resolution to QHD and disable the ethernet interface in dts. Also, include all the changes from Part 2.
Sender's pipeline:
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=1440,width=2560,framerate=30/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1280 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink \
t. ! videocrop top=0 left=1280 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! faeksink
This reduces the power measurement to ~1.005W.
The patch file with the changes made in the dts is attached, for reference.
Please note, the attached patch file has ethernet enabled.
5 Release notes
| Version | Description | Date |
|---|---|---|
| 1.0.0 | Initial release | April 10th 2024 |