The VPU hardware Encoder in i.MX 8M Plus doesn’t support 4K resolution. The objective of this article is to achieve 4K (3840X2160) Encode by performing two '1920X2160' encodes per 4K frame on i.MX 8M Plus SOC. On the decoding side, two '1920X2160' Decodes shall be performed simultaneously and combine them together to recreate the 4K Decoded frame.
For this POC, i.MX 8M Plus is used as the Encoder and i.MX 8QM/i.MX 95 is used as the Decoder. This POC uses H.264 codec, but this approach can be used for H.265 codec too.
This article as has various experiments performed on i.MX 8M Plus for power optimizations, for use-cases that require 4K Encoding at lower power consumption. The power numbers documented this page are the GROUP_SOC_FULL data captured by the BCU tool.
Note: Even though the i.MX 8M Plus Reference Manual claims support only for FHD H.264/H.265 encode, we were able to showcase 4K@30 H.264/H.265 encode equivalent on i.MX 8M Plus. This was done specific to a customer use case. Please discuss with marketing before sharing this POC with customers.
The i.MX 8M Plus captures 4K@30fps stream on MIPI CSI interface and splits frame into 2 halves (1920X2160) and encodes them separately and transmit them over the network (as RTP packets on two separate ports).
On the receiving side, i.MX 8QM/i.MX 95 SOC receives the frames over network (from two different ports), decode and combine them two form a 4K frame.
Note: This POC doesn't document the mechanism to synchronize the two encoded/decoded frames over network. It is possible that one of the two halves gets dropped in the network transmission or in the pipeline and may result in incorrect 4K frame.
A conceptual block diagram is shown below.
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=2160,width=3840,framerate=15/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1920 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5004 \
t. ! videocrop top=0 left=1920 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5005
gst-launch-1.0 \
imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=2160 \
sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=2160 ! waylandsink sync=false \
udpsrc port=5004 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_0 \
udpsrc port=5005 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_1
With various power optimizations, below is the Power measurement of i.MX 8M Plus at Camera-side.
Power Rails Group |
3840x2160@30fps Average Power (mWatt) |
2560x1440@30fps Average Power (mWatt) |
1920x1080@30fps Average Power (mWatt) |
Notes |
|||
H.264 Encode |
H.265 Encode |
H.264 Encode |
H.265 Encode |
H.264 Encode |
H.265 Encode |
||
GROUP_SOC_FULL |
1336 |
1342 |
1037 |
1039 |
857 |
856 |
SOC power consumption |
GROUP_DRAM |
347 |
319 |
199 |
185 |
110 |
100 |
DRAM device power supply. Excluded from i.MX 8M Plus internal power |
GROUP_PLATFORM |
2458 |
2430 |
1910 |
1893 |
1573 |
1561 |
Complete EVK power consumption |
For details on power rails, refer to AN13054
The software used during the experiments is part of Linux BSP available at Embedded Linux for i.MX Applications Processors, the test was evaluated on Linux 6.6.3_1.0.0 release.
Hardware components required are i.MX 8M Plus or i.MX 8QM/i.MX 95 EVKs.
Component | i.MX 8M Plus | i.MX 8QM/i.MX 95 |
---|---|---|
Power Supply | Yes |
Yes |
HDMI 4K Display | Yes | |
USB micro-B cable (Type-A male to Micro-B male) | Yes |
Yes |
HDMI cable | Yes | |
IMX-MIPI-HDMI (MIPI-DSI to HDMI adapter) | Yes | |
MIPI-CSI camera module | Yes |
Yes |
The picture depicts board setup demonstrated for the POC. i.MX 8M Plus EVK is used for camera input, a 4K MIPI CSI camera. i.MX 8Quad Max is used on the receiving side to reconstruct 4K frame and display on 4K monitor.
This section has details on various experiments performed to optimize power on i.MX 8M Plus SOC.
Default DDR data rate on i.MX 8M Plus is 4000MTS. We reduced it to 2400MTS. Change following lines in u-boot Makefile (..../board/freescale/imx8mp_evk/Makefile). Generate a new flash.bin file using appropriate toolchain and flash the image.
obj-$(CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS) += lpddr4_timing.o
Add following lines in uboot defconfig (configs/imx8mp_evk_defconfig)
CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS=y
This reduces the power measurement to ~1.64W. See the snapshot below.
In overdrive mode, VDD_SOC is 0.95V. We switch to nominal drive mode where in VDD_SOC is 0.85V. Change following lines in uboot Makefile (.../board/freescale/imx8mp_evk/Makefile)
obj-$(CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS) += lpddr4_timing.o
Add following lines in uboot defconfig (configs/imx8mp_evk_defconfig)
CONFIG_IMX8M_LPDDR4_FREQ0_2400MTS=y
CONFIG_IMX8M_VDD_SOC_850MV=y
Generate a new flash.bin file using appropriate toolchain and flash the image.
With new flash.bin, it reduced the power measurement is ~1.33W. See snapshot below.
While keeping the above optimizations (from Part 1 and Part 2), disable the ethernet in dts and use below GST pipeline.
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=2160,width=3840,framerate=15/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1920 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink \
t. ! videocrop top=0 left=1920 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink
This reduces the power measurement to ~1.28W. See snapshot below.
For use-cases that required QHD resolution (2560X1440), this experiment shows the GST pipeline and power numbers for 2560X1440 resolution.
Also, include all the changes from Part 2.
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=1440,width=2560,framerate=30/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1280 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5004 \
t. ! videocrop top=0 left=1280 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! udpsink host=<RECEIVER_IP> port=5005
gst-launch-1.0 \
imxcompositor_g2d name=c \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=1440 \
sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=1440 ! waylandsink sync=false \
udpsrc port=5004 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_0 \
udpsrc port=5005 ! "application/x-rtp, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! h264parse ! v4l2h264dec ! c.sink_1
This reduces the power measurement to ~1.033W. See below snapshot.
Reduced the resolution to QHD and disable the ethernet interface in dts. Also, include all the changes from Part 2.
gst-launch-1.0 v4l2src device=<CAMERA_PATH> ! "video/x-raw,height=1440,width=2560,framerate=30/1" ! tee name=t \
t. ! videocrop top=0 left=0 right=1280 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! fakesink \
t. ! videocrop top=0 left=1280 right=0 bottom=0 ! imxvideoconvert_g2d videocrop-meta-enable=true ! v4l2h264enc ! rtph264pay ! faeksink
This reduces the power measurement to ~1.005W.
Note: no need to split frames. Directly encode the entire frame using VPU.
Power measurements for this use-case is 856mW.
The patch file with the changes made in the dts is attached, for reference.
Please note, the attached patch file has ethernet enabled.
Version | Description | Date |
---|---|---|
1.0.0 | Initial release | April 10th , 2024 |
1.0.1 | Added additional power measurements and clean-up | Jun 4th, 2024 |
只有注册用户才能在此添加评论。 如果您已经注册,请登录。 如果您还没有注册,请注册并登录。