8MMINILPD4-EVK rtsp stream slow processing

[ashish_m]

Development platform: 8MMINILPD4-EVK board

Application as follows:

• Capture IP camera video stream.
• Store video stream to Hard Disk (Time based / event driven storage)
• Perform Object detection logic.

Setup Information:

• 8MMINILPD4-EVK development board
• Linux kernal LF5.15.5_1.0.0
• Yocto Project Setup : build type :imx-image-full
• IP camera and development board connected to local LAN network.

IP camera protocol          : RTSP

Frame Setting                    : 1080P @ 30 pfs, Mainstream: video H264, bitrate 2048 kbps

During IP camera RTSP stream integration with development board we face following problem –

When ffmpeg library transcodes the input video from rtsp stream using default /specific options, cpu utilization is very high and resulting video stream has lag and missing frames.

The performance is optimum when no video transcoding happens and input stream is copied as is from the input.

The above conclusion can be inferred from following results -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[following commands were executed on the target board to record 15s of video from the IP camera using FFMPEG library (binaries)]

Case 1

Ip cam settings:

Res - 1080p

Compression - h.264

Command 1 - time ffmpeg -i rtsp://admin:123456@192.168.0.123:554/videoMain -t 15s out11.mp4

Execution time - 47.43s

CPU Utilisation - ~60-70%

Command 2 - time ffmpeg -c:v h264 -i rtsp://admin:123456@192.168.0.123/videoMain -c:v copy -t 15s out12.mp4

Execution time - 15.516s

CPU Utilisation - ~5-10%

Case 2

Ip cam settings:

Res - 1080p

Compression - h.265

Command 1 - time ffmpeg -i rtsp://admin:123456@192.168.0.123:554/videoMain -t 15s out21.mp4

Execution time - 46.58s

CPU Utilisation - ~60-70%

Command 2 - time ffmpeg -c:v hevc -i rtsp://admin:123456@192.168.0.123/videoMain -c:v copy -t 15s ~/sample_ipcam_vid/out22.mp4

Execution time - 15.483s

CPU Utilisation - ~5-10%

Case 3

Ip cam settings:

Res - 720p

Compression - h.264

Command 1 - time ffmpeg -i rtsp://admin:123456@192.168.0.123:554/videoMain -t 15s out31.mp4

Execution time - 24.516s

[Missed Frames]
CPU Utilisation - ~60-70%

Command 2 - time ffmpeg -c:v h264 -i rtsp://admin:123456@192.168.0.123/videoMain -c:v copy -t 15s ~/sample_ipcam_vid/out32.mp4

Execution time - 15.494s

CPU Utilisation - ~5-10%

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Similar performance is observed when using pre-saved local video file as input - thus ruling out bottlenecks caused due to network infrastructure

also, similar performance/lag is observed when reading video frames from opencv-python (both when input is ip camera url or saved local video file)

Python line -

video = cv2.VideoCapture(camUrl)

The performance doesn’t much vary when resolution/compression settings are changed in th eIP camera settings.

Desired performance – (to achieve live streaming, recording and ComputerVision applications)
CPU utilisation – as low as possible
Execution time –  around 1x (i.e. approx 15s for 15s Stream)

Kindly suggest how we can optimized same.