i.MX8MPlus crashes when using gstreamer vpudec plugin

[MicMoba]

Hi,

I'm using Phytec i.MX8MPlus LPDDR4 Pollux EVK board.

Kernel: 5.4.70 (aarch64)

GStreamer: 1.16.2

VPUDEC: 4.5.7

   warpper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 17 2020 07:03:22)

   vpulib: 1.1.1

   firmware: 1.1.1.0

autovideosink -> waylandsink

GStreamer pipeline: gst-launch1.0 -v udpsrc port=8884 ! application /x-rtp,payload=96 ! rtph264depay ! queue ! h264parse ! vpudec ! autovideosink sync=false

Streaming a video file with HW acceleration via gstreamer constantly causes a process stall. The stall can be recovered only by resetting the board.

First the stream will show correctly on the screen. When I cancel the task (STRG+C) the process stall. The only way to get the board running again is resetting the board.

When I exchange the vpudec with avdec_h264 everything works well.

I also tried to play a h264 video file. But when the video finished the process also stall.

Sometimes, but really sometimes, I can cancel the task by using STRG+C. Then the pipeline stops successfully.

How can I do more investigations?

[joanxie]

“I also tried to play a h264 video file. But when the video finished the process also stall.”
> what do you mean? how could we reproduce this? I playback 1080p@30 mp4 video successfully, could you share your log file about "stall", I playback the mp4 file which has 10 mins, I didn't see any abnormal finish the process, I use the command like "gplay-1.0 xxx.mp4"

[MicMoba]

My first thoughts were that the stall has something to do with STRG+C. So I played a short video because I want to see if the process also stall when the video finished normaly. 

I used this gstreamer-pipeline:

gst-launch-1.0 filesrc location=test.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false

But have you seen my post from friday? The problem seems to be the power management.

[joanxie]

I couldn't reproduce this on my i.mx8mp evk board

root@8mpevk:~# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4

Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:05:27.685532375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (19035), playing for (0:05:27.686027375), fps (58.089).
Freeing pipeline ...

what imx8mp board do you use?A0 or A1? to check the schematic of board, mine is SCH-46370 REV A2

[MicMoba]

I'm using Phytec i.MX8MPlus LPDDR4 Pollux EVK board.

Kernel: 5.4.70 (aarch64)

GStreamer: 1.16.2

VPUDEC: 4.5.7

   warpper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 17 2020 07:03:22)

   vpulib: 1.1.1

   firmware: 1.1.1.0

autovideosink -> waylandsink

RTP-Stream:

GStreamer pipeline: gst-launch1.0 -v udpsrc port=8884 ! application /x-rtp,payload=96 ! rtph264depay ! queue ! h264parse ! vpudec ! autovideosink sync=false

MP4-File:

gst-launch-1.0 filesrc location=/home/user0/recording_10-08-2020_023237.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false

Did you activate PowerManagement in your kernel config? 

First the stream will show correctly on the screen. When I cancel the task (STRG+C) the process stall. The only way to get the board running again is resetting the board.

[joanxie]

Did you activate PowerManagement in your kernel config? 

> I don't change any source code, use the pre-built images

I tested again, use ctrl+C to stop the video and replay again, I didn't find any issue on my board

root@imx8mpevk:/mnt# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false
Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.961084125
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (419), playing for (0:00:06.961601125), fps (60.187).
Freeing pipeline ...
root@imx8mpevk:/mnt# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false
Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.209655000
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (255), playing for (0:00:04.210131625), fps (60.568).
Freeing pipeline ...

"^Chandling interrupt." is for my ctrl+c command

[MicMoba]

Yes PM is activated in my kernel config. Currently I do not find a way to disable it. It seems that it is firmly rooted with the architecture.

I attached my current .config.

[joanxie]

the imx8mp is still under preproduction, I just got the imx8mp evk board, I need spend time on reproducing this, any update, I will let you know it

[MicMoba]

Hi joanxie,

it has something to do with the powermanagement. I comment all lines that started with "pm_runtime" in the file /drivers/mxc/hantro_845/hantrodec_845s.c and after compiling everything works fine.

To comment these lines is not the solution but the proof that the power management is involved.

[joanxie]

I didn't find "pm_runtime" in my source code:

"https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=..."

[xinhkang]

there are tons of function calls starts with pm_runtime_, for example:
https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=...

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=...

...

we are hitting similar issue (NXP beta2 FW):

Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881498] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881518] rcu: 3-...0: (0 ticks this GP) idle=0f2/1/0x4000000000000000 softirq=10466/10466 fqs=25097
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881523] (detected by 2, t=52517 jiffies, g=27385, q=6357)
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881526] Task dump for CPU 3:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881533] rsp_srv_1 R running task 0 2667 2535 0x00000002
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881540] Call trace:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881552] __switch_to+0x104/0x170
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881560] genpd_power_off+0x114/0x278
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881564] genpd_power_off+0x198/0x278
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881570] genpd_runtime_suspend+0x188/0x258
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881576] __rpm_callback+0x88/0x140
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881582] rpm_callback+0x20/0x80
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881586] rpm_suspend+0xf0/0x410
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881592] rpm_idle+0xac/0x150
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881597] __pm_runtime_idle+0x58/0xa0
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881607] hantrodec_release+0xdc/0x1c0
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881613] __fput+0x88/0x228
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881616] ____fput+0xc/0x18
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881626] task_work_run+0x80/0x130
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881633] do_notify_resume+0x1f0/0x450

[MicMoba]

Meanwhile I think it has something to do with the kernels power management. Look at this trace:

Interrupt: Stopping pipeline ...
Execution ended after 0:00:21.204112375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
[  118.273197] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  118.279305] rcu:     1-...0: (1 GPs behind) idle=6fa/1/0x4000000000000000 softirq=7550/7551 fqs=2563  
[  118.288261]  (detected by 0, t=5255 jiffies, g=11145, q=80)
[  118.293831] Task dump for CPU 1:
[  118.297058] gst-launch-1.0  R  running task        0   671    624 0x00000202
[  118.304107] Call trace:
[  118.306561]  __switch_to+0x108/0x1c0
[  118.310138]  genpd_power_off+0x114/0x278
[  118.314059]  genpd_power_off+0x198/0x278
[  118.317980]  genpd_runtime_suspend+0x188/0x258
[  118.322425]  __rpm_callback+0x88/0x140
[  118.326172]  rpm_callback+0x20/0x80
[  118.329659]  rpm_suspend+0xf0/0x410
[  118.333146]  rpm_idle+0xac/0x150
[  118.336372]  __pm_runtime_idle+0x58/0xa0
[  118.340297]  hantrodec_release+0xdc/0x1c0
[  118.344307]  __fput+0x88/0x228
[  118.347360]  ____fput+0xc/0x18
[  118.350414]  task_work_run+0x80/0x130
[  118.354076]  do_notify_resume+0x1f0/0x688
[  118.358084]  work_pending+0x8/0x10

[joanxie]

when did you get these log file? after delete the "pm_runtime" ?

[MicMoba]

Hi joanxie,

no this was the log I got when I use the not modified hantrodec_845s.c file and I interrupt (CTRL+C) a current gstreamer stream.

After deleting all lines that started with "pm_" in the hantrodec_845s.c file everything works fine. 

[joanxie]

obviously you use the third party bsp, I think you should contact their engineer and confirm why they they add the PM there