topic Re: i.MX8MPlus crashes when using gstreamer vpudec plugin in i.MX Processors

i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Wed, 24 Feb 2021 06:12:28 GMT

Hi,

I'm using Phytec i.MX8MPlus LPDDR4 Pollux EVK board.

Kernel: 5.4.70 (aarch64)

GStreamer: 1.16.2

VPUDEC: 4.5.7

warpper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 17 2020 07:03:22)

vpulib: 1.1.1

firmware: 1.1.1.0

autovideosink -> waylandsink

GStreamer pipeline: gst-launch1.0 -v udpsrc port=8884 ! application /x-rtp,payload=96 ! rtph264depay ! queue ! h264parse ! vpudec ! autovideosink sync=false

Streaming a video file with HW acceleration via gstreamer constantly causes a process stall. The stall can be recovered only by resetting the board.

First the stream will show correctly on the screen. When I cancel the task (STRG+C) the process stall. The only way to get the board running again is resetting the board.

When I exchange the vpudec with avdec_h264 everything works well.

I also tried to play a h264 video file. But when the video finished the process also stall.

Sometimes, but really sometimes, I can cancel the task by using STRG+C. Then the pipeline stops successfully.

How can I do more investigations?

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Fri, 26 Feb 2021 06:58:17 GMT

the imx8mp is still under preproduction, I just got the imx8mp evk board, I need spend time on reproducing this, any update, I will let you know it

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Fri, 26 Feb 2021 08:45:51 GMT

Meanwhile I think it has something to do with the kernels power management. Look at this trace:

Interrupt: Stopping pipeline ...
Execution ended after 0:00:21.204112375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
[ 118.273197] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 118.279305] rcu: 1-...0: (1 GPs behind) idle=6fa/1/0x4000000000000000 softirq=7550/7551 fqs=2563
[ 118.288261] (detected by 0, t=5255 jiffies, g=11145, q=80)
[ 118.293831] Task dump for CPU 1:
[ 118.297058] gst-launch-1.0 R running task 0 671 624 0x00000202
[ 118.304107] Call trace:
[ 118.306561] __switch_to+0x108/0x1c0
[ 118.310138] genpd_power_off+0x114/0x278
[ 118.314059] genpd_power_off+0x198/0x278
[ 118.317980] genpd_runtime_suspend+0x188/0x258
[ 118.322425] __rpm_callback+0x88/0x140
[ 118.326172] rpm_callback+0x20/0x80
[ 118.329659] rpm_suspend+0xf0/0x410
[ 118.333146] rpm_idle+0xac/0x150
[ 118.336372] __pm_runtime_idle+0x58/0xa0
[ 118.340297] hantrodec_release+0xdc/0x1c0
[ 118.344307] __fput+0x88/0x228
[ 118.347360] ____fput+0xc/0x18
[ 118.350414] task_work_run+0x80/0x130
[ 118.354076] do_notify_resume+0x1f0/0x688
[ 118.358084] work_pending+0x8/0x10

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Fri, 26 Feb 2021 14:24:24 GMT

Hi joanxie,

it has something to do with the powermanagement. I comment all lines that started with "pm_runtime" in the file /drivers/mxc/hantro_845/hantrodec_845s.c and after compiling everything works fine.

To comment these lines is not the solution but the proof that the power management is involved.

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Wed, 03 Mar 2021 03:27:34 GMT

“I also tried to play a h264 video file. But when the video finished the process also stall.”
> what do you mean? how could we reproduce this? I playback 1080p@30 mp4 video successfully, could you share your log file about "stall", I playback the mp4 file which has 10 mins, I didn't see any abnormal finish the process, I use the command like "gplay-1.0 xxx.mp4"

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Wed, 03 Mar 2021 06:36:01 GMT

My first thoughts were that the stall has something to do with STRG+C. So I played a short video because I want to see if the process also stall when the video finished normaly.

I used this gstreamer-pipeline:

gst-launch-1.0 filesrc location=test.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false

But have you seen my post from friday? The problem seems to be the power management.

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Thu, 18 Mar 2021 02:56:06 GMT

I couldn't reproduce this on my i.mx8mp evk board

root@8mpevk:~# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4

Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:05:27.685532375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (19035), playing for (0:05:27.686027375), fps (58.089).
Freeing pipeline ...

what imx8mp board do you use?A0 or A1? to check the schematic of board, mine is SCH-46370 REV A2

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Thu, 18 Mar 2021 07:47:52 GMT

I'm using Phytec i.MX8MPlus LPDDR4 Pollux EVK board.

Kernel: 5.4.70 (aarch64)

GStreamer: 1.16.2

VPUDEC: 4.5.7

warpper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Aug 17 2020 07:03:22)

vpulib: 1.1.1

firmware: 1.1.1.0

autovideosink -> waylandsink

RTP-Stream:

GStreamer pipeline: gst-launch1.0 -v udpsrc port=8884 ! application /x-rtp,payload=96 ! rtph264depay ! queue ! h264parse ! vpudec ! autovideosink sync=false

MP4-File:

gst-launch-1.0 filesrc location=/home/user0/recording_10-08-2020_023237.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false

Did you activate PowerManagement in your kernel config?

First the stream will show correctly on the screen. When I cancel the task (STRG+C) the process stall. The only way to get the board running again is resetting the board.

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Tue, 23 Mar 2021 02:06:35 GMT

Did you activate PowerManagement in your kernel config?

> I don't change any source code, use the pre-built images

I tested again, use ctrl+C to stop the video and replay again, I didn't find any issue on my board

root@imx8mpevk:/mnt# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false
Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.961084125
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (419), playing for (0:00:06.961601125), fps (60.187).
Freeing pipeline ...
root@imx8mpevk:/mnt# gst-launch-1.0 filesrc location=/mnt/bbb_sunflower_1080p_30fps_normal.mp4 ! qtdemux ! h264parse ! vpudec ! autovideosink sync=false
Setting pipeline to PAUSED ...
====== VPUDEC: 4.5.7 build on Dec 6 2020 16:44:14. ======
wrapper: 3.0.0 (VPUWRAPPER_ARM64_LINUX Build on Nov 17 2020 17:08:21)
vpulib: 1.1.1
firmware: 1.1.1.0
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.209655000
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total showed frames (255), playing for (0:00:04.210131625), fps (60.568).
Freeing pipeline ...

"^Chandling interrupt." is for my ctrl+c command

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Tue, 23 Mar 2021 05:24:27 GMT

Yes PM is activated in my kernel config. Currently I do not find a way to disable it. It seems that it is firmly rooted with the architecture.

I attached my current .config.

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Wed, 24 Mar 2021 12:16:58 GMT

I didn't find "pm_runtime" in my source code:

"https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=imx_5.4.70_2.3.0"

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Wed, 24 Mar 2021 12:18:29 GMT

when did you get these log file? after delete the "pm_runtime" ?

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

MicMoba — Wed, 24 Mar 2021 12:45:44 GMT

Hi joanxie,

no this was the log I got when I use the not modified hantrodec_845s.c file and I interrupt (CTRL+C) a current gstreamer stream.

After deleting all lines that started with "pm_" in the hantrodec_845s.c file everything works fine.

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

joanxie — Thu, 25 Mar 2021 09:27:02 GMT

obviously you use the third party bsp, I think you should contact their engineer and confirm why they they add the PM there

Re: i.MX8MPlus crashes when using gstreamer vpudec plugin

xinhkang — Fri, 16 Apr 2021 18:13:43 GMT

there are tons of function calls starts with pm_runtime_, for example:
https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=imx_5.4.70_2.3.0#n316

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/mxc/hantro_845/hantrodec_845s.c?h=imx_5.4.70_2.3.0#n1352

...

we are hitting similar issue (NXP beta2 FW):

Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881498] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881518] rcu: 3-...0: (0 ticks this GP) idle=0f2/1/0x4000000000000000 softirq=10466/10466 fqs=25097
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881523] (detected by 2, t=52517 jiffies, g=27385, q=6357)
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881526] Task dump for CPU 3:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881533] rsp_srv_1 R running task 0 2667 2535 0x00000002
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881540] Call trace:
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881552] __switch_to+0x104/0x170
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881560] genpd_power_off+0x114/0x278
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881564] genpd_power_off+0x198/0x278
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881570] genpd_runtime_suspend+0x188/0x258
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881576] __rpm_callback+0x88/0x140
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881582] rpm_callback+0x20/0x80
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881586] rpm_suspend+0xf0/0x410
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881592] rpm_idle+0xac/0x150
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881597] __pm_runtime_idle+0x58/0xa0
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881607] hantrodec_release+0xdc/0x1c0
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881613] __fput+0x88/0x228
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881616] ____fput+0xc/0x18
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881626] task_work_run+0x80/0x130
Apr 15 23:06:31 imx8mpevk kernel[47]: [ 412.881633] do_notify_resume+0x1f0/0x450