iMX8MP: VPU hangs after playback

[santhanakumar]

Hi Everyone,

We have a custom board based on imx8mp.
I tried to build the latest Yocto (Yocto version: Kirkstone, Linux Kernel version: 5.15) following the below document:
https://www.nxp.com/docs/en/user-guide/IMX_YOCTO_PROJECT_USERS_GUIDE.pdf

I configured Yocto with the below parameters:
Distro: fsl-imx-xwayland
Machine: imx8mp-lpddr4-evk
Image: imx-image-full
I could able to build the image successfully.

I did not make any changes to the SDK and generated default image for imx8mp evk.
I then tried to evaluate VPU by running gstreamer and able to play 1080p video without issues.
But system freezes after playback is complete.
Please find the below logs for the same.

root@imx8mp:~# gst-pl ay -1.0 /sample_video.MOV
Press 'k' to see a list of keyboard shortcuts.
Now playing /sample_video.MOV
Prerolling...
====== AIUR: 4.7.1 build on Jul 19 2022 06:15:03. ======
Core: MPEG4PARSER_06.20.06 build on May 7 2022 07:52:31
file: /usr/lib/imx-mm/parser/lib_mp4_parser_arm_elinux.so.3.2
------------------------
Track 00 [audio_0] Enabled
Duration: 0:00:16.904126000
Language: und
Mime:
audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)44100, bitrate=(int)0, stream-format=(string)raw, codec_data=(buffer)1210
------------------------

====== BEEP: 4.7.1 build on Jul[ 22.313492] remoteproc remoteproc0: powering up imx-dsp-rproc
19 2022 06:15:03. ======
Core: DSP decoder Wrapper build on May 19 2022 08:49:06
file: /usr/lib/imx-mm/audio-codec/wrap/lib_dsp_wrap_arm_elinux.so
[ 22.333189] remoteproc remoteproc0: Booting fw image imx/dsp/hifi4.bin, size 804421
[ 22.344940] remoteproc0#vdev0buffer: assigned reserved memory node vdev0buffer@94300000
[ 22.353798] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 22.354296] virtio_rpmsg_bus virtio0: creating channel rpmsg-raw addr 0x1
[ 22.359491] remoteproc0#vdev0buffer: registered virtio0 (type 7)
[ 22.366735] virtio_rpmsg_bus virtio0: creating channel rpmsg-raw addr 0x2
[ 22.372792] remoteproc remoteproc0: remote processor imx-dsp-rproc is now up
Audio Device Ready
------------------------
Track 01 [video_0] Enabled
Duration: 0:00:16.850000000
Language: und
Mime:
video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, codec_data=(buffer)0164002affe100102764002aac56c0780227e59a8080808101000428ee3cb0fdf8f800
------------------------
Track 02 [subtitle]: Disabled
Codec: 0, SubCodec: 0
------------------------
Track 03 [subtitle]: Disabled
Codec: 0, SubCodec: 0
------------------------
Track 04 [subtitle]: Disabled
Codec: 0, SubCodec: 0
------------------------

====== V4L2DEC: 1.20.0 build on Jul 28 2022 06:57:16. ======
Redistribute latency...

===!!! Current pulsesink device is alsa_output.platform-sound-xcvr.iec958-stereo !!!===

Redistribute latency...
0:00:00.0 / 0:00:16.8
0:00:16.9 / 0:00:16.8
Reached end of play list.
Total showed frames (1011), playing for (0:00:16.905064000), fps (59.805).
[ 39.566308] remoteproc remoteproc0: stopped remote processor imx-dsp-rproc
[ 60.574505] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 60.580627] rcu: 3-...0: (0 ticks this GP) idle=0c1/1/0x4000000000000000 softirq=6730/6732 fqs=2571
[ 60.589853] (detected by 2, t=5254 jiffies, g=8953, q=5495)
[ 60.595514] Task dump for CPU 3:
[ 60.598744] task:vsidaemon state:R running task stack: 0 pid: 836 ppid: 2 flags:0x00000202
[ 60.608666] Call trace:
[ 60.611109] __switch_to+0x108/0x160
[ 60.614698] genpd_runtime_suspend+0x118/0x2c0
[ 60.619147] __rpm_callback+0x48/0x170
[ 60.622898] rpm_callback+0x6c/0x80
[ 60.626388] rpm_suspend+0x100/0x554
[ 60.629965] rpm_idle+0xf4/0x154
[ 60.633194] __pm_runtime_idle+0x60/0xb0
[ 60.637120] hantrodec_release+0x1c8/0x240
[ 60.641221] __fput+0x70/0x234
[ 60.644280] ____fput+0x14/0x20
[ 60.647423] task_work_run+0x84/0x17c
[ 60.651088] do_notify_resume+0x208/0x12ec
[ 60.655186] el0_svc+0x6c/0x80
[ 60.658246] el0t_64_sync_handler+0xa8/0x130
[ 60.662518] el0t_64_sync+0x1a0/0x1a4
[ 123.683477] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 123.689590] rcu: 3-...0: (0 ticks this GP) idle=0c1/1/0x4000000000000000 softirq=6730/6732 fqs=10343
[ 123.698903] (detected by 1, t=21032 jiffies, g=8953, q=20683)
[ 123.704737] Task dump for CPU 3:
[ 123.707963] task:vsidaemon state:R running task stack: 0 pid: 836 ppid: 2 flags:0x00000202
[ 123.717884] Call trace:
[ 123.720329] __switch_to+0x108/0x160
[ 123.723916] genpd_runtime_suspend+0x118/0x2c0
[ 123.728367] __rpm_callback+0x48/0x170
[ 123.732122] rpm_callback+0x6c/0x80
[ 123.735613] rpm_suspend+0x100/0x554
[ 123.739192] rpm_idle+0xf4/0x154
[ 123.742421] __pm_runtime_idle+0x60/0xb0
[ 123.746345] hantrodec_release+0x1c8/0x240
[ 123.750445] __fput+0x70/0x234
[ 123.753503] ____fput+0x14/0x20
[ 123.756646] task_work_run+0x84/0x17c
[ 123.760310] do_notify_resume+0x208/0x12ec
[ 123.764410] el0_svc+0x6c/0x80
[ 123.767468] el0t_64_sync_handler+0xa8/0x130
[ 123.771740] el0t_64_sync+0x1a0/0x1a4
[ 166.793653] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-... } 5581 jiffies s: 157 root: 0x8/.
[ 166.804366] rcu: blocking rcu_node structures (internal RCU debug):
[ 166.810666] Task dump for CPU 3:
[ 166.813912] task:vsidaemon state:R running task stack: 0 pid: 836 ppid: 2 flags:0x00000202
[ 166.823894] Call trace:
[ 166.826362] __switch_to+0x108/0x160
[ 166.829968] genpd_runtime_suspend+0x118/0x2c0
[ 166.834435] __rpm_callback+0x48/0x170
[ 166.838205] rpm_callback+0x6c/0x80
[ 166.841738] rpm_suspend+0x100/0x554
[ 166.845318] rpm_idle+0xf4/0x154
[ 166.848599] __pm_runtime_idle+0x60/0xb0
[ 166.852549] hantrodec_release+0x1c8/0x240
[ 166.856676] __fput+0x70/0x234
[ 166.859755] ____fput+0x14/0x20
[ 166.862917] task_work_run+0x84/0x17c
[ 166.866630] do_notify_resume+0x208/0x12ec
[ 166.870752] el0_svc+0x6c/0x80
[ 166.873828] el0t_64_sync_handler+0xa8/0x130
[ 166.878119] el0t_64_sync+0x1a0/0x1a4
[ 186.729335] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 186.735454] rcu: 3-...0: (0 ticks this GP) idle=0c1/1/0x4000000000000000 softirq=6730/6732 fqs=18090
[ 186.744772] (detected by 1, t=36794 jiffies, g=8953, q=36033)
[ 186.750608] Task dump for CPU 3:
[ 186.753835] task:vsidaemon state:R running task stack: 0 pid: 836 ppid: 2 flags:0x00000202
[ 186.763756] Call trace:
[ 186.766202] __switch_to+0x108/0x160
[ 186.769789] genpd_runtime_suspend+0x118/0x2c0
[ 186.774237] __rpm_callback+0x48/0x170
[ 186.777989] rpm_callback+0x6c/0x80
[ 186.781478] rpm_suspend+0x100/0x554
[ 186.785056] rpm_idle+0xf4/0x154
[ 186.788284] __pm_runtime_idle+0x60/0xb0
[ 186.792208] hantrodec_release+0x1c8/0x240
[ 186.796308] __fput+0x70/0x234
[ 186.799367] ____fput+0x14/0x20
[ 186.802510] task_work_run+0x84/0x17c
[ 186.806174] do_notify_resume+0x208/0x12ec
[ 186.810273] el0_svc+0x6c/0x80
[ 186.813330] el0t_64_sync_handler+0xa8/0x130
[ 186.817602] el0t_64_sync+0x1a0/0x1a4
[ 230.281056] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-... } 21453 jiffies s: 157 root: 0x8/.
[ 230.291851] rcu: blocking rcu_node structures (internal RCU debug):
[ 230.298149] Task dump for CPU 3:
[ 230.301398] task:vsidaemon state:R running task stack: 0 pid: 836 ppid: 2 flags:0x00000202
[ 230.311338] Call trace:
[ 230.313830] __switch_to+0x108/0x160
[ 230.317447] genpd_runtime_suspend+0x118/0x2c0
[ 230.321916] __rpm_callback+0x48/0x170
[ 230.325688] rpm_callback+0x6c/0x80
[ 230.329195] rpm_suspend+0x100/0x554
[ 230.332776] rpm_idle+0xf4/0x154
[ 230.336022] __pm_runtime_idle+0x60/0xb0
[ 230.339992] hantrodec_release+0x1c8/0x240
[ 230.344122] __fput+0x70/0x234
[ 230.347203] ____fput+0x14/0x20
[ 230.350364] task_work_run+0x84/0x17c
[ 230.354048] do_notify_resume+0x208/0x12ec
[ 230.358167] el0_svc+0x6c/0x80
[ 230.361241] el0t_64_sync_handler+0xa8/0x130
[ 230.365560] el0t_64_sync+0x1a0/0x1a4

The logs are taken from serial console and has both gstramer and kernel logs.
System hangs and not usable after playback is complete.

Please confirm gstreamer is validated with video playback in the latest SDK.
I am playing the below video in Gstreamer:
https://hq0epm0west0us0storage.blob.core.windows.net/development/Temp/sample_video.MOV

Regards,
Santhana Kumar

[santhanakumar · Accepted solution]

I could able to fix the issue myself.
Actually, the problem exists only in 4G memory version of our board and no issues are observed in 2G version.
I have removed power management routines from hantro decoder; playback now works fine.

[artsiomstaliaro]

Hi Kumar,

Its looks like you have issues with Audio DSP. Try to play video without audio output (disable it).

[santhanakumar]

Hi Joanxie,

As I mentioned before, I am using EVK image to test VPU.
I am just appending our uboot on top of EVK image and this should replicate EVK behaviour in our board as well.

Could you please share the image that you used for testing VPU in EVK so that i can test here as well to understand the issue.

Also, we are using 4G memory compared to 6G memory used in EVK.
I find the below cma settings in evk dtb.

linux,cma {
compatible = "shared-dma-pool";
reusable;
size = <0 0x3c000000>;
alloc-ranges = <0 0x40000000 0 0xC0000000>;
linux,cma-default;
};

Do we need to modify CMA node in DTB for 4G memory size?
Please share your inputs.
Also please share the evk image used for testing VPU.

Regards,
Santhana Kumar

[joanxie]

I just use 5.15 demo images from

"https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

since you use new DDR, did you pass the stress test tool? and did you change the uboot according to your new DDR? pls try to pass stress test firsly

"https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-8M-Family-DDR-Tool-Release/ta-p/110..."

[joanxie]

I have tested your sample video on i.mx8mp evk board successfully, pls compare your own code with nxp bsp to debug your own code