imx8mq yocto4.14.98 memory leak when play video

[jiangyaqiang]

hi anyone:

          I got memory leak when play video with imx8mq evk.

environment:

hardware : imx8mq evk

software : yocto-4.14.98

test command(the vidoe file the a 4K video, and 1 minute long):

root@imx8mqevk:~#cat video_stress_test.sh
input_value=$1

while [ "1" = "1" ]
do
gst-launch-1.0 playbin uri=file:///home/root/a.mp4 video-sink=waylandsink
done

test result:

====== BEEP: 4.4.5 build on Mar 22 2019 14:30:34. ======
Core: AAC decoder Wrapper build on Dec 7 2017 18:13:51
file: /usr/lib/imx-mm/audio-codec/wrap/lib_aacd_wrap_arm_elinux.so.3
CODEC: BLN_MAD-MMCODECS_AACD_ARM_03.09.00_ARMV8 build on Sep 20 2017 15:02:50.
[1058365.419206] alloc_contig_range: [5c000, 5ced5) PFNs busy
[1058365.427176] alloc_contig_range: [5c100, 5cfd5) PFNs busy
[1058365.442904] alloc_contig_range: [5d100, 5dfd5) PFNs busy
[1058365.444523] pulseaudio invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
[1058365.449745] alloc_contig_range: [5d200, 5e0d5) PFNs busy
[1058365.459399] pulseaudio cpuset=/ mems_allowed=0
[1058365.469692] CPU: 0 PID: 3692 Comm: pulseaudio Tainted: G O 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[1058365.480231] Hardware name: Freescale i.MX8MQ EVK (DT)
[1058365.480564] alloc_contig_range: [5d300, 5e1d5) PFNs busy
[1058365.485456] Call trace:
[1058365.485470] [<ffff000008089c48>] dump_backtrace+0x0/0x3c8
[1058365.485476] [<ffff00000808a024>] show_stack+0x14/0x20
[1058365.492665] alloc_contig_range: [5d400, 5e2d5) PFNs busy

===!!! Current pulsesink device[1058365.493574] [<ffff000008d8ecc0>] dump_stack+0x9c/0xbc
is alsa_output.platform-sound-hd[1058365.493583] [<ffff00000818f928>] dump_header+0x90/0x1e8
mi.stereo-fallback !!!===

[1058365.500502] alloc_contig_range: [5d400, 5e3d5) PFNs busy
[1058365.504370] [<ffff00000818eb44>] oom_kill_process+0x2a4/0x5c8
[1058365.504374] [<ffff00000818f500>] out_of_memory+0x198/0x4c0
[1058365.504379] [<ffff000008194e04>] __alloc_pages_nodemask+0xb04/0xb38
[1058365.504384] [<ffff0000081e8664>] alloc_pages_current+0x7c/0xe8
[1058365.504389] [<ffff0000081906bc>] __get_free_pages+0xc/0x28
[1058365.504394] [<ffff00000822b3e0>] __pollwait+0x58/0x100
[1058365.504399] [<ffff000008263b0c>] eventfd_poll+0x34/0x70
[1058365.504403] [<ffff00000822c5c8>] do_sys_poll+0x330/0x4e8
[1058365.504407] [<ffff00000822d2b0>] SyS_ppoll+0x178/0x1f0
[1058365.504410] Exception stack(0xffff0000239bbec0 to 0xffff0000239bc000)
[1058365.504415] bec0: 0000000019398760 0000000000000013 0000ffffca7c4510 0000000000000000
[1058365.504419] bee0: 0000000000000008 0000000000000000 0000000000000000 000000001f83b8a6
[1058365.504423] bf00: 0000000000000049 003b9aca00000000 000000000010263d 0000000014accc9c
[1058365.504426] bf20: 0000000000000018 000000005c7dddac 00174c3b1b020291 00003b8da02832cc
[1058365.504430] bf40: 0000ffffb0886000 0000ffffb05d10e0 0000ffffb065aa70 0000ffffca7c4510
[1058365.504434] bf60: 0000000000000000 0000000000000013 0000000019398760 0000000000000000
[1058365.504438] bf80: 000000001937a810 0000000000000000 0000000000000001 0000000000410000
Redistribute latency...
[1058365.504441] bfa0: 00000000ffffffff 0000ffffca7c44e0 0000ffffb05d1158 0000ffffca7c44e0
[1058365.504445] bfc0: 0000ffffb05d1178 0000000080000000 0000000019398760 0000000000000049
[1058365.504449] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[1058365.504454] [<ffff000008083b18>] __sys_trace_return+0x0/0x4
[1058365.504633] Mem-Info:
[1058365.512019] alloc_contig_range: [5d400, 5e4d5) PFNs busy
[1058365.518199] active_anon:519736 inactive_anon:16619 isolated_anon:0
[1058365.518199] active_file:719 inactive_file:2301 isolated_file:0
[1058365.518199] unevictable:0 dirty:0 writeback:0 unstable:0
[1058365.518199] slab_reclaimable:2047 slab_unreclaimable:6735
[1058365.518199] mapped:4557 shmem:19801 pagetables:1699 bounce:0
[1058365.518199] free:88572 free_pcp:0 free_cma:83959
[1058365.518211] Node 0 active_anon:2078944kB inactive_anon:66476kB active_file:2876kB inactive_file:9204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:18228kB dirty:0kB writeback:0kB shmem:79204kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1703936kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[1058365.529029] alloc_contig_range: [5d400, 5e5d5) PFNs busy
[1058365.534580] Node 0 DMA free:354276kB min:30000kB low:37500kB high:45000kB active_anon:2078956kB inactive_anon:66476kB active_file:2840kB inactive_file:9140kB unevictable:0kB writepending:0kB present:3108864kB managed:3037148kB mlocked:0kB kernel_stack:2256kB pagetables:6796kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:335824kB
[1058365.534596] lowmem_reserve[]: 0 0 0
[1058365.794469] Node 0 DMA: 267*4kB (UEHI) 103*8kB (UEHI) 69*16kB (UEHCI) 57*32kB (UEHCI) 26*64kB (UEHI) 8*128kB (UEI) 38*256kB (UECI) 37*512kB (ECI) 32*1024kB (CI) 34*2048kB (ECI) 23*4096kB (CI) 12*8192kB (UCI) 1*16384kB (C) 0*32768kB = 347476kB
[1058365.816294] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[1058365.825403] 23127 total pagecache pages
[1058365.829499] 0 pages in swap cache
[1058365.833246] Swap cache stats: add 0, delete 0, find 0/0
[1058365.838749] Free swap = 0kB
[1058365.841897] Total swap = 0kB
[1058365.845037] 777216 pages RAM
[1058365.848231] 0 pages HighMem/MovableOnly
[1058365.852378] 17929 pages reserved
[1058365.855870] 245760 pages cma reserved
[1058365.859792] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[1058365.868542] [ 2090] 0 2090 9637 2527 15 3 0 0 systemd-journal
[1058365.878223] [ 3394] 994 3394 20031 91 9 4 0 0 systemd-timesyn
[1058365.887909] [ 3397] 0 3397 3394 335 9 4 0 -1000 systemd-udevd
[1058365.897435] [ 3410] 993 3410 1428 97 6 4 0 0 systemd-network
[1058365.907135] [ 3460] 0 3460 522 27 5 3 0 0 atd
[1058365.915776] [ 3463] 0 3463 754 139 5 3 0 0 crond
[1058365.924602] [ 3491] 998 3491 831 69 6 3 0 0 rpcbind
[1058365.933584] [ 3492] 0 3492 1381 108 7 4 0 0 systemd-logind
[1058365.943192] [ 3502] 996 3502 1069 59 5 3 0 0 avahi-daemon
[1058365.952650] [ 3506] 999 3506 1067 141 6 3 0 -900 dbus-daemon
[1058365.962006] [ 3537] 996 3537 1069 59 5 3 0 0 avahi-daemon
[1058365.971433] [ 3561] 0 3561 484 37 4 3 0 0 syslogd
[1058365.980464] [ 3569] 0 3569 553 22 5 3 0 0 tee-supplicant
[1058365.990117] [ 3574] 0 3574 1677 148 6 3 0 0 ofonod
[1058365.999002] [ 3593] 0 3593 2459 172 8 3 0 0 connmand
[1058366.008110] [ 3599] 0 3599 2432 116 8 3 0 0 wpa_supplicant
[1058366.017801] [ 3602] 0 3602 2456 2027 9 3 0 0 klogd
[1058366.026604] [ 3607] 997 3607 746 141 6 3 0 0 rpc.statd
[1058366.035749] [ 3609] 0 3609 481 26 5 3 0 0 agetty
[1058366.044633] [ 3618] 0 3618 1076 100 6 3 0 0 login
[1058366.053434] [ 3624] 0 3624 1027 50 6 3 0 0 weston-launch
[1058366.062925] [ 3626] 0 3626 1863 160 8 3 0 0 systemd
[1058366.071897] [ 3627] 0 3627 2421 421 9 3 0 0 (sd-pam)
[1058366.080955] [ 3631] 0 3631 2561 413 10 3 0 0 (sd-pam)
[1058366.090064] [ 3632] 0 3632 606873 498200 1121 6 0 0 weston
[1058366.098948] [ 3670] 0 3670 113455 4681 156 4 0 0 weston-keyboard
[1058366.108613] [ 3671] 0 3671 113382 4636 158 4 0 0 weston-desktop-
[1058366.118279] [ 3676] 0 3676 813 99 5 3 0 0 sh
[1058366.126816] [ 3680] 0 3680 813 100 5 3 0 0 sh
[1058366.135353] [ 3692] 0 3692 178067 841 23 3 0 0 pulseaudio
[1058366.144584] [ 3698] 0 3698 1416 73 6 3 0 0 bluetoothd
[1058366.153825] [23820] 0 23820 241898 2775 42 4 0 0 gst-launch-1.0
[1058366.163506] Out of memory: Kill process 3632 (weston) score 637 or sacrifice child
[1058366.171402] Killed process 3670 (weston-keyboard) total-vm:453820kB, anon-rss:18940kB, file-rss:8kB, shmem-rss:4kB
[1058366.189302] oom_reaper: reaped process 3670 (weston-keyboard), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
[1058366.473728] alloc_contig_range: 229 callbacks suppressed
[1058366.473733] alloc_contig_range: [5d100, 5dfd5) PFNs busy
[1058366.486192] alloc_contig_range: [5d200, 5e0d5) PFNs busy
[1058366.493468] alloc_contig_range: [5d300, 5e1d5) PFNs busy
[1058366.500406] alloc_contig_range: [60000, 60ed5) PFNs busy
[1058366.508635] alloc_contig_range: [60000, 60fd5) PFNs busy
[1058366.515166] alloc_contig_range: [60200, 610d5) PFNs busy
[1058366.521696] alloc_contig_range: [60300, 611d5) PFNs busy
[1058366.528982] alloc_contig_range: [60400, 612d5) PFNs busy
[1058366.537043] alloc_contig_range: [60400, 613d5) PFNs busy
[1058366.543545] alloc_contig_range: [60400, 614d5) PFNs busy

more log in the attachment. 

[nxf65233]

Hi jiangyaqiang

Can you provide the MP4 file for testing? I need to reproduce it on the imx8mq evk board.

[jiangyaqiang]

HI :

   sorry, the max size of the attachment is 9M , too small to upload a video file. 

But you can try your mp4 file, I have try several files, all have the problem.

video file fomat: AVC , HEVC

audio format: AAC.

It doesn't leak much for one loop, but you can see it by command free, if your video don't have such a problem , please let me know.

[nxf65233]

hi

When I used your script to test the 4K file, there was no problem as you said. Maybe you can use the following command to obtain the kernel source code and compile the test.

git clone https://source.codeaurora.org/external/imx/linux-imx \
-b imx_4.14.98-2.0.0_ga
git tag
git checkout -b rel_imx_4.14.98-2.0.0_ga
make defconfig

You can click the link below to get the 4.14.98demo file for testing

imx8mq 4.14.98 demo 

[jiangyaqiang]

HI :

I have download the release you provided, but it still happen.

You don't see the problem maybe you don't play long enough , it doesn't leave too much, in my test case, the video is about 1minute , it leave 100M to 200M for 24hours.

here is my test result:

root@imx8mqevk:~# cat /proc/version
Linux version 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea (oe-user@oe-host) (gcc version 7.3.0 (GCC)) #1 SMP PREEMPT Sun Apr 14 10:53:57 UTC 2019
root@imx8mqevk:~#

before play video:

root@imx8mqevk:~# top -d 1

KiB Mem : 3037148 total, 2084236 free, 424216 used, 528696 buff/cache

after play twice:

KiB Mem : 3037148 total, 2083232 free, 425168 used, 528748 buff/cache

it can see that some memory lost.

you can play 24hours, and check it again . it will be more obviously.

[nxf65233]

hi

After 20 hours or so of playing video, I leaked about 100MB of memory.
File caching is a common problem with Linux, and multiple reads/writes of files can result in memory fragmentation and caching.
You may be able to do this in the following ways：

To free pagecache, use
echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes, use
echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes, use
echo 3 >/proc/sys/vm/drop_caches

In addition, you can also reduce the pressure on memory all at once by using a SWAP partition.

Please share the test results in time

[jiangyaqiang]

HI nxf65233 

    Thank you for your reply.

    I have try the drop cache, but it doesn't help. The "usage" memory is not in "cache".

    I think usage a "swap" parition will make the crash more slowly,  the the cash will still happen. 

[nxf65233]

This may be a BSP problem, and you should see better results with the latest version of the BSP

[igorpadykov]

Hi jiangyaqiang

could you try to reproduce issue on i.MX8MQ EVK board with latest Demo

i.MX 8M Quad EVK​

use as reference

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-mx8M-EVK-board-encoder-and-decoder/ta-...

Best regards
igor

[jiangyaqiang]

HI :

    Thank you for your reply.

    But the custom have accept the 4.14.98, they don't accept the system upgrade , we need to fix it in 4.14.98.