IMX6 & BSP 4.1.15: Memory leak?

[grim]

We have a product that uses the i.mx6 solo.

We've recently needed to upgrade from the 3.14.28 BSP to 4.1.15.

When our product is active and idle, it just plays videos until somebody touches the screen.

Initially, with 3.14.28, we experienced memory losses and were able to fix them by

1. Switching from gstreamer .10 to gstreamer 1.0
2. Periodically clearing the cache buffers (echo 3 > /proc/sys/vm/drop_caches)

Once we made these changes, the system would run indefinitely without running out of memory.

We made the switch to 4.1.15.

We're still using gstreamer 1.0 and periodically clearing the cache buffers but the system eventually runs out of memory and our application crashes and burns.

On one of the systems that crashed (after 5 days) I captured a couple of things that I thought were unusual...

I ran the free command and observed the output...

  free

  total        used        free      shared  buff/cache   available
  Mem:        1026448      222500      220688        1916      583260      186664
  Swap:             0           0           0

Then I cleared the cache...

echo 3 > /proc/sys/vm/drop_caches

Run the free command again...

         free

         total        used        free      shared  buff/cache   available
         Mem:        1026448      222472      221680         488      582296      187884
         Swap:             0           0           0

"Clearing" the cache freed up VERY little memory.

That system was eventually rebooted and is running again, waiting for it to crash again.

On another system, I made some observations while the system was running.

I went and captured "/proc/meminfo" periodically and observed it.

Here's the output from a machine that's only been running for a few hours...

cat /proc/meminfo

MemTotal:        1026404 kB
MemFree:          684864 kB
MemAvailable:     739740 kB
Buffers:            1268 kB
Cached:            63176 kB
SwapCached:            0 kB
Active:            63948 kB
Inactive:          46632 kB
Active(anon):      46120 kB
Inactive(anon):      468 kB
Active(file):      17828 kB
Inactive(file):    46164 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:        1026404 kB
LowFree:          684864 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         46116 kB
Mapped:            53976 kB
Shmem:               472 kB
Slab:              42312 kB
SReclaimable:       3032 kB
SUnreclaim:        39280 kB
KernelStack:         752 kB
PageTables:         1152 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      513200 kB
Committed_AS:     308276 kB
VmallocTotal:    1024000 kB
VmallocUsed:        8784 kB
VmallocChunk:     824796 kB
CmaTotal:         327680 kB
CmaFree:          155088 kB

Observing /proc/meminfo an hour or so later...

cat /proc/meminfo

MemTotal:        1026404 kB
MemFree:          644440 kB
MemAvailable:     729676 kB
Buffers:            2376 kB
Cached:            92420 kB
SwapCached:            0 kB
Active:           104140 kB
Inactive:          37532 kB
Active(anon):      46868 kB
Inactive(anon):      460 kB
Active(file):      57272 kB
Inactive(file):    37072 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:        1026404 kB
LowFree:          644440 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                80 kB
Writeback:             0 kB
AnonPages:         46856 kB
Mapped:            53976 kB
Shmem:               472 kB
Slab:              48476 kB
SReclaimable:       3048 kB
SUnreclaim:        45428 kB
KernelStack:         752 kB
PageTables:         1156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      513200 kB
Committed_AS:     308276 kB
VmallocTotal:    1024000 kB
VmallocUsed:        8784 kB
VmallocChunk:     826636 kB
CmaTotal:         327680 kB
CmaFree:          155076 kB

What I've noticed is that SUnreclaim continues to grow. I have yet to observe how high it gets before the system crashes but it is definitely growing.

Is this a known issue? Are there any known workarounds/patches/fixes? I see that kernel 4.9 is now available.... will I experience the same issue with 4.9?

I'm in the process of adding "slabtop" support to my system so that I can try to get a better idea of what's going on.

Any tips or pointers would be greatly appreciated.

[igorpadykov]

Hi grim

one can look at similar L4.1.15 issue on

i.MX6 Memory issues after long encoding 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[grim]

Hello Igor,

Thank you for your attention to this.

I wasn't able to remedy the solution by modifying anything in the kernel.

I did some more digging around and discovered that there were other people having similar issues running gstreamer 1.8.X in their non-imx systems.

I tried to just update my BSP with the most recent version of gstreamer (1.12.X) and ran into many issues. Ended up abandoning that path.

I ended up downloading and installing NXP's 4.9.88 BSP. It uses gsteamer 1.12.2.

I'm still going through some of the pains of modifying the BSP to work with our custom board but I managed to get everything up and running enough so that I could run our application. After a short run time, I can see that there IS an enormous improvement (hesitant to say that it's "fixed" at this point).

Now, when I read /proc/meminfo, I can see that the SUnreclaim value isn't growing anymore. I want to get some more run-time with this BSP but it's looking pretty good so far.

I am seeing this message come up on the debug port..... I would like to discuss it... Should I start another thread to discuss this?

Here's the message...

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1035 at mm/page_alloc.c:7393 cma_release+0x7c/0x94

====== AIUR: 4.3.5 build on Apr  2 2019 18:30:56. ======
        Core: MPEG4PARSER_06.14.02  build on Mar 27 2018 02:55:00
 file: /us338 pages are still in use!
r/lib/imx-mm/parser/lib_mp4_parseModules linked in:r_arm11_elinux.so.3.2

CPU: 0 PID: 1035 Comm: QSGRenderThread Tainted: G        W       4.9.88-M821-Ver_0_15-04776-g6507266 #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[<8010ecb4>] (unwind_backtrace) from [<8010b394>] (show_stack+0x10/0x14)
[<8010b394>] (show_stack) from [<803c158c>] (dump_stack+0x78/0x8c)
[<803c158c>] (dump_stack) from [<8012dbc4>] (__warn+0xe8/0x100)
[<8012dbc4>] (__warn) from [<8012dc14>] (warn_slowpath_fmt+0x38/0x48)
[<8012dc14>] (warn_slowpath_fmt) from [<801fcd94>] (cma_release+0x7c/0x94)
[<801fcd94>] (cma_release) from [<80113c48>] (__arm_dma_free.constprop.2+0x11c/0x130)
[<80113c48>] (__arm_dma_free.constprop.2) from [<80727298>] (vpu_free_dma_buffer+0x8c/0xac)
[<80727298>] (vpu_free_dma_buffer) from [<80728760>] (vpu_ioctl+0x8cc/0xb94)
[<80728760>] (vpu_ioctl) from [<80211be0>] (do_vfs_ioctl+0x9c/0x920)
[<80211be0>] (do_vfs_ioctl) from [<80212498>] (SyS_ioctl+0x34/0x58)
[<80212498>] (SyS_ioctl) from [<80107880>] (ret_fast_syscall+0x0/0x48)
------------------------
    Track 00 [video_0] Enabled
        Duration: 0:01:02.020288000
        Language: eng
    Mime:
        video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)1280, height=(int)720, framerate=(fraction)24000/1001, codec_data=(buffer)0164002affe100186764002aac2ca5014016e9a808080a000007d2000177010801000568e9093525fdf8f800
------------------------
[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0
[WARN]  VPU iram is less than needed, some parts don't use iram
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
---[ end trace 9d69178300dccf36 ]---
alloc_contig_range: [46600, 4678a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy
alloc_contig_range: [46500, 4668a) PFNs busy

I'm seeing this message every time a new video starts.

Everything in the system seems to be running just fine.....with the exception of the message above.

I guess I should directly ask.... Is 4.9.88 recommended/intended to be used on an i.MX6 Solo core processor?

Thanks, again, for your attention to this Igor!!

[igorpadykov]

Hi grim

recommended to use latest Linux L4.14.78

Linux L4.14.78_1.0.0 Documentation

Best regards
igor

[grim]

Hello Igor.

I got 4.14.78 up and running on my system.

Had a difficult time building the Linux BSP when I switched from 'systemd' to 'sysvinit' in fsl-imx-preferred-env.inc.

Had to add "wifi" to DISTRO_FEATURES_remove, otherwise yocto would error out when putting the root filesystem together.

Now that everything is running, I'm still getting the following warning (when stopping or looping video)...

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1077 at mm/page_alloc.c:7666 cma_release+0x7c/0x94

====== AIUR: 4.4.4 build on Apr  7 2019 05:31:45. ======
        Core: MPEG4PARSER_06.15.02  build on 338 pages are still in use!
Nov  2 2018 10:35:46
 file: /usrModules linked in:
/lib/imx-mm/parser/lib_mp4_parserCPU: 0 PID: 1077 Comm: QSGRenderThread Tainted: G        W       4.14.78-M821-Ver_0_15-05595-g94da7bd-dirty #7
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[<8010ef44>] (unwind_backtrace) from [<8010b49c>] (show_stack+0x10/0x14)
[<8010b49c>] (show_stack) from [<809c2524>] (dump_stack+0x78/0x8c)
[<809c2524>] (dump_stack) from [<8012dd18>] (__warn+0xe8/0x100)
[<8012dd18>] (__warn) from [<8012dd68>] (warn_slowpath_fmt+0x38/0x48)
[<8012dd68>] (warn_slowpath_fmt) from [<80202bc4>] (cma_release+0x7c/0x94)
[<80202bc4>] (cma_release) from [<8011464c>] (__arm_dma_free.constprop.2+0x11c/0x130)
[<8011464c>] (__arm_dma_free.constprop.2) from [<806f65e4>] (vpu_free_dma_buffer+0x8c/0xac)
[<806f65e4>] (vpu_free_dma_buffer) from [<806f7aac>] (vpu_ioctl+0x8cc/0xb94)
[<806f7aac>] (vpu_ioctl) from [<80218b8c>] (do_vfs_ioctl+0x9c/0x8c4)
[<80218b8c>] (do_vfs_ioctl) from [<802193e8>] (SyS_ioctl+0x34/0x58)
[<802193e8>] (SyS_ioctl) from [<80107980>] (ret_fast_syscall+0x0/0x54)
_arm11_elinux.so.3.2
------------------------
    Track 00 [video_0] Enabled
        Duration: 0:01:02.020288000
        Language: eng
    Mime:
        video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)1280, height=(int)720, framerate=(frac
------------------------
[INFO]  bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0
[WARN]  VPU iram is less than needed, some parts don't use iram
---[ end trace 232912683f64c254 ]---

In spite of the warning, the video still seems to play fine and I'm not seeing any immediate signs of any memory leaks or anything.

Before I dig into the kernel source, does this warning look familiar to you?

I don't want to release this to the field without, at least, understanding the implications of this warning.

Thanks and Best Regards!

[grim]

I will add that I've modified the default device  tree for LCD, backlight, serial, USB and I2C devices for kernel version 4.14.78. These modifications are minor.

The kernel is based off imx6_v7_defconfig for 4.14.78, with the exception of enabling drivers for the types of devices listed above.

Regards!

[grim]

Thanks Igor!

I'll give 4.14.78 a shot and post an update in a few days.